Inference

Notes

The economics differ sharply from training: inference is bought on tokens per second per dollar and per watt.

Overview

Inference is running a trained AI model on new inputs to produce outputs, such as answering a question, recognising an image or translating a sentence. Training builds the model once, or occasionally; inference happens every time the model is used, so across a popular model's life it can consume more computing than training did. It is the workload most AI chips in phones, cars and data centres are designed to run.

How it works

During inference the model's parameters are fixed. Input data passes forward through the network, layer by layer, and the output is returned. There is no backward pass and no parameter update, so inference needs less memory and arithmetic per input than training, and it can often use lower-precision numbers, such as 8-bit integers or smaller floating-point formats, with little loss of accuracy.

What matters depends on the setting. In a data centre serving many users, the goals are throughput and cost per request while keeping response times acceptable; requests are batched together to use the hardware efficiently. For large language models, generating each new word requires reading much of the model's weights from memory, so memory bandwidth and capacity often limit speed more than arithmetic does. At the edge, in phones, cameras and cars, the limits are power, latency and cost, so neural processing units are built into SoCs.

Software matters as much as hardware. Inference runtimes and compilers, such as NVIDIA TensorRT, optimise models for a specific chip by fusing operations, choosing precisions and arranging memory. MLCommons publishes MLPerf Inference benchmarks so that systems can be compared on standard tasks.

Examples

• A data centre serving a large language model, batching many user requests on each accelerator.
• A smartphone running speech recognition on its built-in neural processing unit without sending audio to the cloud.
• A car's driver assistance system detecting pedestrians in camera images in real time.
• Quantising a model to 8-bit integers so it runs faster and uses less memory.
• MLPerf Inference results comparing systems on image classification and language tasks.

Why it matters

Inference demand drives much of the market for AI accelerators, data centre GPUs, high bandwidth memory and neural processing units in edge devices. Accelerator architects, inference runtime and compiler engineers, performance engineers and product managers all focus on cost, speed and energy per inference.

In the chip lifecycle, target inference workloads shape architecture and memory choices, models are run on simulators and emulators to estimate performance before tapeout, and benchmark results and software optimisation follow first silicon.

It matters most to: AI Performance Benchmarking Engineer, Edge AI SoC Architect, Inference Runtime Engineer, ML Compiler Engineer, NPU / AI Accelerator Architect

Common confusions

Not the same as training

Training adjusts a model's parameters using large datasets and needs far more memory and compute per example. Inference uses the finished model. Inference vs training

Not the same as statistical inference

In statistics, inference means drawing conclusions about a population from a sample. In AI hardware it means running a trained model.

Not the same as logical inference

Rule-based inference engines in older expert systems derived conclusions from logic rules. Today the word mostly refers to neural network models.

Sources

1. MLPerf Inference: Datacenter benchmark (opens in a new tab), MLCommons, 2026-10-22
2. What is AI inference? (opens in a new tab), IBM, 2026-10-22
3. TensorRT SDK (opens in a new tab), NVIDIA, 2026-10-22
4. Introduction to Cloud TPU (opens in a new tab), Google Cloud, 2026-10-22
5. Machine learning (Wikipedia) (opens in a new tab), Wikimedia Foundation, 2026-10-22

Related terms

• Training
• Prefill
• Decode
• Batching

Where it appears on the Hub

Used on 86 pages of the Hub. In each group, the pages that mention it most come first.