Term · Compute & AI Hardware
Decode
The autoregressive phase of language model inference that emits one token at a time, reading the whole KV cache and all weights each step. It is memory bandwidth bound and sets inter-token latency.
- Also known as
- Generation phase, Token generation
- Type
- Term
- Domain
- Compute & AI Hardware, Inference and serving
Notes
Decode throughput scales with bandwidth rather than FLOPS, which is why HBM capacity and speed dominate serving cost.
Related terms
Part of the Compute & AI Hardware glossary, with every term in the topic and the lifecycles, tools and standards that use them.
Where it appears on the Hub
No other page of the Hub uses this term yet.
Suggest an improvement
Suggest an improvement to this entry
A clearer definition, an example, a correction or a source an editor can check. Editors review every suggestion before the entry changes.
Last reviewed: not yet reviewed in depth. Entry last updated . The full glossary