News / Design & IP / Rethinking KV Caching for LLM Decode: Free-Space Optical KV Regeneration for Reduction of Memory Traffic
Rethinking KV Caching for LLM Decode: Free-Space Optical KV Regeneration for Reduction of Memory Traffic
By The Semiconductor Hub news desk ·
Link post
Long-context large language model (LLM) decoding repeatedly reads the historical key-value (KV) cache, making multi-head attention (MHA) memory-bandwidth-bound. Caching layer-input activations instead halves historical-cache storage and traffic, but requires regenerating KV from each cached activation. On digital accelerators such as GPUs, this regeneration takes far longer than fetching the activation from HBM. The KV regeneration overhead outweighs the savings from reduced memory traffic, shi…
Read at arXiv cs.AR (Hardware Architecture)Source: arXiv cs.AR (Hardware Architecture), published . The summary is the Hub's own; the full story is at the source. Report a rights or accuracy concern