When Is High-Bandwidth Flash Cost-Effective for LLM Inference?

Can high-bandwidth flash (HBF) make LLM inference cheaper than adding GPUs? By storing weights closer to the GPU than SSD, HBF may reduce the number of GPUs needed for model capacity. However, this integration is only justified if it meets service targets and its cost is recovered. We simulate this integration using public component data across nine dense and mixture-of-experts (MoE) models, five workloads, two GPU platforms, and 1--100 GPUs. At fixed demand, HBF reduces the minimum GPU count t…

Source: arXiv cs.AR (Hardware Architecture), published 2026-11-10. The summary is the Hub's own; the full story is at the source. Report a rights or accuracy concern