SlipStream: Enabling HBF for GPU Systems by Mitigating the Latency and SRAM Overheads

Large language model (LLM) inference demands ever more memory capacity for larger models and batch sizes. High-Bandwidth Flash (HBF) replaces the DRAM dies of HBM with NAND flash and provides up to 16x the capacity at comparable read bandwidth. However, HBF can incur a read latency of nearly 20$\mu$s, 20x that of HBM, so on-demand HBF accesses slow inference by up to 12.8x. Conventional cache-line prefetchers require 20x more in-flight tracking state to cover this latency. Tensor-streaming desi…

Source: arXiv cs.AR (Hardware Architecture), published 2026-11-13. The summary is the Hub's own; the full story is at the source. Report a rights or accuracy concern