CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism

As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation-communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propo…

Source: arXiv cs.AR (Hardware Architecture), published 2026-11-05. The summary is the Hub's own; the full story is at the source. Report a rights or accuracy concern