Term · Compute & AI Hardware
Tensor parallelism
Splitting individual weight matrices across accelerators so each computes part of every layer, requiring an all-reduce at layer boundaries. It needs very high bandwidth and low latency links between the participating devices.
- Also known as
- TP, Intra-layer parallelism
- Type
- Term
- Domain
- Compute & AI Hardware, Inference and serving
Notes
Tensor parallelism is normally confined within one scale-up domain because it cannot tolerate network-class latency.
Related terms
Part of the Compute & AI Hardware glossary, with every term in the topic and the lifecycles, tools and standards that use them.
Where it appears on the Hub
No other page of the Hub uses this term yet.
Suggest an improvement
Suggest an improvement to this entry
A clearer definition, an example, a correction or a source an editor can check. Editors review every suggestion before the entry changes.
Last reviewed: not yet reviewed in depth. Entry last updated . The full glossary