Term · Compute & AI Hardware
Model parallelism
Splitting a single model across several accelerators because it does not fit on one, as opposed to replicating it. Tensor, pipeline and expert parallelism are the main forms, usually combined.
- Also known as
- Model sharding
- Type
- Term
- Domain
- Compute & AI Hardware, Inference and serving
Notes
Choosing the parallelism mix is a topology problem: each form imposes a different communication pattern on the fabric.
Related terms
Part of the Compute & AI Hardware glossary, with every term in the topic and the lifecycles, tools and standards that use them.
Where it appears on the Hub
Used on 1 page of the Hub. In each group, the pages that mention it most come first.
Suggest an improvement
Suggest an improvement to this entry
A clearer definition, an example, a correction or a source an editor can check. Editors review every suggestion before the entry changes.
Last reviewed: not yet reviewed in depth. Entry last updated . The full glossary