Process · Compute & AI Hardware
Training
The process of fitting model parameters by repeated forward and backward passes over data with gradient updates. It is throughput bound, tolerant of response latency though sensitive to communication latency between accelerators, and demands tight synchronisation across thousands of accelerators.
- Also known as
- Pre-training, Fine-tuning
- Type
- Process
- Domain
- Compute & AI Hardware, Inference and serving
Notes
A single large training run can occupy a cluster for months, which is why job restart and fault tolerance are first-order concerns.
Overview
Training is the process of teaching a neural network by showing it data, measuring how wrong its outputs are, and adjusting its parameters step by step to reduce that error. It produces the model that inference later runs, and for large models it is among the heaviest workloads data centre chips run. The hardware it needs differs from inference hardware: higher numerical precision, far more memory and very fast links between many accelerators.
How it works
Each training step has three parts. A forward pass runs a batch of examples through the network and computes a loss that measures the error. A backward pass applies the chain rule layer by layer to work out how much each parameter contributed to that loss, giving a gradient for every weight. An optimiser then updates the weights using those gradients. This repeats over many batches until the model reaches a target quality.
Training is also heavier on memory and precision than inference. Mixed precision eases the load: NVIDIA's guidance keeps a 32-bit master copy of the weights, runs forward and backward passes in 16-bit, and scales the loss so small gradients do not round to zero. Newer accelerators extend this to 8-bit floating point.
One chip is rarely enough. In data parallelism every accelerator holds a full copy of the model, processes different data, and averages gradients with an all-reduce; PyTorch's DistributedDataParallel overlaps that communication with the backward pass. When a model is too large for one device, model parallelism splits each layer, or the sequence of layers, across devices. Megatron-LM, for example, trained an 8.3-billion-parameter language model across 512 GPUs this way. All of these methods put heavy traffic on the interconnect, which is why training systems combine fast scale-up links inside a node with large scale-out networks between nodes.
Examples
- Pretraining a large language model on a large accelerator cluster, with gradients averaged across the cluster at every step.
- Fine-tuning an existing model on a smaller, task-specific dataset, which usually needs much less compute than training from scratch.
- MLPerf Training, which records the wall-clock time each submitted system needs to train reference models to a target quality.
- Mixed precision training with 16-bit Tensor Core arithmetic, 32-bit master weights and dynamic loss scaling.
Why it matters
Training cost and time decide how quickly AI developers can build new models, so data centre accelerators, memory, networking and power delivery are all designed with it in mind. Distributed training engineers, AI systems architects, collective communication developers and cluster infrastructure teams work on making large runs fast and reliable, and benchmarks such as MLPerf Training compare systems on time to reach a set quality.
Related terms
Part of the Compute & AI Hardware glossary, with every term in the topic and the lifecycles, tools and standards that use them.
Where it appears on the Hub
Used on 4 pages of the Hub. In each group, the pages that mention it most come first.
Suggest an improvement
Suggest an improvement to this entry
A clearer definition, an example, a correction or a source an editor can check. Editors review every suggestion before the entry changes.
Last reviewed . Written by The Semiconductor Hub editorial team. The full glossary