Compute & AI Hardware glossary

Compute and AI hardware terms describe the processors and systems that run software and machine learning: CPUs, GPUs and accelerators, their memory systems and interconnects, and the numbers used to compare them, such as throughput, utilisation and energy per operation.

Architects, system designers and buyers of compute use these terms to size a model to a chip, compare accelerators fairly, and see where memory bandwidth or networking limits performance rather than raw arithmetic.

100 terms in 13 groups, 37 with a full explanation and examples. Each term has its own page.

AI acceleration

Arithmetic intensity: The ratio of arithmetic operations performed to bytes moved from memory for a given computation.
Dataflow architecture: A design in which computation is triggered by the availability of data flowing through a spatial network of units, rather than by a program counter stepping through instructions in a shared memory.
MAC: Multiply-AccumulateIn depth: The fundamental operation of neural network and signal processing arithmetic, multiplying two values and adding the product to a running sum.
Matrix multiply unit: A hardware block that computes a small dense matrix product per instruction, amortising control and operand fetch over many multiply-accumulates.
PruningIn depth: Removing weights, channels or whole layers judged unimportant, then usually fine-tuning to recover accuracy.
Roofline model: A visual performance model that plots attainable throughput against arithmetic intensity, capped by a horizontal peak-compute roof and a sloping memory-bandwidth roof.
SparsityIn depth: The property of a tensor having many zero entries, which hardware can exploit by skipping the corresponding multiplications and by storing only non-zeros.
Systolic array: A grid of simple processing elements that pass operands rhythmically to their neighbours, so each datum is reused many times on chip.

AI workloads

AI: Artificial IntelligenceIn depth: Artificial intelligence (AI): software that learns patterns from data to classify, predict or generate, rather than following only hand-written rules.
LLM: Large Language ModelIn depth: Large language model (LLM): a neural network trained on large amounts of text to predict and generate language, used for chat assistants, code generation and document search.
ML: Machine LearningIn depth: Machine learning (ML): the part of artificial intelligence in which a model is trained on example data and then used to make predictions on new data.

Benchmarks and metrics

Geekbench: A cross-platform benchmark from Primate Labs producing single-core and multi-core scores from short real-world-inspired workloads, plus separate GPU compute and ML tests.
LINPACK: A benchmark that solves a dense system of linear equations in double precision, whose High Performance Linpack variant ranks supercomputers on the TOP500 list by achieved FLOPS.
MLPerfIn depth: A suite of machine learning benchmarks from MLCommons covering training, inference, storage and edge scenarios, with strict rules on submission divisions, accuracy targets and result review.
SPEC CPU: Standard Performance Evaluation Corporation CPU benchmarkIn depth: The industry-standard suite of compute-intensive integer and floating point workloads used to compare processor performance, reported as speed for single tasks and rate for throughput.
TFLOPS: Tera Floating-point Operations Per Second: Peak floating point throughput in trillions of operations per second, always tied to a precision such as FP64, FP16 or FP8.
TOPS/W: Tera Operations Per Second per Watt: Energy efficiency of an inference engine, expressed as integer throughput divided by power.

CPU architecture

AVX: Advanced Vector Extensions: Intel's family of x86 SIMD instruction sets, widening vectors from 128 to 256 bits and, in AVX-512, to 512 bits with masking and extra registers.
big.LITTLE: Arm's heterogeneous CPU arrangement pairing large high-performance cores with small energy-efficient cores of the same instruction set, letting the scheduler move each thread to whichever core suit...
Branch predictionIn depth: Hardware that guesses the direction and target of a conditional branch before it resolves, so the pipeline can keep fetching.
Cache hierarchyIn depth: The layered arrangement of small fast memories between cores and DRAM, typically private L1 and L2 per core plus a shared last level cache, which hides main memory latency by exploiting locality.
Core: One complete instruction-executing unit inside a processor, with its own fetch, decode, execution and usually private L1 and L2 caches.
Heterogeneous compute: Building a system from processing elements of different kinds, such as CPU, GPU, NPU and fixed-function blocks, and assigning each task to the unit that runs it most efficiently.
ISA: Instruction Set ArchitectureIn depth: The contract between hardware and software: the instructions, registers, memory model and exception behaviour a processor exposes.
NEON: Arm's fixed 128-bit SIMD instruction set, present on essentially all modern Arm application cores and the workhorse for mobile media and quantised neural network kernels.
RISC-VIn depth: An open, royalty-free instruction set architecture with a small base integer core and modular standard extensions, governed by RISC-V International.
RVV: RISC-V Vector extension: The standard vector extension to RISC-V, ratified as version 1.0, providing variable-length vector registers and vector-length agnostic code in the same spirit as Arm SVE.
SMT: Simultaneous MultithreadingIn depth: A technique that lets one physical core issue instructions from two or more hardware threads in the same cycle, filling execution slots that a single thread would leave idle on stalls.
Speculative execution: Running instructions whose necessity is not yet proven, typically past an unresolved branch, and discarding the results if the guess was wrong.
Superscalar: A processor design that issues more than one instruction per clock cycle by duplicating execution units and widening fetch and decode.
SVE: Scalable Vector Extension: Arm's vector instruction set in which the register width is left unspecified in the binary, so the same code runs on implementations from 128 to 2048 bits.
Thread: An independent stream of instructions with its own architectural state.
Vector extension: An optional set of instructions and wide registers added to a base instruction set to perform SIMD arithmetic.

Data centre systems

Accelerator tray: The serviceable chassis in a rack-scale system holding one or more accelerator modules with their cold plates, power delivery and fabric connectors, designed to be replaced without taking down the...
AI factory: A marketing and architectural framing of a data centre purpose-built for AI, where tokens rather than transactions are the output and the whole facility, from power to fabric, is designed as one ma...
Liquid cooling: Removing heat with a circulating fluid instead of air, usually through cold plates bolted to the hot components, with facility water carrying the heat away via a coolant distribution unit.
Power per rack: The electrical load a single rack draws, in kilowatts, which sets the busway, breaker and cooling design of the hall.
PUE: Power Usage Effectiveness: The ratio of a facility's total energy draw to the energy delivered to IT equipment.
Rack scale: Designing a whole rack as the unit of product, with shared power delivery, cooling and a fabric that makes dozens of accelerators behave as one machine, rather than assembling independent servers.
RDMA: Remote Direct Memory AccessIn depth: Remote direct memory access (RDMA): a network technique that lets one computer read or write another's memory without involving the remote processor, giving low latency and low overhead.
Superchip: NVIDIA's term for a module that packages a CPU and one or more GPUs together with a coherent high bandwidth link and shared memory address space, sold as a single component.
TDP: Thermal Design Power: The sustained heat output a cooling solution must handle for a given part, quoted in watts.

Edge AI

Always-on: A low-power hardware domain that stays awake continuously to monitor sensors and wake the main system only when something of interest occurs, typically operating at microwatt to low milliwatt levels.
Edge AIIn depth: Deploying inference at or near the point where data is generated, in cameras, vehicles, gateways and factory equipment, rather than in a central data centre.
Edge computing: Running compute and storage close to where data is produced or used, such as in roadside units, cellular base stations or nearby data centres, rather than in a distant cloud region.
Model compression: The umbrella set of techniques that shrink a trained model's size and compute cost, including quantisation, pruning, low-rank factorisation and distillation, usually with a small accuracy trade-off.
On-device inference: Running a model locally on the user's hardware instead of sending data to a server, which removes network latency, works offline and keeps personal data on the device.
TinyML: The practice of running machine learning models on microcontroller-class hardware within roughly a milliwatt of power and a few hundred kilobytes of memory, typically for keyword, gesture or anomal...
VPU: Vision Processing Unit: An accelerator tuned for computer vision pipelines, combining image signal processing, feature extraction and small neural network inference at very low power for cameras, drones and sensors.
Wake word detection: Continuously listening for a specific trigger phrase with a very small always-on model, so the larger speech pipeline only runs once the phrase is recognised.

Emerging compute

Analog in-memory computing: Using the physics of a resistive memory crossbar to compute a matrix-vector product in one step, where conductance encodes weights, voltage encodes inputs and Kirchhoff's law performs the summation.
Analog matrix multiply: Carrying out a matrix-vector multiplication with continuous physical quantities such as current or light intensity rather than digital arithmetic, trading exactness for very low energy per operation.
Compute-in-memoryIn depth: Performing arithmetic inside or immediately adjacent to the memory array, so weights never cross a narrow bus.
Neuromorphic computing: Hardware that mimics the event-driven, co-located memory and compute style of biological neurons, communicating with spikes and consuming energy only when activity occurs.
Photonic computing: Performing computation with light, typically by encoding values in optical amplitude or phase and using interferometer meshes or microring arrays to carry out linear algebra at very low latency.
SNN: Spiking Neural Network: A network model in which neurons communicate by discrete timed events rather than continuous activation values, so information is carried in spike timing and rate and computation is naturally sparse.

GPU architecture

CUDA core: NVIDIA's name for a single scalar arithmetic lane inside a streaming multiprocessor, capable of one floating point or integer operation per cycle.
KernelIn depth: In GPU programming, a function launched across a grid of many threads on the device.
Ray tracing core: A fixed-function GPU unit that accelerates bounding volume hierarchy traversal and ray-triangle intersection tests, the two dominant costs in real-time ray traced rendering.
ShaderIn depth: A small program executed per vertex, pixel or work item on a GPU's programmable units.
Streaming multiprocessor: The repeated execution block of an NVIDIA GPU, containing arithmetic lanes, tensor cores, register file, shared memory and warp schedulers.
Tensor Core: A fixed-function unit inside NVIDIA streaming multiprocessors that performs a small matrix multiply-accumulate per instruction at reduced precision, providing most of the chip's AI throughput.
VRAM: Video Random Access MemoryIn depth: The memory attached directly to a GPU, historically GDDR and now HBM on data centre parts.
Warp: A group of 32 GPU threads that execute the same instruction in lockstep on an NVIDIA architecture.

Inference and serving

BatchingIn depth: Grouping several inputs so one pass through the weights serves them all, amortising weight reads across requests.
Decode: The autoregressive phase of language model inference that emits one token at a time, reading the whole KV cache and all weights each step.
InferenceIn depth: Running a trained model on new inputs to produce outputs.
KV cache: Key-Value cache: The stored per-layer key and value tensors for tokens already processed, kept so each new token attends to the context without recomputing it.
Model parallelism: Splitting a single model across several accelerators because it does not fit on one, as opposed to replicating it.
Prefill: The first phase of language model inference, processing the entire input prompt in parallel to populate the KV cache and produce the first output token.
Tensor parallelism: Splitting individual weight matrices across accelerators so each computes part of every layer, requiring an all-reduce at layer boundaries.
TrainingIn depth: The process of fitting model parameters by repeated forward and backward passes over data with gradient updates.

Interconnect and fabric

Collective communicationIn depth: Coordinated data exchange patterns across a group of processes, such as all-reduce, all-gather, reduce-scatter and all-to-all.
InfiniBandIn depth: A switched fabric with native RDMA, credit-based flow control and very low latency, long the default back-end network for HPC clusters and large AI training deployments.
Interconnect fabric: The switched network of links tying accelerators, CPUs and storage into a system, characterised by its topology, bisection bandwidth and latency.
Scale-outIn depth: Adding capacity by connecting many independent nodes over a switched network such as InfiniBand or Ethernet.
Scale-upIn depth: Enlarging a single coherent compute domain by connecting more accelerators over a short, very high bandwidth link so they behave as one large device.

Numerics and precision

BF16: Brain Floating Point 16: A 16-bit float with the same 8-bit exponent as FP32 but only 7 mantissa bits, so it covers the full FP32 dynamic range and can be converted by truncation.
FP16: 16-bit floating point: The IEEE 754 half-precision format with 5 exponent and 10 mantissa bits.
FP4: 4-bit floating point: A 4-bit float format, usually E2M1, applied with fine-grained microscaling blocks so each small group of values carries its own exponent.
FP8: 8-bit floating pointIn depth: An 8-bit float defined by OCP in two variants, E4M3 favouring precision and E5M2 favouring range, used with per-tensor scaling or, in the OCP MX formats, a shared scale per block of 32 values, for...
INT4: 4-bit integer: A 4-bit integer format used for aggressive weight-only quantisation of large language models, typically with group-wise scales.
INT8: 8-bit integerIn depth: An 8-bit signed integer format used with per-tensor or per-channel scale factors for neural network inference.
Mixed precision: Running different parts of a computation at different numeric precisions, typically multiplying in a narrow format while accumulating and keeping master weights in a wider one, to gain speed withou...
QuantisationIn depth: Representing model weights and activations in fewer bits than they were trained in, mapping floating point ranges onto integer or narrow float grids.

Processor classes

AcceleratorIn depth: Any specialised processor attached to a host system to run a class of work faster or more efficiently than a CPU can.
APU: Accelerated Processing Unit: AMD's term for a processor that combines CPU cores and a substantial integrated GPU on one package sharing a memory space, blurring the line between a desktop CPU and a discrete graphics part.
ASIC: Application-Specific Integrated CircuitIn depth: A chip designed and fabricated for one application or customer rather than sold as a general-purpose part.
CGRA: Coarse-Grained Reconfigurable Array: A reconfigurable fabric whose programmable elements are word-width arithmetic units and routing channels rather than single-bit lookup tables, giving far better area and power efficiency than an FP...
CPU: Central Processing UnitIn depth: The general-purpose processor that executes the main instruction stream of a system, optimised for low latency on branchy, serial control code rather than raw parallel throughput.
DPU: Data Processing UnitIn depth: A programmable network-attached processor that offloads networking, storage and security work from the host CPU, typically combining Arm cores, a NIC pipeline and hardware accelerators on one device.
DSP: Digital Signal ProcessorIn depth: A processor architected for streaming numerical signal work, with multiply-accumulate hardware, saturating arithmetic, circular buffers and deterministic timing.
GPGPU: General-Purpose computing on Graphics Processing Units: The use of graphics hardware for non-graphics numerical workloads, enabled by programmable shaders and later by dedicated compute APIs.
GPU: Graphics Processing UnitIn depth: A massively parallel processor built from many simple arithmetic lanes grouped into wide execution units, originally for rasterising graphics and now the dominant engine for training and serving ne...
IPU: Intelligence Processing Unit: Graphcore's accelerator architecture built from many small cores each with private on-chip SRAM, using bulk synchronous parallel execution instead of relying on external high bandwidth memory.
TPU: Tensor Processing Unit: Google's family of custom AI accelerators built around large systolic matrix multiply units and high bandwidth memory, deployed in pods with a dedicated inter-chip interconnect for large-scale trai...

Quantum computing

Cryo-CMOS: Cryogenic Complementary Metal-Oxide-SemiconductorIn depth: Conventional CMOS circuits characterised and designed to operate at cryogenic temperatures, used for qubit control and readout electronics placed inside the refrigerator to avoid thousands of room-...
Quantum error correction: Encoding one logical qubit across many noisy physical qubits and repeatedly measuring stabilisers to detect and correct errors without collapsing the encoded state.
Qubit: Quantum bitIn depth: The basic unit of quantum information, a two-level quantum system that can occupy any superposition of its basis states and become entangled with others.
Superconducting qubit: A qubit built from a superconducting circuit containing a Josephson junction, operated in a dilution refrigerator near 10 millikelvin and controlled with microwave pulses.

Missing a term? Search the full glossary for Compute & AI Hardware and suggest it there.