Graph compiler

Notes

Graph compiler maturity is now the main reason a technically competitive NPU still loses benchmarks to an established GPU.

Overview

A graph compiler is a compiler that takes a neural network expressed as a graph of tensor operations and turns it into an efficient program for a particular processor, such as a GPU, NPU or CPU. It works at the level of whole layers and tensors, deciding how to combine operations, lay out data in memory and schedule work, rather than optimising individual scalar instructions. For an AI accelerator, the graph compiler is often what determines how much of the hardware's peak performance real models actually reach.

How it works

The compiler first imports a model from a framework such as PyTorch, JAX or TensorFlow, or from an exchange format, into its own intermediate representation. XLA, for example, accepts graphs in StableHLO, a versioned set of operations that acts as a portability layer between frameworks and the compiler, and converts them into its internal HLO form.

Target-independent passes then clean up and restructure the graph. Common subexpressions are merged, and operation fusion joins chains of operations so that intermediate tensors pass straight from one step to the next instead of being written to memory and read back. Buffer analysis plans where each tensor lives and reuses memory once a tensor is no longer needed.

Target-dependent passes use knowledge of the hardware: which operations map onto matrix engines, what tile sizes fit in on-chip memory, how to split work across cores or streams, and when to call a hand-tuned library routine instead of generating code. Finally each fused operation is lowered to machine code, often through LLVM for CPUs and GPUs or through a vendor's own backend for an NPU. TVM adds a learned cost model that searches automatically among possible low-level implementations.

The hard parts are coverage and tuning: every new operator in a popular model needs support, dynamic shapes complicate memory planning, and a fusion or tiling choice that suits one model can hurt another.

Examples

• XLA, the open-source compiler from the OpenXLA project, compiling JAX, PyTorch or TensorFlow models for CPUs, GPUs and TPUs.
• Apache TVM, which optimises models at graph and operator level and was shown targeting an FPGA-based deep learning accelerator.
• An NPU vendor's compiler that fuses a convolution, bias add and activation into one kernel that never leaves on-chip memory.
• XLA's GPU backend generating code for NVIDIA GPUs through LLVM's NVPTX backend.

Why it matters

AI chip companies often find that software, not silicon, decides whether customers can run their models fast. Graph and ML compiler engineers, NPU software architects and compiler QA teams build and test these tools, and hardware architects use compiler results to judge which features are worth adding.

In the chip lifecycle, a prototype compiler is used at architecture to evaluate design options on real workloads, it is developed against simulators and emulators before tapeout so software is ready for first silicon, and it is updated throughout production as new models and operators appear.

It matters most to: AI Compiler QA Engineer, AI Framework Integration Engineer, Graph Compiler Engineer, ML Compiler Engineer, NPU Software Architect

Common confusions

Not the same as a traditional compiler such as GCC

A conventional compiler optimises loops, registers and instructions in source code. A graph compiler works on tensor operations and often hands its final code generation to a conventional compiler backend such as LLVM.

Not the same as an inference runtime

A runtime loads a compiled model and executes it, managing devices and memory at run time. Some products combine both: NVIDIA describes the TensorRT library as an AI compiler and an inference runtime.

Sources

1. XLA architecture (opens in a new tab), OpenXLA project, 2026-10-22
2. XLA (Accelerated Linear Algebra) (opens in a new tab), OpenXLA project, 2026-10-22
3. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning (Chen et al., arXiv:1802.04799, 2018) (opens in a new tab), arXiv, 2026-10-22
4. NVIDIA TensorRT (opens in a new tab), NVIDIA Developer, 2026-10-22

Related terms

• TVM
• TensorRT
• ONNX
• LLVM

Where it appears on the Hub

Used on 5 pages of the Hub. In each group, the pages that mention it most come first.