TVM (Sometimes expanded as Tensor Virtual Machine, though the Apache TVM project does not expand the name itself)

Notes

Its autotuning approach searches thousands of loop schedules on real hardware, trading long compile times for hand-tuned-level performance.

Overview

Apache TVM is an open source machine learning compiler framework that takes a trained model from a framework such as PyTorch or ONNX and compiles it into an efficient module for a particular processor, GPU or accelerator. Chip companies use it to bring neural network support to new hardware without hand-writing every operator. TVM is a compiler stack, not a training framework: it optimises and deploys models that have already been trained elsewhere.

How it works

A model is first imported into TVM's central data structure, an IRModule, which can hold two kinds of function. Relax functions describe the model as a graph of operations, with support for control flow. TensorIR functions describe individual operators as low-level programs, with explicit loop nests, memory accesses, threads and vector or tensor instructions.

Compilation is a pipeline of passes. Graph-level passes fold constants, remove dead code and fuse neighbouring operators so that intermediate results stay in fast memory. Lower-level passes rewrite each operator's loops to suit the target, either by fixed rules or by search: TVM's MetaSchedule explores many candidate schedules and measures or predicts which runs fastest. The original research paper from the University of Washington reported results from this learning-based tuning that were competitive with hand-tuned libraries on a low-power CPU, a mobile GPU and server GPUs, and also targeted an FPGA-based accelerator.

Finally, the module is translated for the target: CPU code through LLVM, GPU code as CUDA or OpenCL source, and selected subgraphs handed to vendor libraries or custom code generators through a bring-your-own-codegen mechanism. The result is loaded by a small runtime that can run outside Python, including on mobile, edge and bare-metal devices. Tuning can take a long time, and supporting a new accelerator still needs real compiler work.

Examples

• An NPU start-up adding a custom code generator so that supported subgraphs of a model run on its accelerator while the rest falls back to the CPU.
• An edge AI team compiling an ONNX vision model with TVM to run on an Arm-based board without a Python interpreter.
• A compiler engineer running MetaSchedule to tune convolution operators for a new GPU before publishing benchmark results.
• A research group using TVM's TensorIR to try out new tensor instructions on a proposed accelerator design.

Why it matters

For a company building an AI accelerator or an NPU inside an SoC, software support often decides whether customers can use the hardware at all. ML compiler engineers use TVM, or build on its ideas, to map popular models onto new hardware, and deployment engineers use it to fit models on edge devices with tight memory.

Related terms

• Graph compiler
• ONNX
• Kernel library
• LLVM

Where it appears on the Hub

Used on 12 pages of the Hub. In each group, the pages that mention it most come first.