ONNX (Open Neural Network Exchange)

Notes

The practical friction is opset coverage: an exporter can emit operators a given accelerator back end has never implemented.

Overview

ONNX (Open Neural Network Exchange) is an open file format and specification for describing trained machine learning models as a graph of standard operators, so that a model built in one framework can be run or compiled by other tools. Chip companies use it as a common entry point: a customer exports a model to ONNX, and the vendor's compiler or runtime maps it onto the vendor's NPU, GPU or CPU. ONNX is a graduate project of the LF AI and Data Foundation, run under open governance.

How it works

An ONNX model is a serialised protocol buffer file. At its centre is a graph: nodes, each naming an operator such as convolution, matrix multiply or softmax, connected by named tensors. The graph declares its inputs and outputs, and constant tensors hold the trained weights. Tensors are strongly typed, with element types ranging from 64-bit floats down to 8-bit and 4-bit formats, and ONNX performs no implicit type conversion.

Operators are grouped into versioned operator sets, called opsets. A model declares which opset it uses for each domain, and once an operator version is marked stable its meaning cannot change, so a model exported years ago still has a defined meaning. Vendors can add custom operators in their own domains. The format originally described only inference; later versions added optional training information, which inference-only tools can ignore.

ONNX itself is only the description. Runtimes and compilers do the work. ONNX Runtime, for example, uses execution providers to hand parts of the graph to hardware-specific libraries, such as CUDA or TensorRT on NVIDIA GPUs, OpenVINO on Intel hardware or QNN on Qualcomm devices, and runs any unsupported operators on the CPU. For a new accelerator, the practical question is which operators and opsets it supports, and how well.

Examples

• A PyTorch model exported to ONNX and then compiled by an NPU vendor's SDK for an edge device.
• ONNX Runtime running most of a model through the TensorRT execution provider on an NVIDIA GPU, with unsupported nodes falling back to the CPU.
• A laptop application running the same ONNX model through the OpenVINO execution provider on Intel hardware.
• An accelerator team adding a custom operator domain for a fused layer that its hardware supports natively.
• A model file inspected in a graph viewer such as Netron to check which operators and opset version it uses before porting.

Why it matters

Related terms

• Graph compiler
• Runtime delegate
• TensorRT
• Model zoo

Where it appears on the Hub

Used on 21 pages of the Hub. In each group, the pages that mention it most come first.