Term · Embedded Software & Tools
Kernel library
A collection of hand-optimised compute routines for a specific processor or accelerator, covering convolutions, matrix multiplies, activations and reductions. Frameworks call into it rather than generating these inner loops themselves.
- Also known as
- Operator library, Primitive library
- Type
- Term
- Domain
- Embedded Software & Tools, AI enablement
Notes
Examples include cuDNN, oneDNN and CMSIS-NN; a missing kernel for a new operator can stall an entire model bring-up.
Overview
A kernel library is a collection of hand-tuned routines, called kernels, that perform the core mathematical operations of AI and numerical software, such as matrix multiplication, convolution, attention, normalisation and pooling, as fast as possible on one particular family of processors. AI frameworks and runtimes call these libraries rather than generating their own inner loops, so a chip's real-world performance depends heavily on how good its kernel library is. Well-known examples include NVIDIA's cuDNN, Intel's oneDNN and Arm's CMSIS-NN.
How it works
Each kernel implements one operation, often in many variants for different data types (32-bit and 16-bit floating point, 8-bit integers), tensor layouts and shapes. Library engineers write them with intimate knowledge of the hardware: the width of the vector or matrix units, the cache and register sizes, and the memory bandwidth. They split the work into tiles that fit the on-chip memory, arrange data so that loads are contiguous, and use special instructions such as Arm's Helium vector extension on Cortex-M. A library may hold several algorithms for the same operation and choose among them for each problem size.
Modern libraries go beyond single operations. NVIDIA describes cuDNN as supporting patterns that fuse several operations into one kernel, saving trips to memory, and oneDNN provides post-operations and quantisation support for int8 inference. At the small end, CMSIS-NN offers integer kernels designed to maximise speed and minimise memory use on Cortex-M microcontrollers, following the conventions of TensorFlow Lite for Microcontrollers so that the runtime can call it directly.
The idea is older than AI. The BLAS routines, defined in levels for vector, matrix-vector and matrix-matrix operations, gave numerical software a standard interface that each vendor could optimise for its own processors, and AI kernel libraries follow the same pattern.
Examples
- A deep learning framework calling cuDNN for convolution, matrix multiplication and attention on an NVIDIA GPU.
- oneDNN providing optimised int8 matrix multiplication for inference on Intel and Arm 64-bit server processors.
- TensorFlow Lite for Microcontrollers using CMSIS-NN kernels to run a quantised network quickly on a Cortex-M device.
- A team bringing up AI software on a new RISC-V processor using oneDNN, which lists RISC-V among its experimental targets.
Why it matters
Related terms
Part of the Embedded Software & Tools glossary, with every term in the topic and the lifecycles, tools and standards that use them.
Where it appears on the Hub
Used on 5 pages of the Hub. In each group, the pages that mention it most come first.
Lifecycle stages and gates
Suggest an improvement
Suggest an improvement to this entry
A clearer definition, an example, a correction or a source an editor can check. Editors review every suggestion before the entry changes.
Last reviewed . Written by The Semiconductor Hub editorial team. The full glossary