CUDA (Compute Unified Device Architecture)

Notes

Launched in 2007; the accumulated library and framework ecosystem around it is widely described as NVIDIA's real moat.

Overview

CUDA (Compute Unified Device Architecture) is NVIDIA's parallel computing platform and programming model for running general-purpose work on its GPUs, made up of language extensions, a compiler, a runtime and driver, and a large set of libraries and tools. Developers write kernels, functions that run as thousands of parallel threads on the GPU, and launch them from code running on the CPU. CUDA runs only on NVIDIA hardware, so code written for it must be ported before it can run on other vendors' accelerators.

How it works

CUDA treats a system as a host, the CPU with its own memory, and one or more devices, the GPUs with theirs. The application runs on the host, moves data to the device or uses unified memory, launches kernels and waits for them to finish.

A kernel launch creates a grid of thread blocks. Each block runs on one streaming multiprocessor (SM), where its threads can share fast on-chip memory and synchronise. Threads execute in groups of 32 called warps: all threads in a warp follow the same instruction stream, and lanes that take a different branch are masked off, which wastes capacity. Blocks may run in any order, which lets the same program scale from small GPUs to very large ones.

Memory runs from per-thread registers, through shared memory and caches on each SM and a shared L2 cache, to global memory in the GPU's attached DRAM, and much tuning is about keeping data close to the SMs. The toolchain compiles kernels through NVVM, an LLVM-based compiler, into PTX, a virtual instruction set designed to stay stable across GPU generations, which is then translated for the target GPU. Developers can write C++, Python or Fortran, or call NVIDIA's tuned libraries, as frameworks such as PyTorch do.

Examples

• A PyTorch training job whose matrix multiplications and convolutions run as CUDA kernels from NVIDIA's libraries.
• A GPU kernel engineer rewriting an attention kernel to use shared memory and avoid warp divergence.
• An application compiled to PTX so that the driver can translate it for GPU generations released later.
• A team porting CUDA code to AMD GPUs with HIP, whose API mirrors CUDA's design.
• Nsight tools used to profile kernel performance on NVIDIA desktop and mobile targets.

Why it matters

Widely used AI frameworks such as PyTorch rely on CUDA libraries when they run on NVIDIA GPUs, so CUDA's libraries and tools shape what customers expect from any accelerator. GPU kernel engineers, library performance engineers and framework integration teams work in it directly, while competing chip companies must offer a comparable software stack, or tools to port CUDA code, to be considered.

Related terms

• ROCm
• OpenCL
• cuDNN
• BLAS
• GPGPU
• GPU

Where it appears on the Hub

Used on 24 pages of the Hub. In each group, the pages that mention it most come first.