CUDA compute capability explained
CUDA compute capability is a version number that defines the hardware features and supported instructions of an NVIDIA GPU. It is written as X.Y and is used to determine which CUDA toolkit and software features a GPU can use. The version is listed by NVIDIA for each GPU model and affects compatibility with frameworks like PyTorch and LLM optimization techniques.
What compute capability is
Compute capability is a numeric version that identifies the architectural features and instruction set supported by an NVIDIA GPU, as described in NVIDIA's CUDA GPU table. Each GPU architecture receives a distinct compute capability number, such as 9.0 or 8.9.
The number consists of a major and a minor version. The major version indicates the base architecture, while the minor version reflects incremental feature additions within that architecture. A higher compute capability generally means newer instructions and hardware capabilities are available.
Compute capability is separate from the CUDA Toolkit version. The toolkit includes the compiler and libraries, while the compute capability describes the GPU's hardware capabilities that the toolkit can exploit.
How to find a GPU's compute capability
NVIDIA maintains a table listing the compute capability for each GPU model, covering data center, workstation, consumer, and Jetson products. Users can find their GPU by model name and read the corresponding compute capability from that table.
System tools such as nvidia-smi show the installed driver and CUDA runtime version but do not directly report compute capability. The model name is the primary identifier needed to look up the value.
The CUDA Installation Guide for Linux lists system requirements, including a CUDA-capable GPU, a supported Linux distribution, and a GCC toolchain. These prerequisites do not include a specific compute capability, but the GPU must be in NVIDIA's supported list.
Why compute capability matters for CUDA and PyTorch
CUDA applications often require a minimum compute capability to run certain kernels or use newer features. A GPU with an older compute capability may not support instructions that newer code relies on, so developers must set the appropriate architecture flags when compiling.
PyTorch offers installation commands that depend on the compute platform, including specific CUDA versions like 11.8 or 12.6, as shown on the PyTorch get started page. The selected CUDA version must be compatible with the GPU's compute capability for the compiled kernels to run correctly.
For Linux systems, PyTorch recommends using a CUDA-enabled GPU for full functionality, and the install command includes the CUDA version that the binaries were built for. Checking the compute capability helps ensure that the chosen platform matches the hardware.
Compute capability in LLM inference
Large language model inference is memory-intensive and often benefits from hardware features introduced in newer compute capabilities. The Hugging Face LLM optimization guide discusses lower precision formats and Flash Attention, which rely on specific GPU instruction support.
The guide provides a rule of thumb: loading a model with X billion parameters requires roughly 2 GB of VRAM per billion parameters when using bfloat16 or float16. This estimate helps determine whether a GPU's memory capacity is sufficient, and it applies across compute capabilities.
When a single GPU cannot hold the entire model, tensor parallelism across multiple GPUs is necessary. The compute capability of each GPU affects the available memory bandwidth and instruction set, which in turn influences the achieved inference throughput.
Inference benchmark suites such as MLPerf Inference Datacenter measure how quickly systems process inputs and produce results. Submissions use specific hardware and software stacks, and the compute capability of the accelerators is a key factor in system selection and optimization.
| GPU model | Compute capability |
|---|---|
| GeForce RTX 5090 | 12.0 |
| GeForce RTX 4090 | 8.9 |
| NVIDIA A100 | 8.0 |
| NVIDIA T4 | 7.5 |
| GeForce RTX 3080 | 8.6 |
| Jetson AGX Orin | 8.7 |
Frequently asked questions
What is CUDA compute capability?
CUDA compute capability is a version number that defines the hardware features and supported instructions for an NVIDIA GPU. It is written as X.Y, where the major version X indicates the base architecture and the minor version Y indicates incremental features.
How does one find a GPU's compute capability?
Look up the GPU model name in NVIDIA's CUDA GPU table, which lists compute capability for each data center, workstation, consumer, and Jetson GPU.
Does compute capability affect which PyTorch version to install?
Yes. PyTorch provides installation commands for different CUDA versions, and the selected CUDA version must be compatible with the GPU's compute capability for kernels to run correctly.
What happens if a GPU has a low compute capability?
A low compute capability may prevent the use of newer CUDA features, and some optimized kernels, such as those using lower precision or Flash Attention, might not be supported or run less efficiently.
Where can one see a list of supported GPUs?
NVIDIA publishes a table at the CUDA GPU page that lists compute capability for current and legacy GPUs, as well as a separate legacy table for older models.