How much GPU memory local LLMs need

Updated October 20, 2026

Estimating GPU memory for a local LLM starts with the model's parameter count: roughly 2 GB of VRAM per billion parameters at bfloat16, and 4 GB at float32. Quantization to 4-bit or 8-bit can cut memory substantially, and context length adds overhead. Checking the GPU's memory and compute capability determines whether a model fits.

Estimating weight memory

An LLM's memory footprint is dominated by its weights, which are stored as tensors. According to the Hugging Face optimization guide, loading a model with X billion parameters requires roughly 4X GB of VRAM in float32 and 2X GB in bfloat16 or float16. This rule applies to inference with short inputs, where weight memory dominates.

The guide gives concrete examples: the 175-billion-parameter GPT3 needs about 350 GB at bfloat16, while Llama-2-70b needs 140 GB and Falcon-40b needs 80 GB. These figures illustrate how quickly memory scales with model size.

For longer sequences, the context adds to memory usage, but the weight memory is the first figure to check when evaluating a GPU. The same source notes that for inputs under 1024 tokens, the weight load is the primary cost.

Quantization reduces memory requirements

Quantization lowers the precision of weights to reduce memory. The llama.cpp project supports integer quantization from 1.5-bit to 8-bit, which reduces memory use and can speed up inference. Its description notes that quantization is available for faster inference and reduced memory use.

Common choices are 8-bit and 4-bit quantization; storing each weight in fewer bits shrinks the weight memory in proportion, though the exact reduction depends on the model and implementation. Lower precision trades a degree of model quality for a smaller footprint.

Many local LLM tools, including llama.cpp, allow running models that do not fully fit in VRAM by using CPU+GPU hybrid inference, as the project's page explains. This lets a user fit larger models than the GPU memory alone would allow.

Context length and runtime overhead

Beyond weights, inference requires memory for activations and the key-value cache, especially for long contexts. Hugging Face's guide mentions Flash Attention as a memory-efficient attention algorithm that reduces this overhead, but for short inputs the weight memory still dominates.

The MLPerf Inference: Datacenter benchmark suite defines scenarios and metrics for server-side inference, such as throughput and latency. These benchmarks help compare hardware under standardized loads, but they are not memory figures and should not be used directly for VRAM planning.

In practice, the memory needed for a given model grows with sequence length. A practical approach is to estimate the weight memory for the chosen precision, add a buffer for context, and verify against the GPU's total memory.

Matching GPUs to local model sizes

GPU specifications list compute capability, which defines the hardware features and supported instructions, as shown on NVIDIA's CUDA GPU page. Compute capability matters for selecting the right CUDA version and ensuring compatibility with libraries.

For local inference, the GPU's memory capacity is the key constraint. A practical estimate starts from the 2X rule for bfloat16 and scales it down for the chosen quantization. For guidance on selecting hardware, the graphics card specifications page covers both gaming and AI workloads, including memory comparisons.

Additionally, the PyTorch local install guide describes how to choose a CUDA version for an NVIDIA GPU, which is part of setting up a GPU-based LLM environment.

Frequently asked questions

How much VRAM is needed for a 7B model?

The memory requirement can be estimated using the 2X rule for bfloat16, but the exact figure depends on precision and quantization.

What is quantization?

Quantization reduces the numerical precision of weights, such as from 16-bit to 4-bit, cutting memory usage. llama.cpp supports 1.5-bit to 8-bit integer quantization.

Can a model run if it does not fit in GPU memory?

Yes, some tools like llama.cpp support CPU+GPU hybrid inference, offloading part of the model to system memory.

Does context length affect GPU memory?

Yes, longer sequences require more memory for activations and the key-value cache, but weight memory dominates for short inputs.

How to check NVIDIA GPU compute capability?

NVIDIA publishes a table of compute capability for current GPUs; the compute capability defines supported instructions and features.