GPU memory is the high-bandwidth memory physically mounted on a graphics card. It stores everything the GPU needs to run AI models: weights, intermediate computations, and KV cache. Whether a model loads at all, and how fast it runs, depends on GPU memory capacity and bandwidth more than any other hardware specification.
If you opened Windows Task Manager and saw "Dedicated GPU Memory: 8.0GB" alongside "Shared GPU Memory: 16.0GB": dedicated is fast, on-board VRAM; shared is system RAM the OS reserves as a slower fallback. For gaming and light tasks, the distinction is academic. For running large language models, it is the difference between a working deployment and a CUDA out-of-memory crash.
What Is GPU Memory?
GPU memory is a pool of high-bandwidth memory on the GPU itself, separate from CPU system RAM. The GPU cannot work on data in system RAM directly; it must first load that data into its own memory. For AI workloads, every model weight, activation, and cached key-value pair lives in GPU memory during inference or training.
GPU Memory vs VRAM: Are They the Same?
GPU memory and VRAM are not the same, though the terms are often used interchangeably. VRAM (Video Random Access Memory) originally referred to memory designed for graphics tasks: frame buffers, texture maps, render targets. GPU memory is the broader category that includes VRAM plus on-chip memory like registers, shared memory (SMEM), and L1/L2 caches.
In AI, "my model needs 40GB of GPU memory" means VRAM: the large, off-chip pool listed on a GPU's spec sheet. The on-chip hierarchy plays a different role in how individual GPU operations execute.
The GPU Memory Hierarchy at a Glance
Each memory type in a GPU serves a different purpose, from registers used by individual threads to the large VRAM pool that holds model weights:
| Memory Type | Location | Typical Size | Latency | Who Controls It |
|---|---|---|---|---|
| Registers | On-chip, per thread | 256 KB per SM1 | ~1 cycle | Compiler |
| Shared Memory (SMEM) | On-chip, per SM | Up to 228 KB (H100) | ~20-30 cycles | Programmer (CUDA) |
| L1 / Texture Cache | On-chip, per SM | 32-128 KB per SM | ~30-50 cycles | Hardware |
| L2 Cache | On-chip, shared | 40-60 MB (H100) | ~200 cycles | Hardware |
| VRAM (HBM / GDDR) | Off-chip, on GPU board | 8-192GB | ~600-800 cycles | Developer / framework |
| System RAM (shared) | CPU motherboard | 32-512GB+ | Much higher + PCIe latency | OS |
| 1Streaming Multiprocessor |
Dedicated vs Shared GPU Memory
Dedicated GPU Memory (VRAM)
Dedicated GPU memory is VRAM physically mounted on the GPU board itself. It connects to the GPU's compute cores through a wide memory bus built for parallel access. Two memory technologies dominate modern GPU hardware:
GDDR (Graphics Double Data Rate) is used in consumer and mid-tier data center GPUs. GDDR6 and GDDR6X chips sit alongside the GPU die on the PCB, connected through a 256-bit or 384-bit memory bus. The RTX 5090 reaches 1,792GB/s; the RTX 4090 delivers 1,008GB/s. GDDR is cost-effective for inference on models that fit within 24–48GB.
HBM (High Bandwidth Memory) stacks memory dies vertically using through-silicon vias, creating an extremely wide memory interface in a compact footprint. The A100 80GB SXM reaches 2,039GB/s; the H100 SXM5 hits 3,350GB/s; and the H200's HBM3e achieves 4.8 TB/s. HBM's combination of high capacity and extreme bandwidth makes it the standard for large-scale AI training and production inference.
Shared GPU Memory: Integrated GPUs, Unified Memory, and Apple Silicon
Shared GPU memory is not a physical memory type. It is system RAM that the GPU accesses when it needs more memory than its VRAM provides.
Integrated GPUs have no dedicated VRAM. Instead, they use a portion of system RAM as their primary memory pool. This is what Task Manager reports as "Shared GPU Memory": up to 50% of installed RAM available for graphics use.
Discrete GPUs access shared memory through a different mechanism. CUDA Unified Memory (cudaMallocManaged) allows the NVIDIA driver to transparently move data between VRAM and system RAM when a VRAM allocation fails. In Apple Silicon's unified memory CPU and GPU share one physical pool at full bandwidth, unlike PCIe-based sharing where the bus itself is the bottleneck.
GPU Memory Bandwidth: Why the Gap Matters
Bandwidth, specifically how fast the GPU can read and write data, is the performance difference between dedicated and shared memory. Spilling from VRAM to system RAM is never a graceful degradation; the table below shows why.
| Memory Type | Example Hardware | Bandwidth | Typical Role |
|---|---|---|---|
| DDR5 (system RAM) | PC | 90GB/s | Shared GPU memory fallback |
| PCIe Gen5 x16 | CPU-to-GPU interconnect | 128GB/s bidirectional (~64GB/s per direction) | Data transfer bottleneck during spill |
| GDDR6 | NVIDIA L4 (24GB) | 300GB/s | Efficient inference, small models |
| Apple Unified Memory | M4 Max (128GB) | 546GB/s | Local inference on Mac hardware |
| GDDR6X | RTX 4090 (24GB) | 1,008GB/s | Development, small-scale inference |
| GDDR7 | RTX 5090 (32GB) | 1,792GB/s | High-end consumer inference |
| HBM2e | A100 80GB | 2,039GB/s | Training and inference (previous gen) |
| HBM3 | H100 SXM5 80GB | 3,350GB/s | Production training and inference |
| HBM3e | H200 141GB | 4,800GB/s | Large-model inference, long context |
| Bandwidth figures are approximate peak values from NVIDIA product specifications. Real-world sustained bandwidth during inference is typically 80–90% of peak. | |||
DDR5 system RAM to H200 HBM3e spans roughly 50x in bandwidth. When a model spills from VRAM to system RAM, the GPU's available bandwidth drops by that margin. LLM inference is almost entirely memory-bandwidth-bound at low batch sizes, so that gap maps directly to tokens/second.
What "Shared Memory" Means in CUDA
In CUDA programming, "shared memory" refers to the fast, on-chip scratchpad inside each streaming multiprocessor (SMEM in the table above). It is not shared with the CPU and has nothing to do with shared GPU memory as Windows Task Manager uses the phrase. In kernel optimization literature and CUDA code, "shared memory" means SMEM. In memory budget and model fit discussions, "shared GPU memory" means system RAM borrowed by the GPU. The two concepts are unrelated despite sharing a name.
What Actually Fills GPU Memory When You Run a Model
Three allocations fill VRAM during inference: model weights, the KV cache, and framework overhead. Understanding each one prevents OOM errors.
Model Weights
Model weights are the first and largest allocation. A rough formula:
Weight memory (GB) = num_parameters_B × bytes_per_parameter
At FP16 or BF16, each parameter takes 2 bytes. At INT8, 1 byte. At INT4 (common in quantized models), ~0.5 bytes.
That means a 70B model needs ~140GB for weights alone, but for that same model at INT4 the memory required drops to ~35GB. This is why quantization is usually the first tool applied when a model doesn't fit.
KV Cache
The KV (key-value) cache stores key and value vectors for every token at every layer during inference. It grows linearly with sequence length, batch size, and number of layers. A simplified estimate:
KV cache (GB) = 2 × batch_size × seq_len × num_layers × head_dim × num_heads × bytes_per_param / 1e9
For a 70B model at FP16 with 4K context and batch size 1, the KV cache is roughly 4–8GB. At 32K context, it grows to 30–50GB. In multi-turn chat applications, the KV cache often becomes the dominant VRAM consumer, exceeding the model weights themselves for long conversations. This is why a model that loads cleanly on short inputs OOMs when a user sends a long prompt or conversation history accumulates.
Modern attention optimizations reduce KV cache growth significantly: grouped-query attention (GQA), multi-head latent attention (MLA), and sparse attention variants like GLM-5.2's IndexShare all lower real-world KV cache usage below what the formula above predicts.
Framework and CUDA Overhead
Beyond weights and KV cache, the inference runtime consumes additional memory. CUDA kernels require workspace allocations. PyTorch's memory allocator holds a pool of pre-allocated blocks to avoid repeated cudaMalloc calls. vLLM, TensorRT-LLM, and other serving frameworks each add 0.5–2GB of overhead depending on the framework and GPU.
Why a 60GB Model Won't Reliably Run on an 80GB GPU
The 20GB headroom fills quickly. A 70B model at INT4 (35GB weights on a 48GB card) plus a 16K context window adds 15–20GB of KV cache, plus 1–2GB of framework overhead, putting total usage at ~52GB on a 48GB card. The general rule: add 15–20% on top of weights plus KV cache and treat the result as the real minimum VRAM requirement.
How to Measure Your GPU Memory Usage
Most writing on GPU memory describes what happens when VRAM runs out. These three approaches show you how to see it happening in real time.
nvidia-smi: What the Number Actually Means
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 1
This reports memory.used in MiB, refreshed every second. The number reflects total memory allocated by all GPU processes, including driver overhead. To log alongside inference for post-hoc analysis:
nvidia-smi dmon -s mu -d 1 > gpu_memory_log.csv &
python your_inference_script.py
kill %1
How to Tell When You Are Spilling to System RAM
On Linux with standard PyTorch, exceeding VRAM raises torch.cuda.OutOfMemoryError and terminates the process. There is no silent spill. Graceful degradation only occurs when a tool explicitly enables it. The most common cases:
llama.cpp --n-gpu-layers N: setting N below the model's total layer count offloads the remainder to CPU. Watch PCIe transfer activity during inference:
## Monitor PCIe transfer activity during inference
nvidia-smi dmon -s t -d 1
A sustained non-zero rxpci value during steady-state decode (not just model load) is the signature of active GPU-to-CPU data movement. If that number is climbing while generating tokens, layers are running on CPU.
For anyone using the accelerate python library, constrained dispatch achieves the same effect deliberately:
from accelerate import dispatch_model, infer_auto_device_map
device_map = infer_auto_device_map(
model,
max_memory={0: "20GiB", "cpu": "64GiB"}
)
model = dispatch_model(model, device_map=device_map)
What Happens When You Run Out: The Performance Cliff
Windows vs Linux
On Windows, the NVIDIA driver has a system-memory fallback policy since the 536.40 driver branch. When a VRAM allocation fails, the driver transparently pages data to system RAM over PCIe. The application keeps running at reduced throughput rather than crashing.
On Linux instance, a model that doesn't fit on VRAM crashes unless you explicitely resort to one of the offloading mechanisms.
When Linux Does Degrade Instead of Crashing
Four paths to graceful degradation exist on Linux:
llama.cpp layer offload via --n-gpu-layers is the most controllable. Each layer not on the GPU runs on CPU. Varying the flag lets you benchmark throughput impact precisely.
accelerate with device_map="auto" or a custom max_memory dict splits model layers across GPU and CPU memory. Dispatch overhead adds latency, especially when a single forward pass crosses the boundary frequently.
vLLM's cpu_offload_gb offloads a specified number of gigabytes of model weights to CPU, streaming them back to the GPU layer by layer during each forward pass. This allows models whose weights exceed VRAM to run at reduced throughput rather than failing to load.
CUDA Unified Memory via cudaMallocManaged allows the NVIDIA driver to transparently page between VRAM and system RAM, mirroring Windows behavior. This is rarely used in production inference frameworks but is the closest analogue to what Windows does automatically.
How Much GPU Memory Do You Need for LLM Inference?
The 2GB Per Billion Parameters Rule
A model requires ~2GB of VRAM per billion parameters at FP16 or BF16. Add 15–20% on top for KV cache at modest context lengths and framework overhead.
At INT4 quantization, that drops to ~0.5–0.6GB per billion parameters. A 7B model needs ~14–16GB at FP16, or 5–6GB at INT4. A 70B model needs ~140–160GB at FP16, or 38–45GB at INT4.
Context length changes this significantly. At 32K context, KV cache can double the effective VRAM requirement compared to 4K. Plan for the longest context you expect in production, not the average.
Model Size, VRAM Requirement, and GPU Options
| Model | Precision | Min. VRAM (weights + overhead) | Fits on | Thunder Compute pricing1 |
|---|---|---|---|---|
| Llama 3.1 8B | FP16 | ~18GB | RTX A6000 (48GB) | From $0.35/hr |
| Llama 3.3 70B | INT4 (Q4_K_M) | ~43GB | RTX A6000 (48GB) | From $0.35/hr |
| Llama 3.3 70B | FP16 | ~145GB | 2× A100 80GB | From $2.18/hr |
| Llama 4 Scout (109B total) | INT4 | ~70–75GB | A100 80GB (tight) | From $1.09/hr |
| DeepSeek R1 671B | FP8 | ~400GB+ | 8x H100 cluster | $23.12 |
| 1 Pricing as of July 2026. Subject to change. VRAM estimates assume 4K context and include ~15% overhead. Long-context workloads require additional headroom. See the complete GPU-to-model mapping guide for detailed per-model VRAM math. | ||||
Can You Add More GPU Memory?
Why You Can't Increase Dedicated GPU Memory
VRAM is soldered to the PCB on a discrete GPU, meaning the chips cannot be replaced or supplemented. BIOS options labeled "Increase Dedicated GPU Memory" re-allocate a small amount of system RAM into a buffer for certain overhead but do not add to the VRAM available for model weights. The effective increase is negligible; no BIOS configuration will let a 24GB RTX 4090 load a 70B FP16 model.
Quantization, Offloading, and Multi-GPU: What Each Buys You
Three practical approaches exist when your model does not fit:
Quantization reduces precision to compress weights. INT4 quantization (GPTQ, AWQ, Q4_K_M) cuts VRAM usage by ~75% vs FP16 with manageable quality loss for most inference tasks. This is the first thing to try when 5–20GB over budget.
CPU/disk offloading using llama.cpp, accelerate, or vLLM keeps the model running when quantization is not enough. Every layer running on CPU rather than GPU measurably reduces tokens/second. It is appropriate for development, but not for production.
Multi-GPU with tensor parallelism distributes the model across multiple GPUs, each contributing their full VRAM. Two A100 80GB cards provide 160GB of addressable VRAM. NVLink-connected multi-GPU setups communicate at 600GB/s (A100 NVLink 3.0) or 900GB/s (H100 NVLink 4.0), fast enough that the model behaves nearly as if it resided on a single large GPU. PCIe-connected multi-GPU is slower and better suited for pipeline parallelism.
When Renting Is the Right Answer
Buying a GPU with more VRAM is an investement, and locks you into a capacity decision you may outgrow six months from now. Renting a cloud GPU by the hour costs a fraction of the hardware price, is available in minutes, and scales up or down in minutes.
An H100 80GB at $2.19/hr on Thunder Compute provides 80GB of HBM3 at 3,350GB/s. A new H100 card costs $25,000–$40,000 depending on form factor. At $2.19/hr, the breakeven point for a $25,000 card is ~11,400 hours of continuous use, around 16 months, before rental cost exceeds purchase price. That figure excludes server chassis, power, and cooling.
Getting Full Dedicated VRAM on a Cloud GPU
MIG, vGPU, and Overcommitment: What to Check Before You Rent
Not all cloud GPU instances provide the full card. Three mechanisms reduce effective VRAM without prominent disclosure:
MIG (Multi-Instance GPU) partitions a single physical GPU into up to seven isolated slices, each with its own dedicated HBM allocation. A 1g.10gb MIG instance on an H100 provides ~10GB of HBM, not 80GB. If a listing says "H100" without specifying the partition, confirm whether it is a full card or a MIG slice.
vGPU uses hypervisor-level virtualization to share a single physical GPU across multiple tenants. The "GPU memory" shown may include a shared pool that degrades under contention, or may overcommit the physical card's capacity across instances. Performance looks acceptable in isolation but degrades when neighbor tenants run heavy workloads.
Background process overhead from hypervisor agents, monitoring, and drivers typically reserves 500MB-2GB of VRAM that appears available but is not accessible to your workload.
To verify what you actually have, run nvidia-smi immediately after provisioning and compare reported VRAM to the card's published spec. A full RTX A6000 reports 49,152 MiB. An H100 SXM5 reports 81,920 MiB. A smaller number means a partitioned or shared instance.
Thunder Compute GPU Memory and Pricing
Thunder Compute instances provide access to dedicated GPU hardware. The VRAM figure listed for each GPU type reflects the physical card's full specification.
| GPU | VRAM | Memory Bandwidth | Best For | Starting Price1 |
|---|---|---|---|---|
| RTX A6000 | 48GB GDDR6 | 768GB/s | 7B–70B inference (quantized), image generation | $0.35/hr |
| A100 80GB | 80GB HBM2e | 2,039GB/s | 70B inference, fine-tuning (LoRA/QLoRA) | $1.69/hr |
| H100 SXM5 | 80GB HBM3 | 3,350GB/s | Production inference, training, long context | $2.19/hr |
Going from Zero to a Running Instance in VS Code or Cursor
Thunder Compute connects directly to your editor once you select an instance. The VS Code and Cursor extensions open a remote session on the GPU instance as you would any SSH target, with the GPU available to your training or inference script without additional configuration. One-click templates for ComfyUI, Stable Diffusion, and common LLM serving stacks are available for pre-configured environments.
Last Thoughts on GPU Memory
GPU memory capacity and bandwidth determine whether your AI workload runs, how fast it runs, and whether it stays running under production load. The core rule is 2GB of VRAM per billion parameters at FP16, adjusted for quantization and context length. When that number exceeds your hardware, quantization comes first, offloading second, and additional capacity third. For most workloads, renting a cloud GPU with the right VRAM costs less and takes less time than sizing hardware upfront.
FAQ
Is GPU memory the same as VRAM?
Not exactly. VRAM refers to the high-bandwidth memory chips on a discrete GPU card. GPU memory is the broader term that includes VRAM plus on-chip memory like registers, shared memory, and caches. In practice, the two terms are used interchangeably in AI contexts.
What is the difference between dedicated and shared GPU memory?
Dedicated GPU memory (VRAM) is physically mounted on the GPU card and delivers 300–4,800GB/s of bandwidth. Shared GPU memory is system RAM the OS reserves for GPU use; it reaches at most ~64GB/s over PCIe. For AI workloads, dedicated VRAM is required.
Can I increase my dedicated GPU memory?
No. VRAM capacity is fixed in hardware. BIOS options that claim to increase GPU memory on discrete cards adjust a small system-RAM buffer with negligible effect. To get moreout of your VRAM, use quantization, CPU offloading, or else rent a cloud GPU with the capacity you need.
Why does my model run out of VRAM even though the weights fit?
The KV cache grows during inference as the GPU stores key and value vectors for each token at each transformer layer. For long prompts or multi-turn conversations, the KV cache can exceed the model weights in size. Plan VRAM requirements around your maximum context and batch size, not just weight footprint.
Why does nvidia-smi show more memory used than my code allocated?
PyTorch's caching allocator holds freed memory in an internal pool rather than returning it to the OS immediately. The gap between nvidia-smi and torch.cuda.memory_allocated() is this cache, not a leak. Call torch.cuda.empty_cache() to return the pool to the OS when another process needs the memory.
Is Apple Silicon unified memory the same as shared GPU memory?
No. Apple Silicon unified memory gives CPU and GPU shared access at up to 546GB/s on the M4 Max. Traditional shared GPU memory accesses system RAM over PCIe at ~64GB/s. Unified memory is much faster than PCIe sharing but slower than data center HBM (2,000–4,800GB/s).
On Linux, does exceeding VRAM cause a gradual slowdown or an immediate crash?
On Linux with standard PyTorch, exceeding VRAM raises torch.cuda.OutOfMemoryError and terminates the process immediately. There is no silent spill. Graceful degradation requires an explicit offloading tool such as llama.cpp --n-gpu-layers, accelerate device_map, or vLLM cpu_offload_gb.