GPU architecture dictates how your hardware schedules threads, moves data, and executes matrix operations on every training and inference job. Understanding it helps you debug memory bottlenecks faster, choose the right hardware, and make sense of new generations.
This guide covers NVIDIA GPU architecture from the ground up:
- The physical components inside the chip
- How Ampere, Hopper, and Blackwell differ
- What the upcoming Vera Rubin platform changes for AI workloads
What GPU Architecture Actually Is
GPU architecture determines how a chip's compute units, memory systems, and execution pipelines are organized. It governs parallelism, precision support, memory bandwidth, and how efficiently the chip executes matrix multiplications, attention computations, and large memory transfers.
The term covers two layers:
- Physical microarchitecture - how transistors are arranged, what execution units exist, and how they connect.
- Programming model - how software exposes the hardware to developers. For NVIDIA, that model is CUDA which is supported on all its GPUs since 2007.
Why GPU Architecture Differs from CPU Architecture
GPUs and CPUs are architecturally opposite: CPUs minimize latency, GPUs maximize throughput. CPUs achieve low latency through a handful of powerful cores with deep out-of-order execution, large caches, and branch prediction logic. GPUs achieve high throughput by trading per-core intelligence for thousands of simpler cores that execute tens of thousands of calculations simultaneously.
For context, a typical consumer CPU has 8–16 cores, while a comparable consumer GPU like the RTX 4090 has 16,384.
| Dimension | CPU (e.g., AMD EPYC 9654) | GPU (e.g., NVIDIA B200) |
|---|---|---|
| Core count | 96 high-performance cores | 20,480 CUDA Cores |
| Execution model | Sequential, low latency | Parallelism, high throughput |
| GPU Memory | Up to ~12TB system RAM | 192GB HBM3e (B200) |
| Memory bandwidth | ~461GB/s (8-channel DDR5) | 8TB/s (B200) |
| Best workload | Complex branching logic, OS tasks | Dense matrix math, parallel transforms |
High bandwidth makes GPUs indispensable for AI. A transformer model consists almost entirely of matrix multiplications and memory reads. At 8TB/s, a B200 feeds its cores fast enough to keep them busy at any scale. A CPU cannot come close.
For a comparison of how TPUs fit alongside GPUs in AI infrastructure, see our TPU vs GPU comparison.
The Building Blocks of NVIDIA GPU Architecture
NVIDIA organizes each GPU in a strict hierarchy. Understanding it from top to bottom makes spec sheets readable and architectural differences between generations legible.
Graphics Processing Clusters
A Graphics Processing Cluster (GPC) is the top-level organizational unit inside an NVIDIA GPU. Each GPC operates semi-independently, containing its own scheduling logic, a set of Texture Processing Clusters (TPCs), and a cluster-level L1 cache.
GPCs determine how work is distributed across the chip. When batch size and sequence length are large enough to saturate all GPCs simultaneously, the chip is being used efficiently.
For consumer and workstation GPUs, NVIDIA scales its designs by enabling more or fewer GPCs on the same die: the RTX 3050 and RTX 3090 are both built on Ampere architecture but they enable 2 and 7 GPCs respectively.
Disabling GPCs is partly a manufacturing outcome: dies that come out of fabrication with a defective GPC get disabled and sold as lower-tier products rather than discarded. NVIDIA also intentionally disables GPCs on fully functional dies to fill out its product lineup.
Streaming Multiprocessors
The Streaming Multiprocessor (SM) is the fundamental execution unit of an NVIDIA GPU. Each SM contains CUDA Cores, Tensor Cores, a warp scheduler, register files, and a configurable shared memory block. All actual computation happens inside an SM.
SM count meaningfully describes a GPU's raw compute capacity. The A100 has 108 SMs, the H100 SXM has 132, and the B200 has 160. More SMs means more parallelism, but only if your workload generates enough threads to keep them occupied.
Each SM executes several groups of 32 threads called "warps". The warp scheduler selects ready warps and issues instructions to the CUDA Cores or Tensor Cores inside the SM. Because memory latency is high relative to computation speed, the SM keeps many warps in flight at once. When one warp stalls on a memory fetch, the scheduler switches to another that is ready. This latency-hiding through warp switching is what makes the GPU's throughput model work.
CUDA Cores vs Tensor Cores vs RT Cores
CUDA Cores, Tensor Cores, and RT Cores are the three execution unit types inside each SM, and each serves a different purpose.
CUDA Cores perform general-purpose floating-point and integer arithmetic. They execute one operation per clock cycle per core. For AI workloads, CUDA Cores handle everything that does not map to matrix multiplication: activation functions, normalization, data preprocessing, and control flow.
Tensor Cores perform matrix multiplication through a fused multiply-accumulate on small matrix tiles in a single clock cycle. Where a CUDA Core computes one multiply-add, a Tensor Core computes a 4x4x4 matrix operation in the same time. The Tensor Core generation is the most important architectural differentiator between GPU generations for AI.
RT Cores accelerate ray-tracing for real-time graphics rendering. They are irrelevant for AI training, inference, and AI-based image and video generation workloads, so ignore them when evaluating GPUs for compute.
For a deep dive on Tensor Core generations and training throughput, see Tensor Cores Explained.
GPU Memory Hierarchy: Where Most Performance Is Lost
Compute units in an NVIDIA GPU execute operations faster than memory can supply data. The memory hierarchy bridges the gap, and understanding it is the key to diagnosing memory-bound training runs and making informed VRAM decisions.
On-Chip Memory: Registers, Shared Memory, and L1/L2 Cache
Registers are the fastest memory on the chip. Each thread gets a private register file with zero-latency access. Registers are finite: the H100 SM has 65,536 32-bit registers shared across all threads running on that SM simultaneously.
Shared memory is a user-controlled scratchpad that all threads within a block can read and write. It sits inside the SM and has very low latency (roughly 20-30 clock cycles). In CUDA kernels, shared memory stages data for reuse across threads, avoiding repeated round-trips to slower global memory. An H100 SM has 228 KB of configurable L1/shared memory, and getting this right is a primary lever for hand-tuned kernel performance.
L2 cache is shared across all SMs and serves as the primary buffer between the SMs and global DRAM. The H100 has 50 MB of L2. Blackwell's GB200 raises L2 to 126 MB, roughly 2.5x the H100. A larger L2 keeps model weights that are reused across batches warm in cache rather than re-fetching them from HBM on every forward pass.
Off-Chip Memory: GDDR6 vs HBM
GDDR6X is used in consumer and workstation GPU lines like the RTX 40 and 50 series. It connects to the GPU die via a standard memory interface. An RTX 4090 has 24GB of GDDR6X at around 1TB/s bandwidth, fast enough for inference on medium-size models and fine-tuning with QLoRA, but the bandwidth ceiling becomes a bottleneck on large-batch training.
HBM (High Bandwidth Memory) stacks DRAM dies vertically and connects them to the GPU via a silicon interposer, dramatically shortening the electrical path. The H100 SXM uses HBM3 at 3.35TB/s, more than three times the RTX 4090. HBM also provides far more capacity: 80GB on the H100 vs 24GB on the RTX 4090. For LLM training, both matter independently: capacity holds the model and optimizer states, bandwidth feeds the Tensor Cores fast enough to maintain high utilization.
| Memory Type | Representative GPU | Capacity | Bandwidth | Best For |
|---|---|---|---|---|
| GDDR6X | RTX 4090 | 24GB | ~1TB/s | Inference, fine-tuning small models |
| GDDR7 | RTX 5090 | 32GB | ~1.8TB/s | Inference, consumer-grade training |
| HBM2e | A100 80GB | 80GB | 2TB/s | LLM training, large-batch inference |
| HBM3 | H100 SXM | 80GB | 3.35TB/s | Transformer training, inference at scale |
| HBM3e | H200 SXM | 141GB | 4.8TB/s | Frontier LLM training, long-context inference |
| HBM3e | B200 | 192GB | 8TB/s | Multi-trillion parameter training and inference |
| HBM4 | Rubin GPU (H2 2026) | 288GB | 22TB/s | Next-generation agentic AI and MoE workloads |
| Rubin specifications are preliminary. Final HBM4 bandwidth figures have not been officially confirmed as of publication. | ||||
How GPUs Execute Work: SIMT, Warps, and Thread Hierarchy
GPUs execute one instruction across many data elements simultaneously. NVIDIA uses a model called SIMT: Single Instruction, Multiple Threads.
SIMT vs SIMD
SIMD (Single Instruction, Multiple Data) is the CPU's approach to parallelism: one operation applies to a vector of data elements in a single clock cycle, and the programmer packs and manages that vector directly.
SIMT (Single Instruction, Multiple Threads) is NVIDIA's variation on the same idea, and it moves the vector packing from the programmer to the hardware. The programmer writes a scalar kernel from the perspective of one thread, and the GPU runs that instruction across a warp of 32 threads in lockstep. The 32 threads are effectively the vector lanes, so the hardware is still executing SIMD-style underneath, but the programmer never packs a vector by hand.
The abstraction comes with a cost. If threads within the same warp take different paths through a conditional branch, the GPU executes both paths sequentially with the inactive threads masked off. This is warp divergence, and it is a common source of underperformance in non-trivial kernels.
The Thread Hierarchy
CUDA organizes threads into a hierarchy that maps directly onto the GPU's hardware:
- Threads are the base unit, each running the kernel on its own data.
- Warps group 32 threads together. The warp scheduler issues instructions to all 32 simultaneously.
- Thread blocks group warps together. Threads within a block share the SM's local memory and can synchronize.
- Grids group thread blocks together, representing the full kernel launch.
When you launch a kernel, the grid splits into thread blocks distributed across SMs. Each SM holds multiple active blocks at once, limited by its register and shared memory capacity. The more warps an SM keeps in flight, the better it hides memory latency by switching between them when one stalls. That ratio of active to maximum possible warps is called occupancy, and it's one of the main things kernel developers tune for.
NVIDIA GPU Architecture Generations: What Changed and Why It Matters for AI
NVIDIA releases a new GPU architecture roughly every two years. The most relevant changes for AI practitioners are in Tensor Core generation, supported precision formats, memory bandwidth, and the software features that enable efficient transformer training.
Ampere (A100, A10): The LLM Training Baseline
Launched in 2020, Ampere made large-scale LLM training practical. The A100 introduced third-generation Tensor Cores with TF32 and BF16 support, structural 2:4 sparsity (which could double inference throughput on compatible models), and Multi-Instance GPU (MIG), allowing a single A100 to be partitioned into up to seven isolated instances. NVLink 3.0 provided 600GB/s bidirectional bandwidth between GPUs in a node.
The A100 remains widely available and cost-effective for medium-scale workloads in 2026.
Hopper (H100, H200): The Transformer Era Architecture
Launched in 2022, Hopper was designed specifically for transformer workloads. Its most significant innovation is the Transformer Engine: a hardware unit that dynamically switches between FP8 and BF16 precision at the layer level, enabling faster computation without meaningful accuracy loss.
The H100 introduced fourth-generation Tensor Cores with FP8 support, HBM3, and NVLink 4.0 at 900GB/s. The H200 followed with HBM3e and 141GB of capacity, making it the preferred choice for long-context inference.
The H100 remains the dominant GPU for production AI training in mid-2026. Its Tensor Core performance, memory bandwidth, and broad support across PyTorch, JAX, and inference frameworks make it the safest default for most workloads.
Ada Lovelace (L40, RTX 4090): Inference and Edge
Ada Lovelace is NVIDIA's consumer and workstation line running parallel to Hopper. It uses GDDR6X rather than HBM, meaning lower bandwidth and capacity.
The L40 is the data center variant and has become a popular inference GPU at a lower per-hour cost than H100 instances. The RTX 4090 is widely used for fine-tuning smaller models locally. Neither is suited for large-scale LLM training where HBM bandwidth is the limiting factor.
Blackwell (B200, B300, GB200): Generative AI at Scale
Launched in 2024, Blackwell is the current flagship data center architecture. It introduces:
- 5th gen Tensor Cores with FP4 and FP6 precision support
- A dual-die design connecting two reticle-limited chiplets via a 10TB/s chip-to-chip interconnect to form a single logical GPU with 208B transistors
- 192GB of HBM3e at 8TB/s
- NVLink 5 at 1.8TB/s per GPU.
The GB200 superchip pairs two B200 GPUs with a Grace CPU via NVLink-C2C at 900GB/s, eliminating the PCIe bottleneck for CPU-GPU data transfer.
At FP4 precision, NVIDIA claims up to 25x lower inference cost compared to Hopper at FP16. The 192GB HBM3e capacity per B200 GPU and NVLink 5 interconnect also allow training much larger models per node before requiring inter-node communication. The GB200 NVL72 rack-scale system connects 72 B200 GPUs and 36 Grace CPUs into a unified system, blurring the boundary between a GPU cluster and a single logical accelerator.
What's Next: Vera Rubin (H2 2026)
Vera Rubin is NVIDIA's next GPU architecture, now in full production following Jensen Huang's announcement at CES 2026. The platform takes a substantial step from Blackwell:
- Next gen Tensor Cores delivering 50 PFLOPS of FP4 compute.
- A dual-die TSMC 3nm design with 336B transistors on (a full node shrink from Blackwell's 4NP)
- 288GB of HBM4 memory at 22TB/s
- NVLink 6
NVIDIA claims 3.5x training throughput and up to 5x inference throughput improvement over Blackwell for MoE models.
The Vera Rubin Superchip combines two Rubin GPUs with a Vera ARM CPU (88 Olympus cores) in a single package connected by NVLink-C2C at 1.8TB/s. The NVL72 rack, combining 72 Rubin GPUs, is already shipping to hyperscaler partners. Broader enterprise cloud access is expected by mid-2027.
See our dedicated Vera Rubin architecture post for full specs, performance data, and availability timelines.
GPU Architecture Generation Comparison
| Architecture | Year | Baseline GPU | Tensor Core Gen | Best Precision | VRAM | Memory BW |
|---|---|---|---|---|---|---|
| Volta | 2017 | V100 | Gen 1 | FP16 | 32GB HBM2 | 900GB/s |
| Ampere | 2020 | A100 | Gen 3 | TF32, BF16 | 80GB HBM2e | 2TB/s |
| Hopper | 2022 | H100 | Gen 4 | FP8 | 80GB HBM3 | 3.35TB/s |
| Blackwell | 2024 | B200 | Gen 5 | FP4, FP6 | 192GB HBM3e | 8TB/s |
| Rubin | 2026 | Rubin GPU | Gen 6 | FP4 (NVFP4) | 288GB HBM4 | 22TB/s |
| Rubin specifications are based on official NVIDIA announcements and are subject to change. | ||||||
NVIDIA vs AMD GPU Architecture for AI
NVIDIA dominates AI training infrastructure in 2026, but AMD has a meaningful presence where ROCm compatibility and cost-per-FLOP are priorities. Understanding the architectural differences matters when cloud catalogs offer both.
CDNA vs CUDA Architecture
NVIDIA's data center GPU architecture is based on the CUDA compute paradigm: a unified SM-based design that runs the same kernel code across graphics and compute workloads. AMD separates its architectures. RDNA (Radeon) is the consumer and workstation line, optimized for graphics. CDNA (Compute DNA) is AMD's data center architecture, purpose-built for AI and HPC with no graphics-rendering hardware. The MI300X is a CDNA 3 chip containing no display engines, RT Cores, or gaming-oriented features.
The MI300X features 192GB of HBM3 at 5.3TB/s bandwidth, more capacity than an H100 and more bandwidth than an H200. For inference serving of large models that barely fit in VRAM, this capacity advantage is meaningful. AMD's Infinity Fabric connects multiple GPU dies within the MI300X's multi-chiplet design (eight compute dies, eight HBM stacks).
The central limitation is software. CUDA has a two-decade head start and is deeply integrated into PyTorch, TensorFlow, JAX, and virtually every production ML framework. ROCm has made significant progress but still requires more porting work than simply picking a different cloud instance.
See our ROCm vs CUDA comparison for a detailed comparison of both ecosystems, including compatibility, framework support, and migration considerations.
Multi-GPU Scaling: NVLink and How Distributed Training Works
Single-GPU training has practical limits. Once a model does not fit in one GPU's VRAM, or once single-GPU throughput is too slow for your timeline, you need multi-GPU setups. The interconnect architecture between GPUs determines how efficiently they cooperate.
NVLink and NVLink Switch
NVLink is NVIDIA's proprietary high-speed GPU-to-GPU interconnect that bypasses PCIe for direct GPU-to-GPU communication.
NVLink 4.0 (Hopper) provides 900GB/s bidirectional bandwidth per GPU. NVLink 5.0 (Blackwell) doubles that to 1.8TB/s.
NVLink Switch extends NVLink from a single node up to a full rack. NVSwitch began as an intra-node crossbar connecting the 8 GPUs on a baseboard, and Blackwell's rack-level NVLink Switch chip now links all 72 GPUs in a GB200 NVL72 or Rubin NVL72 rack into one non-blocking fabric where every GPU reaches every other at full bandwidth. This is what makes an NVL72 rack function as a single logical accelerator rather than a collection of discrete servers.
Choosing a GPU Architecture for Your AI Workload
Choosing the right architecture means matching its strengths to your workload's bottlenecks, not finding the most powerful GPU on paper.
The four dimensions that determine fit are VRAM capacity (your model and optimizer states must fit), memory bandwidth (training and inference throughput are usually bandwidth-bound), precision support (which Tensor Core generation handles the formats your training stack uses), and interconnect (NVLink version determines scaling efficiency once you go multi-GPU).
| Workload | Key Constraint | Recommended Architecture | Representative GPU |
|---|---|---|---|
| Fine-tuning small models (7B-13B) | VRAM fit at QLoRA | Ampere or Ada Lovelace | RTX 4090, A100 80GB |
| Full fine-tuning, medium models (30B-70B) | VRAM capacity + bandwidth | Hopper | H100 SXM, H200 |
| LLM training from scratch (>70B parameters) | Multi-GPU bandwidth + capacity | Hopper or Blackwell | H100 cluster, B200 NVL72 |
| High-throughput inference (FP8) | Throughput per dollar | Hopper or Ada Lovelace | H100, L40S |
| Large-scale inference (FP4, MoE models) | Precision support, bandwidth | Blackwell | B200, GB200 NVL72 |
| Frontier training and agentic AI | Scale, memory, next-gen MoE | Rubin (H2 2026) | Vera Rubin NVL72 |
Last Thoughts on NVIDIA GPU Architecture
NVIDIA GPU architecture is a layered system: the GPC-to-SM-to-CUDA Core hierarchy schedules work, the memory hierarchy from registers to HBM moves data, and each generation unlocks new precision formats and interconnect speeds that compound into meaningfully different AI performance.
Match your workload's bottleneck to the architecture's strength, and the right GPU choice follows directly.
FAQ
What is GPU architecture?
GPU architecture is the design blueprint for a graphics processing unit, specifying how its compute units, memory systems, and execution pipelines are organized. It determines what operations the GPU accelerates, how much memory it has, how fast that memory moves, and what precision formats its hardware supports.
What is the difference between a Streaming Multiprocessor and a CUDA Core?
A Streaming Multiprocessor (SM) is a self-contained execution cluster containing CUDA Cores, Tensor Cores, a warp scheduler, registers, and shared memory. CUDA Cores are the individual processing units for general-purpose math inside each SM. Each H100 SM has 128 CUDA Cores, giving the chip 16,896 total across its 132 SMs.
What is the difference between Ampere, Hopper, and Blackwell?
Ampere introduced TF32 and BF16 Tensor Cores and made large-scale LLM training practical. Hopper added a Transformer Engine with FP8 precision and higher HBM3 bandwidth, targeting transformer models specifically. Blackwell introduced FP4 precision, a dual-die design with 208B transistors, and 8 TB/s HBM3e on the B200. NVIDIA reports up to 5x inference throughput over Hopper for large mixture-of-experts models running FP4.
What is Vera Rubin and when will it be available?
Vera Rubin is NVIDIA's next GPU platform, announced at CES 2026 with systems arriving in the second half of 2026. The Vera Rubin Superchip pairs two Rubin GPUs with a Vera ARM CPU on a 3nm package at 1.8 TB/s. Rubin uses fifth-generation Tensor Cores with a third-generation Transformer Engine, 288 GB of HBM4 per GPU at 22 TB/s, and NVLink 6. Broader enterprise availability is expected in 2027.
Does GPU architecture matter more than VRAM for AI workloads?
Both are independent constraints. VRAM capacity is a hard limit: if your model and optimizer states do not fit, training fails regardless of architecture. Memory bandwidth and Tensor Core generation then determine how fast training runs. An H100 only outperforms an A100 if your training stack actually uses FP8.