Every time you train a model or run an inference job, the hardware makes millions of micro-decisions: how to schedule threads, move data, and execute matrix operations. Those decisions are dictated by GPU architecture, and understanding it helps you debug memory bottlenecks faster, choose the right hardware, and make sense of new generations.
This guide covers NVIDIA GPU architecture from the ground up: from the physical components inside the chip, to how Ampere, Hopper, and Blackwell differ in practice, to what the upcoming Vera Rubin platform changes for AI workloads.
What GPU Architecture Actually Is
GPU architecture is the design blueprint that determines how a chip's compute units, memory systems, and execution pipelines are organized. It governs parallelism, precision support, memory bandwidth, and how efficiently the chip executes matrix multiplications, attention computations, and large memory transfers.
The term covers two layers:
- Physical microarchitecture - how transistors are arranged, what execution units exist, and how they connect.
- Programming model - how software exposes that hardware to developers. For NVIDIA, that model is CUDA which is supported on all its GPUs since being launched in 2007.
Why GPU Architecture Differs from CPU Architecture
GPUs and CPUs are architecturally opposite by design: CPUs minimize latency, GPUs maximize throughput. CPUs achieve low latency through a handful of powerful cores with deep out-of-order execution, large caches, and branch prediction logic. GPUs achieve high throughput by trading per-core intelligence for thousands of simpler cores that execute tens of thousands of calculations simultaneously.
For context, a typical consumer CPU has 8–16 cores, while a comparable consumer GPU like the RTX 4090 has 16,384.
| Dimension | CPU (e.g., AMD EPYC 9654) | GPU (e.g., NVIDIA B200) |
|---|---|---|
| Core count | 96 high-performance cores | 20,480 CUDA Cores |
| Execution model | Sequential, low latency | Massively parallel, high throughput |
| Memory | Up to ~12 TB system RAM | 192 GB HBM3e (B200) |
| Memory bandwidth | ~461 GB/s (8-channel DDR5) | 8 TB/s (B200) |
| Best workload | Complex branching logic, OS tasks | Dense matrix math, parallel transforms |
The high bandwidth is what makes GPUs indispensable for AI. A transformer model consists almost entirely of matrix multiplications and memory reads. At 8 TB/s, a B200 feeds its cores fast enough to keep them busy at any scale. A CPU cannot come close.
For a comparison of how TPUs fit alongside GPUs in AI infrastructure, see our TPU vs GPU comparison.
The Building Blocks of NVIDIA GPU Architecture
NVIDIA organizes its GPUs in a strict hierarchy. Understanding it from top to bottom makes spec sheets readable and architectural differences between generations legible.
Graphics Processing Clusters
A Graphics Processing Cluster (GPC) is the top-level organizational unit inside an NVIDIA GPU. Each GPC operates semi-independently, containing its own scheduling logic, a set of Texture Processing Clusters (TPCs), and a cluster-level L1 cache.
GPCs determine how work is distributed across the chip. When batch size and sequence length are large enough to saturate all GPCs simultaneously, the chip is being used efficiently. Underutilized GPCs are the hardware-level explanation for why small batch sizes produce poor GPU utilization numbers.
For consumer and workstation GPUs, NVIDIA scales its designs by enabling more or fewer GPCs on the same die: the RTX 3050 and RTX 3090 are both built on Ampere architecture but they enable 2 and 7 GPCs respectively.
This is partly a result of manufacturing: dies that come out of fabrication with a defective GPC get disabled and sold as lower-tier products rather than discarded. NVIDIA also intentionally disables GPCs on fully functional dies to fill out its product lineup.
Streaming Multiprocessors
The Streaming Multiprocessor (SM) is the fundamental execution unit of an NVIDIA GPU. Each SM contains CUDA Cores, Tensor Cores, a warp scheduler, register files, and a configurable shared memory block. All actual computation happens inside an SM.
SM count is the number that meaningfully describes a GPU's raw compute capacity. The A100 has 108 SMs, the H100 SXM has 132, and the B200 has 160. More SMs means more parallelism, but only if your workload generates enough threads to keep them occupied.
Each SM executes several groups of 32 threads called "warps". The warp scheduler selects ready warps and issues instructions to the CUDA Cores or Tensor Cores inside the SM. Because memory latency is high relative to computation speed, the SM keeps many warps in flight at once. When one warp stalls on a memory fetch, the scheduler switches to another that is ready. This latency-hiding through warp switching is what makes the GPU's throughput model work.
CUDA Cores vs Tensor Cores vs RT Cores
These three execution unit types inside each SM serve different purposes.
CUDA Cores are general-purpose floating-point and integer arithmetic units. They execute one operation per clock cycle per core. An H100 SM has 128 CUDA Cores, giving the chip 16,896 total. For AI workloads, CUDA Cores handle everything that does not map to matrix multiplication: activation functions, normalization, data preprocessing, and control flow.
Tensor Cores are specialized matrix multiplication units that perform a fused multiply-accumulate on small matrix tiles in a single clock cycle. Where a CUDA Core computes one multiply-add, a Tensor Core computes a 4x4x4 matrix operation in the same time. Each H100 SM contains four fourth-generation Tensor Cores. The Tensor Core generation is the most important architectural differentiator between GPU generations for AI.
RT Cores accelerate ray-tracing for real-time graphics rendering. They are irrelevant for AI training, inference, and AI-based image and video generation workloads, so ignore them when evaluating GPUs for compute.
For a deep dive on Tensor Core generations and training throughput, see Tensor Cores Explained.
GPU Memory Hierarchy: Where Most Performance Is Lost
The compute units in an NVIDIA GPU execute operations far faster than memory can supply data. The memory hierarchy exists to bridge that gap, and understanding it is the key to diagnosing memory-bound training runs and making informed VRAM decisions.
On-Chip Memory: Registers, Shared Memory, and L1/L2 Cache
Registers are the fastest memory on the chip. Each thread gets a private register file with zero-latency access. Registers are finite: the H100 SM has 65,536 32-bit registers shared across all threads running on that SM simultaneously.
Shared memory is a user-controlled scratchpad that all threads within a block can read and write. It sits inside the SM and has very low latency (roughly 20-30 clock cycles). In CUDA kernels, shared memory stages data for reuse across threads, avoiding repeated round-trips to slower global memory. An H100 SM has 228 KB of configurable L1/shared memory, and getting this right is a primary lever for hand-tuned kernel performance.
L2 cache is shared across all SMs and serves as the primary buffer between the SMs and global DRAM. The H100 has 50 MB of L2. Blackwell's B100 increases this to 192 MB, a 4x jump. A larger L2 matters for AI because model weights accessed repeatedly across batches stay warm in cache rather than being re-fetched from HBM on every forward pass.
Off-Chip Memory: GDDR6 vs HBM
GDDR6X is used in consumer and professional GPU lines like the RTX 40 and 50 series. It connects to the GPU die via a standard memory interface. An RTX 4090 has 24 GB of GDDR6X at around 1 TB/s bandwidth, fast enough for inference on medium-size models and fine-tuning with QLoRA, but the bandwidth ceiling becomes a bottleneck on large-batch training.
HBM (High Bandwidth Memory) stacks DRAM dies vertically and connects them to the GPU via a silicon interposer, dramatically shortening the electrical path. The H100 SXM uses HBM3 at 3.35 TB/s, more than three times the RTX 4090. HBM also provides far more capacity: 80 GB on the H100 vs 24 GB on the RTX 4090. For LLM training, both matter independently: capacity holds the model and optimizer states, bandwidth feeds the Tensor Cores fast enough to maintain high utilization.
| Memory Type | Representative GPU | Capacity | Bandwidth | Best For |
|---|---|---|---|---|
| GDDR6X | RTX 4090 | 24 GB | ~1 TB/s | Inference, fine-tuning small models |
| GDDR7 | RTX 5090 | 32 GB | ~1.8 TB/s | Inference, consumer-grade training |
| HBM2e | A100 80GB | 80 GB | 2 TB/s | LLM training, large-batch inference |
| HBM3 | H100 SXM | 80 GB | 3.35 TB/s | Transformer training, inference at scale |
| HBM3e | H200 SXM | 141 GB | 4.8 TB/s | Frontier LLM training, long-context inference |
| HBM3e | B200 | 192 GB | 8 TB/s | Multi-trillion parameter training and inference |
| HBM4 | Rubin R100 (H2 2026) | TBD | TBD (est. >8 TB/s) | Next-generation agentic AI and MoE workloads |
| Rubin specifications are preliminary. Final HBM4 bandwidth figures have not been officially confirmed as of publication. | ||||
How GPUs Execute Work: SIMT, Warps, and Thread Hierarchy
GPUs execute one instruction across many data elements simultaneously. NVIDIA uses a model called SIMT: Single Instruction, Multiple Threads.
SIMT vs SIMD
SIMD (Single Instruction, Multiple Data) is the CPU's approach to parallelism. A CPU SIMD instruction applies one operation to a vector of data elements in a single clock cycle, and the programmer manages that vector directly. SIMT is more flexible: the programmer writes a scalar kernel from the perspective of a single thread, and the GPU replicates execution across thousands of threads in parallel without manual vector packing.
The flexibility comes with a cost. If threads within the same warp take different execution paths through a conditional branch, the GPU must execute both paths sequentially with some threads masked off. This is warp divergence, and it is a common source of underperformance in non-trivial ker nels.
The Thread Hierarchy
The Thread Hierarchy CUDA organizes threads into a hierarchy that maps directly onto the GPU's hardware:
- Threads are the base unit, each running the kernel on its own data.
- Warps group 32 threads together. The warp scheduler issues instructions to all 32 simultaneously.
- Thread blocks group warps together. Threads within a block share the SM's local memory and can synchronize.
- Grids group thread blocks together, representing the full kernel launch.
When you launch a kernel, the grid splits into thread blocks distributed across SMs. Each SM holds multiple active blocks at once, limited by its register and shared memory capacity. The more warps an SM keeps in flight, the better it hides memory latency by switching between them when one stalls. That ratio of active to maximum possible warps is called occupancy, and it's one of the main things kernel developers tune for.
NVIDIA GPU Architecture Generations: What Changed and Why It Matters for AI
NVIDIA releases a new GPU architecture roughly every two years. The most relevant changes for AI practitioners are in Tensor Core generation, supported precision formats, memory bandwidth, and the software features that enable efficient transformer training.
Ampere (A100, A10): The LLM Training Baseline
Ampere, launched in 2020, made large-scale LLM training practical. The A100 introduced third-generation Tensor Cores with TF32 and BF16 support, structural 2:4 sparsity (which could double inference throughput on compatible models), and Multi-Instance GPU (MIG), allowing a single A100 to be partitioned into up to seven isolated instances. NVLink 3.0 provided 600 GB/s bidirectional bandwidth between GPUs in a node. The A100 remains widely available and cost-effective for medium-scale workloads in 2026.
Hopper (H100, H200): The Transformer Era Architecture
Hopper, launched in 2022, was designed specifically for transformer workloads. Its most significant innovation is the Transformer Engine: a hardware unit that dynamically switches between FP8 and BF16 precision at the layer level, enabling faster computation without meaningful accuracy loss. The H100 introduced fourth-generation Tensor Cores with FP8 support, HBM3, and NVLink 4.0 at 900 GB/s. The H200 followed with HBM3e and 141 GB of capacity, making it the preferred choice for long-context inference.
The H100 remains the dominant GPU for production AI training in mid-2026. Its Tensor Core performance, memory bandwidth, and broad support across PyTorch, JAX, and inference frameworks make it the safest default for most workloads.
Ada Lovelace (L40, RTX 4090): Inference and Edge
Ada Lovelace is NVIDIA's consumer and professional line running parallel to Hopper. It uses GDDR6X rather than HBM, meaning lower bandwidth and capacity. The L40 is the data center variant and has become a popular inference GPU at a lower per-hour cost than H100 instances. The RTX 4090 is widely used for fine-tuning smaller models locally. Neither is suited for large-scale LLM training where HBM bandwidth is the limiting factor.
Blackwell (B100, B200, GB200): Generative AI at Scale
Blackwell, launched in 2024, is the current flagship data center architecture. It introduces fifth-generation Tensor Cores with FP4 and FP6 precision support, a dual-die design connecting two reticle-limited chiplets via a 10 TB/s chip-to-chip interconnect to form a single logical GPU with 208B transistors, HBM3e on the B200 at 8 TB/s and 192 GB capacity, and NVLink 5 at 1.8 TB/s per GPU. The GB200 superchip pairs two B200 GPUs with a Grace CPU via NVLink-C2C at 900 GB/s, eliminating the PCIe bottleneck for CPU-GPU data transfer.
At FP4 precision, NVIDIA claims up to 25x lower inference cost compared to Hopper at FP16. The 192 GB HBM3e capacity per B200 GPU and NVLink 5 interconnect also allow training much larger models per node before requiring inter-node communication. The GB200 NVL72 rack-scale system connects 72 B200 GPUs and 36 Grace CPUs into a unified system, blurring the boundary between a GPU cluster and a single logical accelerator.
What's Next: Vera Rubin (H2 2026)
Vera Rubin is NVIDIA's next GPU architecture, now in full production following Jensen Huang's announcement at CES 2026. The platform takes a substantial step from Blackwell: 336B transistors on a dual-die TSMC 3nm design (a full node shrink from Blackwell's 4NP), HBM4 memory, NVLink 6 at doubled bandwidth over NVLink 5, and sixth-generation Tensor Cores delivering 50 petaflops of FP4 compute. NVIDIA claims 3.5x training throughput and up to 5x inference throughput improvement over Blackwell for MoE models.
The Vera Rubin Superchip combines two Rubin GPUs with a Vera ARM CPU (88 Olympus cores) in a single package connected by NVLink-C2C at 1.8 TB/s. The NVL72 rack, combining 72 Rubin GPUs, is already shipping to hyperscaler partners. Broader enterprise cloud access is expected by mid-2027.
See our dedicated Vera Rubin architecture post for full specs, performance data, and availability timelines.
GPU Architecture Generation Comparison
| Architecture | Year | Key GPUs | Tensor Core Gen | Best Precision | Peak VRAM (SXM/HBM) | Memory BW |
|---|---|---|---|---|---|---|
| Volta | 2017 | V100 | Gen 1 | FP16 | 32 GB HBM2 | 900 GB/s |
| Ampere | 2020 | A100 | Gen 3 | TF32, BF16 | 80 GB HBM2e | 2 TB/s |
| Hopper | 2022 | H100, H200 | Gen 4 | FP8 | 141 GB HBM3e (H200) | 4.8 TB/s (H200) |
| Blackwell | 2024 | B100, B200, GB200 | Gen 5 | FP4, FP6 | 192 GB HBM3e (B200) | 8 TB/s (B200) |
| Rubin | 2026 | R100, Vera Rubin | Gen 6 | FP4 (NVFP4) | HBM4 (TBD) | TBD |
| Rubin specifications are based on official NVIDIA announcements and are subject to change. | ||||||
NVIDIA vs AMD GPU Architecture for AI
NVIDIA dominates AI training infrastructure in 2026, but AMD has a meaningful presence where ROCm compatibility and cost-per-FLOP are priorities. Understanding the architectural differences matters when cloud catalogs offer both.
CDNA vs CUDA Architecture
NVIDIA's data center GPU architecture is based on the CUDA compute paradigm: a unified SM-based design that runs the same kernel code across graphics and compute workloads. AMD separates its architectures. RDNA (Radeon) is the consumer and workstation line, optimized for graphics. CDNA (Compute DNA) is AMD's data center architecture, purpose-built for AI and HPC with no graphics-rendering hardware. The MI300X is a CDNA 3 chip containing no display engines, RT Cores, or gaming-oriented features.
The MI300X features 192 GB of HBM3 at 5.3 TB/s bandwidth, more capacity than an H100 and more bandwidth than an H200. For inference serving of large models that barely fit in VRAM, this capacity advantage is meaningful. AMD's Infinity Fabric connects multiple GPU dies within the MI300X's multi-chiplet design (eight compute dies, eight HBM stacks).
The central limitation is software. CUDA has a two-decade head start and is deeply integrated into PyTorch, TensorFlow, JAX, and virtually every production ML framework. ROCm has made significant progress but still requires more porting work than simply picking a different cloud instance.
See our ROCm vs CUDA comparison for a detailed comparison of both ecosystems, including compatibility, framework support, and migration considerations.
Multi-GPU Scaling: NVLink and How Distributed Training Works
Single-GPU training has practical limits. Once a model does not fit in one GPU's VRAM, or once single-GPU throughput is too slow for your timeline, you need multi-GPU setups. The interconnect architecture between GPUs determines how efficiently they cooperate.
NVLink and NVLink Switch
PCIe, the standard bus connecting a GPU to the rest of the system, provides around 64 GB/s bidirectional bandwidth on PCIe Gen 5. For multi-GPU training, this is a severe bottleneck: gradient synchronization across 8 GPUs requires moving gigabytes of data per step, and at 64 GB/s per GPU that communication overhead dominates training time.
NVLink is NVIDIA's proprietary high-speed GPU-to-GPU interconnect that bypasses PCIe for intra-node communication. NVLink 4.0 (Hopper) provides 900 GB/s bidirectional bandwidth per GPU. NVLink 5.0 (Blackwell) doubles that to 1.8 TB/s. NVLink Switch extends NVLink beyond the server node, connecting up to 576 GPUs in a rack-scale system where every GPU has direct high-bandwidth access to every other. This is what makes the GB200 NVL72 and Rubin NVL72 systems function as single logical accelerators rather than collections of discrete servers.
NVLink connectivity within a node is the difference between near-linear training scaling and diminishing returns from communication overhead. When you go beyond a single node, the interconnect between nodes (InfiniBand or Ethernet) becomes equally critical.
Choosing a GPU Architecture for Your AI Workload
Choosing the right architecture means matching its strengths to your workload's bottlenecks, not finding the most powerful GPU on paper.
The four dimensions that determine fit are VRAM capacity (your model and optimizer states must fit), memory bandwidth (training and inference throughput are usually bandwidth-bound), precision support (which Tensor Core generation handles the formats your training stack uses), and interconnect (NVLink version determines scaling efficiency once you go multi-GPU).
| Workload | Key Constraint | Recommended Architecture | Representative GPU |
|---|---|---|---|
| Fine-tuning small models (7B-13B) | VRAM fit at QLoRA | Ampere or Ada Lovelace | RTX 4090, A100 80GB |
| Full fine-tuning, medium models (30B-70B) | VRAM capacity + bandwidth | Hopper | H100 SXM, H200 |
| LLM training from scratch (>70B parameters) | Multi-GPU bandwidth + capacity | Hopper or Blackwell | H100 cluster, B200 NVL72 |
| High-throughput inference (FP8) | Throughput per dollar | Hopper or Ada Lovelace | H100, L40S |
| Large-scale inference (FP4, MoE models) | Precision support, bandwidth | Blackwell | B200, GB200 NVL72 |
| Frontier training and agentic AI | Scale, memory, next-gen MoE | Rubin (H2 2026) | Vera Rubin NVL72 |
Last Thoughts on NVIDIA GPU Architecture
NVIDIA GPU architecture is a layered system: the GPC-to-SM-to-CUDA Core hierarchy distributes and schedules work; the memory hierarchy from registers to HBM moves data; and each architecture generation unlocks new precision formats and interconnect speeds that compound into meaningfully different performance for AI workloads.
Match your workload's bottleneck to the architecture's strength, and the right GPU choice follows directly.
Frequently Asked Questions
What Is GPU Architecture?
GPU architecture is the design blueprint for a graphics processing unit, specifying how its compute units, memory systems, and execution pipelines are organized. It determines what operations the GPU can accelerate, how much memory it has and how fast it moves, and what precision formats its hardware supports.
What Is the Difference Between a Streaming Multiprocessor and a CUDA Core?
A Streaming Multiprocessor (SM) is a self-contained execution cluster containing CUDA Cores, Tensor Cores, a warp scheduler, registers, and shared memory. CUDA Cores are the individual arithmetic units inside each SM. Each H100 SM contains 128 CUDA Cores, giving the chip 16,896 total across its 132 SMs.
What Is the Difference Between CUDA Cores and Tensor Cores?
CUDA Cores execute general-purpose arithmetic one operation per clock cycle. Tensor Cores are specialized matrix-multiplication units that execute a full matrix multiply-accumulate on a small tile in a single cycle, delivering roughly 10-20x higher throughput for operations that dominate deep learning. Tensor Core generation is the most important architectural differentiator between GPU generations for AI.
How Does GPU Memory Hierarchy Affect AI Training Performance?
Most AI training runs are memory-bandwidth-bound, not compute-bound. Higher HBM bandwidth reduces time the Tensor Cores spend waiting for data. Insufficient VRAM capacity forces batching strategies that reduce throughput or fail outright. Optimizing shared memory usage within CUDA kernels is the primary lever for squeezing more performance from a given architecture.
What Is the Difference Between Ampere, Hopper, and Blackwell?
Ampere introduced TF32 and BF16 Tensor Cores and made large-scale LLM training practical. Hopper added a Transformer Engine with FP8 precision and higher HBM3 bandwidth, targeting transformer models specifically. Blackwell introduced FP4 precision, a dual-die design with 208B transistors, and 8 TB/s HBM3e on the B200, delivering 2.5-5x inference throughput improvement over Hopper for large MoE models.
What Is Vera Rubin and When Is It Available?
Vera Rubin is NVIDIA's next GPU architecture, now in full production. The Vera Rubin Superchip pairs two Rubin GPUs with a Vera ARM CPU on a single 3nm package connected at 1.8 TB/s. It features sixth-generation Tensor Cores with 50 petaflops of FP4, HBM4, and NVLink 6. Hyperscaler cloud access begins H2 2026, with broader enterprise availability expected mid-2027.
Does GPU Architecture Matter More Than VRAM for AI Workloads?
Both are independent constraints that matter. VRAM capacity is a hard limit: if your model and optimizer states do not fit, training fails regardless of architecture. Memory bandwidth and Tensor Core generation then determine how fast training runs once those constraints are satisfied. An H100 only outperforms an A100 if your training stack actually exploits FP8.
What Is the Difference Between GDDR6 and HBM Memory on a GPU?
GDDR6X is used in consumer and professional GPUs like the RTX 40 series. It is cheaper but offers lower bandwidth (around 1 TB/s on the RTX 4090) and less capacity. HBM stacks DRAM dies vertically and connects them via a silicon interposer, giving data center GPUs like the H100 3.35 TB/s and 80 GB. HBM is the key reason data center GPUs outperform consumer cards on large AI workloads.