High-Bandwidth Memory & Chiplets
AI chips now combine several dies and stacked high-bandwidth memory in one package; memory stacks and packaging capacity, not transistors alone, limit how many accelerators can be built.
Open in the interactive tree →SK Hynix made the first high-bandwidth memory in 2013 by stacking DRAM dies; AMD's Fiji GPU (June 2015) was the first to use it. Bandwidth per stack rose from 128 GB/s (HBM1) to 819 GB/s (HBM3, 2022), and the HBM4 standard of April 2025 allows up to 2 TB/s. Chiplets, small dies joined on a silicon interposer, let designers mix logic and memory from different process nodes and test each part before assembly; Nvidia's H100 pairs one processor with five HBM3 stacks and 80 GB.
As of now
In 2026 AMD's Instinct MI400 is among the first devices with HBM4, and SK Hynix, Samsung and Micron are the three large producers; HBM wafers now crowd out standard DRAM, whose prices have risen sharply since early 2025. Packaging is the other bottleneck: TrendForce (15 June 2026) reports that TSMC's CoWoS capacity could reach 120,000 to 140,000 wafers a month in 2026, that the supply gap should shrink from about 20% to about 10% by year-end, and that the severe shortage of 2.5D packaging should start to ease in 2027. In July 2026 JEDEC published SPHBM4, which delivers HBM4-class bandwidth on cheaper organic substrates instead of silicon interposers.
Where today's AI could help
Medium AI leverage. AI already speeds the design side, such as chip floorplans, but stacking yield, heat and packaging capacity are physical manufacturing limits.
- Reinforcement learning for chip and package floorplans
- Fast surrogate models for heat and signal integrity in stacked dies
- Automated defect inspection and yield analysis in bonding and test
Shown so far
- Google DeepMind's AlphaChip (Nature 2021, named in a September 2024 addendum) lays out chip floorplans in hours instead of weeks and has been used in the last three TPU generations. source
Prerequisites
- DRAM Main Memory1966HBM stacks DRAM dies vertically and links them with through-silicon vias.
- Graphics Processor (GPU)1999Graphics processors were the first chips to need more memory bandwidth than ordinary DRAM gave.
Unlocks
- AI Accelerators & Data Centers2026AI accelerators pair processor dies with stacked HBM in advanced packages.