Sub-system
AI memory: the constraint that decides how fast a model runs
Arithmetic has become cheap relative to moving the numbers it operates on. In modern AI systems the memory hierarchy, not the arithmetic units, is what usually determines both the speed of a run and the cost of a token.
In one sentence
AI memory is the hierarchy of storage around an accelerator — from stacked high-bandwidth memory beside the die, through system memory, to the storage holding datasets and checkpoints — sized to keep the arithmetic units busy.
Every generation of accelerator has increased arithmetic throughput faster than memory bandwidth. The result is that an increasing share of real workloads are limited by how fast operands can be delivered, not by how fast they can be multiplied. Practitioners call this the memory wall, and most of the interesting engineering in AI systems is an attempt to work around it.
That pressure shows up as three separate purchases with three separate supply pictures: the stacked memory co-packaged with the chip, the conventional memory in the server, and the storage that feeds the cluster. Only the first is genuinely scarce, but the other two set what a full system costs.
How this breaks down
Split by distance from the arithmetic units — and therefore by bandwidth, cost and scarcity.
- DRAM manufacturingWhere stacked memory, server memory and everything else are all made — from three companies.Definition pageChokepoint
- NAND and SSDsVertically stacked flash and the drives built from it, where AI storage bandwidth comes from.Definition pageChokepoint
- High-bandwidth memoryWhy AI chips carry their memory in vertical stacks a few millimetres away, and why it is scarce.Definition pageChokepoint
- Server memoryThe conventional DRAM in an AI server, and the interconnect standards trying to make it shareable.Definition pageChokepoint
- AI storageThe datasets, checkpoints and caches underneath a training run, and why they are a throughput problem.Definition page
What this depends on
1 of these is marked as a chokepoint: a handful of qualified suppliers, a multi-year lead time, or a single geography.
- Supply chainChokepoint
DRAM manufacturing capacity
All three tiers are built on the same underlying DRAM and NAND fabs, so a shift of capacity toward one tier tightens the others.
Capacity cycles - Technology
Through-silicon vias and stacking
Stacked memory only exists because dies can be thinned, drilled through and bonded vertically with acceptable yield.
Hybrid bonding - Standard
JEDEC memory interface specifications
Every tier here is interchangeable between suppliers only because the interface, timing and signalling are specified in common. Without that a processor would be tied to one memory vendor for its life.
What depends on this
Other pages in this map that name AI memory as something they cannot do without.
- Artificial intelligence · AI compute · AI acceleratorsInference siliconInference parts either co-package expensive high-bandwidth memory or make an explicit trade for cheaper, larger, slower memory. Either way memory is the design's centre of gravity.
- Artificial intelligence · Model architectureMixture of expertsAll experts must be resident even though few are used per token, so memory scales with total parameters rather than active ones.
- Artificial intelligenceInference and servingToken generation is bandwidth-bound; the hardware ceiling is set there.
Companies across AI memory
Every company named on a step below this page, ordered by how many of those steps it appears at. Compiled from the pages themselves rather than written separately, so the two cannot disagree. Not a ranking and not a recommendation.
- Kingston TechnologyPrivate2 steps
- ASM InternationalAmsterdam1 step
34 more companies appear at a single step each; they are named on the pages for those steps.
How these pages are written
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.