Sub-system
Generating each new token requires attending to every previous one. Rather than recomputing that work, models cache it — and that cache, growing linearly with context and multiplied by every concurrent request, is usually what limits how many users a serving system can hold at once.
In one sentence
The KV cache is the stored per-token key and value vectors from previous positions that a transformer reuses when generating each new token, sized in proportion to context length, model dimensions and the number of concurrent requests.
Without a cache, producing token number one thousand would mean recomputing the attention inputs for all nine hundred and ninety-nine before it. The cache removes that repetition, turning generation from quadratic into something manageable — at the price of holding a growing block of memory per request for its whole lifetime.
That memory is the real constraint in most deployments. It is why long-context requests are priced higher, why concurrency drops as contexts grow, and why several distinct architectural techniques exist purely to make the cache smaller.
Roughly: context length, times the number of layers, times the size of the attention state per layer, times two for keys and values, times the bytes per number. For a large model and a long context this reaches many gigabytes for a single request — which is why concurrency and context length trade directly against each other.
Sharing key and value projections across several attention heads cuts the cache by a large factor at a small quality cost, and has become standard. Compressing the cached state into a smaller latent representation goes further. These are architecture decisions made at training time, not serving-time switches — the model has to be built for them.
Applications typically prepend the same long system prompt or document to many requests. Since the cached state for an identical prefix is identical, it can be computed once and shared across requests, or kept between turns of one conversation. For chat and agent workloads this is often the single largest saving available.
1 of these is marked as a chokepoint: a handful of qualified suppliers, a multi-year lead time, or a single geography.
Sharing and freeing cache blocks on demand is what turns the theoretical saving into usable concurrency.
Serving enginesThe largest reductions come from how the model was built, not from what the serving layer does at run time.
TransformersCache competes with model weights for the same expensive memory; capacity per accelerator sets the concurrency ceiling.
High-bandwidth memoryPrefixes worth keeping between turns outlive the accelerator memory available to hold them, so caches are spilled to host DRAM and back. Without that tier the cache is evicted and the prefix is recomputed.
Server memoryOther pages in this map that name Context and caching as something they cannot do without.
What each company supplies at this step, and — where a public figure exists — its share of this specific market — with what that share measures, the period it covers and who published it. Some rows also show the company’s own reported revenue for the segment covering this step, which is a different thing: it says how much this business matters to that company, not how much of the market it holds. Not a ranking and not a recommendation.
Supplies the disaggregated cache management in Dynamo, and the memory the cache lives in.
Supplies the fast storage tier that offloaded caches are paged to.
Supplies a flash platform used to hold caches and context outside accelerator memory.
Supplies the data layer used to share cached context across a cluster.
Supplies the in-memory store used for cached prefixes in many serving stacks.
Sells context caching as a priced feature, which made the cache something buyers reason about rather than an implementation detail.
Discounts cached input automatically, so prompt structure became a cost decision for every developer on it.
Prices cache writes and reads separately — the most explicit statement in the market of what this memory costs.
Published and open-sourced the disaggregated cache architecture much of the field's offloading work now follows.
Ships latent attention that shrinks the cache rather than moving it, and prices disk-cached tokens separately.
Supplies the memory pooling used to hold caches outside the accelerator without falling back to storage.
Whether cache offloading to cheaper memory tiers becomes standard for long-context serving.
Whether new attention designs shrink the cache enough to change long-context pricing.
Whether prefix caching across users becomes a standard platform feature rather than an application-level optimisation.
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.
Plutux is not an investment adviser. Market data and AI-generated analysis are for information and education only, not investment advice. Disclaimer