Segment
Inference and serving: what it costs to answer
Training is a capital project. Serving is an operating cost that never stops, and it is where most of the engineering that determines unit economics happens. This branch covers the techniques that decide what a request costs and how quickly it comes back.
In one sentence
Inference and serving is the runtime layer that executes trained models against live requests — batching, caching, compressing and routing work to keep latency acceptable and cost per token low.
Generating an answer has two phases with opposite characteristics. Reading the prompt processes all its tokens at once and is arithmetic-heavy. Generating the response produces one token at a time, each requiring a pass over the model's weights, and is bandwidth-heavy. Nearly every serving technique targets one phase or the other, and confusing them leads to optimising the wrong thing.
The result is that cost per token is a systems outcome, not a hardware specification. Two operators with the same accelerators can differ by a large factor on cost per million tokens, depending on batching, cache management, precision and routing.
How this breaks down
Split by the lever being pulled — scheduling, precision, memory, or what the model is given to read.
- Inference and servingthis page
- Serving enginesContinuous batching, paged memory and speculative decoding — where throughput is actually won.Definition page
- Context and cachingThe per-request memory that grows with every token, and the techniques that keep it in check.Definition pageChokepoint
- QuantisationStoring weights in fewer bits to cut memory and bandwidth, and what it costs in accuracy.Definition page
- Retrieval-augmented generationFetching relevant material at answer time instead of relying on what the model memorised.Definition page
What this depends on
Technology dependencies are solved by engineering; supply dependencies are solved by building something, which takes years.
- Technology
Accelerators with high memory bandwidth
Token generation is bandwidth-bound; the hardware ceiling is set there.
AI memory - Technology
A trained model
Serving inherits whatever the training and post-training stages produced, including their failure modes.
Model architecture - Technology
Cluster scheduling and autoscaling
Serving capacity is allocated, drained and re-allocated continuously against traffic that moves by the hour. Without that layer an operator sizes for peak and pays for it all day.
Cluster orchestration
What depends on this
Other pages in this map that name Inference and serving as something they cannot do without.
Companies across Inference and serving
Every company named on a step below this page, ordered by how many of those steps it appears at. Compiled from the pages themselves rather than written separately, so the two cannot disagree. Not a ranking and not a recommendation.
- DatabricksPrivate2 steps
- DeepSeekPrivate2 steps
- Hugging FacePrivate2 steps
- AnthropicPrivate1 step
- AnyscalePrivate1 step
21 more companies appear at a single step each; they are named on the pages for those steps.
How these pages are written
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.
Plutux is not an investment adviser. Market data and AI-generated analysis are for information and education only, not investment advice. Disclaimer