Sub-system
A serving engine is the software between an API request and the accelerator. Its job is to keep the hardware saturated with work from many concurrent requests while none of them waits too long, and the techniques it uses are worth several times more than the difference between adjacent chip generations.
In one sentence
A serving engine is the runtime that schedules many concurrent inference requests onto accelerators — batching them dynamically, managing the memory each holds, and applying decoding optimisations to raise throughput without breaking latency targets.
The naive approach — collect a fixed batch, run it to completion, collect the next — wastes most of the hardware. Requests finish at different times, so the batch runs at the pace of its longest member while finished slots sit idle. Modern engines instead admit and retire requests continuously, so a finished request's slot is refilled on the very next step.
The second problem is memory. Each in-flight request holds a cache of its context that grows as it generates. Reserving the worst case for every request wastes most of the memory; managing it in small pages, allocated on demand, allows far more concurrent requests on the same hardware.
The scheduler works at the granularity of one generation step rather than one request. Every step it decides which live requests advance, admits new ones into free slots, and retires finished ones. That single change typically multiplies throughput, because the accelerator stops waiting for the slowest member of a fixed batch.
The context cache is stored in fixed-size blocks with an index, rather than one contiguous reservation per request. Blocks are allocated as generation proceeds and freed immediately at completion, and identical prefixes across requests can share the same blocks — which matters enormously when many requests share a long system prompt.
A small, fast model proposes several tokens ahead; the large model verifies them in a single pass and accepts the prefix that matches what it would have produced. Because verification is one arithmetic-heavy pass rather than several bandwidth-bound ones, accepted tokens come almost free. The output is identical to what the large model would have generated alone.
Technology dependencies are solved by engineering; supply dependencies are solved by building something, which takes years.
The engine's scheduling only helps if the underlying kernels reach a high fraction of the hardware's capability.
Programming modelsHow many requests fit at once is decided by cache memory management more than by anything else in the system.
Context and cachingAbove one node, a router that places requests where the relevant cache and weights already live is part of the engine's job.
Cluster orchestrationMost of the memory and bandwidth headroom the scheduler exploits comes from serving weights at reduced precision. At full precision far fewer requests fit on a device and the batching machinery has much less to work with.
QuantisationA model whose weights exceed one device is served across several joined into one memory domain, and every token crosses that link. If it is slow, per-token latency is set by the network rather than by the arithmetic.
Scale-up fabricsOther pages in this map that name Serving engines as something they cannot do without.
What each company supplies at this step, and — where a public figure exists — its share of this specific market — with what that share measures, the period it covers and who published it. Some rows also show the company’s own reported revenue for the segment covering this step, which is a different thing: it says how much this business matters to that company, not how much of the market it holds. Not a ranking and not a recommendation.
Supplies TensorRT-LLM, Triton and Dynamo, the serving stack tuned to its own accelerators.
Sponsors vLLM through Red Hat, the open engine most self-hosted serving now uses.
Operates a serving platform and contributes the kernels behind much of the open tooling.
Operates a hosted inference platform tuned for low latency on open weights.
Supplies managed serving infrastructure for teams deploying their own models.
Serves models inside the same platform that holds the customer's data.
Maintains the serving stack for its own accelerators, and the port that runs the open engines on them.
Operates the largest hosted serving fleet, and maintains the runtime under much of the non-GPU inference.
Serves other developers' models as a product, which is where most enterprise inference is actually executed.
Funds the work that keeps the open engines running on its accelerators, which is the condition of them being bought at all.
Maintains the serving library and endpoints most teams use before they operate their own.
Supplies the serving layer built on Ray, used where inference sits inside a larger distributed application.
Whether prompt-processing and token-generation phases are increasingly run on separate, differently specified hardware pools.
Whether open serving engines remain the default over vendor-specific runtimes.
How much of the remaining gap to theoretical hardware limits is left for scheduling to close.
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.