Sub-system
A mixture-of-experts model holds many more parameters than it uses on any single token. A router picks a small number of expert sub-networks per token, so total capacity rises while arithmetic per token stays roughly flat — a trade that moves the cost from compute onto memory and interconnect.
In one sentence
A mixture-of-experts model replaces some feed-forward layers with many parallel expert sub-networks plus a router that activates only a few of them per token, so the model's total parameter count far exceeds the parameters used for any one token.
Making a dense model bigger raises both what it knows and what it costs to run, in lockstep, because every parameter participates in every token. Sparsity breaks that link: if only a fraction of the network is active per token, capacity can grow much faster than cost per token.
The trade is not free. Every expert must be resident in memory even though most are idle for any given token, so memory requirements track the total parameter count rather than the active one. And when experts are spread across accelerators, tokens must be routed to the machine holding the right expert — turning a compute problem into a networking one.
A small learned gating network scores the experts for each token and sends it to the top few. Training has to keep that assignment balanced: without a pressure to spread load, the router collapses onto a handful of favourites and the rest of the capacity is wasted. Auxiliary balancing objectives and capacity limits per expert are the standard fixes.
Experts are usually distributed across accelerators, so each token's activations travel to whichever device holds its chosen experts and the results come back. That is an all-to-all exchange every layer, which places heavy demand on the scale-up fabric and makes interconnect quality a determinant of how well the model runs.
Batching is less predictable, because different requests in a batch activate different experts, and memory must hold the whole model regardless. In exchange, cost per token is far below what a dense model of the same total size would need — which is why the technique has spread quickly in production systems.
2 of these are marked as a chokepoint: a handful of qualified suppliers, a multi-year lead time, or a single geography.
Expert parallelism means an all-to-all exchange every layer; a weak fabric turns the saving in arithmetic into a loss in waiting.
Scale-up fabricsAll experts must be resident even though few are used per token, so memory scales with total parameters rather than active ones.
AI memoryEvery layer sends each token's activations to whichever devices hold its experts and gathers the results. If the collective library and the fabric under it are slow, the routing overhead eats the efficiency the architecture was adopted for.
Cluster fabricsServing a sparse model means holding every expert resident while only a few run per token, and batching requests that do not activate the same ones. A serving engine with no notion of experts wastes most of the advantage.
Serving enginesWhat each company supplies at this step, and — where a public figure exists — its share of this specific market — with what that share measures, the period it covers and who published it. Some rows also show the company’s own reported revenue for the segment covering this step, which is a different thing: it says how much this business matters to that company, not how much of the market it holds. Not a ranking and not a recommendation.
Ships fine-grained expert routing at large scale and publishes the details, which much of the field now copies.
Released the first widely used open-weight mixture-of-experts model.
Originated the sparse expert work and uses it in production models.
Ships mixture-of-experts variants in the open-weight Qwen line.
Builds very large sparse models with a small active parameter count per token.
Released open-weight sparse models, and its served models are widely understood to be sparse as well.
Moved the Llama line to sparse routing at its fourth generation.
Builds sparse models at frontier scale.
Ships sparse variants of the GLM line under open weights.
Combines sparse routing with linear attention, cutting the arithmetic and the cache in the same design.
Releases sparse Hunyuan models with the routing details published.
Supplies the expert-parallel kernels and the fabric the all-to-all exchange actually runs over.
Whether sparsity ratios keep rising, and how far interconnect can follow.
Whether serving stacks close the efficiency gap between sparse and dense models under real request mixes.
Whether memory capacity, rather than arithmetic, becomes the headline specification buyers optimise for.
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.
Plutux is not an investment adviser. Market data and AI-generated analysis are for information and education only, not investment advice. Disclaimer