Sub-system
No single accelerator can hold a frontier model, so training frameworks split the work in several dimensions at once. Each way of splitting has a different communication cost, and choosing the combination is one of the more consequential engineering decisions in a training run.
In one sentence
A training framework is the software that expresses a model and distributes its training across many accelerators, coordinating the parallelism strategy, the communication between devices and the memory optimisations that make the run fit.
There are four ways to split, and large runs use several together. Replicate the model and split the data. Split individual layers across devices. Split the sequence of layers into stages. Split experts across devices in a sparse model. Each generates a different communication pattern, and the art is matching them to the network's actual shape.
The other half of the framework's job is fitting in memory at all. Optimiser state and gradients are several times the size of the weights, so techniques that shard that state across devices, or recompute activations instead of storing them, are what make a run possible rather than merely fast.
Data parallelism replicates the model and averages gradients — simple, communication-light, but requires the model to fit on one device. Tensor parallelism splits individual matrices across devices and communicates within every layer, so it needs the fastest links. Pipeline parallelism assigns consecutive layers to different devices and pays in idle time at the ends. Expert parallelism spreads a sparse model's experts and routes tokens between them.
Common optimisers keep several state tensors per parameter, so the memory needed is a multiple of the model itself. Sharding those tensors across data-parallel replicas — and gathering the pieces only when needed — cuts per-device memory sharply at the cost of extra communication, and is standard in large runs.
Activation recomputation discards intermediate values in the forward pass and recomputes them during the backward pass. It costs extra arithmetic and buys a large reduction in memory, which is usually the right trade at scale because memory, not arithmetic, is what stops a configuration from fitting.
Technology dependencies are solved by engineering; supply dependencies are solved by building something, which takes years.
Every parallelism strategy is implemented as collective operations; their efficiency on the actual topology determines the run's scaling.
Cluster fabricsThe framework's throughput is the throughput of the kernels it dispatches to.
Programming modelsFrameworks must save and restore sharded state consistently, or a long run cannot survive the failures it will certainly have.
AI storageSplitting individual matrices across devices means communicating inside every layer. That is only worth doing where the devices sit in one high-bandwidth domain; across a slower link the strategy loses more than it buys.
Scale-up fabricsA distributed run needs its whole allocation at once and has to be restarted as a unit after a failure. Without a scheduler that understands that, the framework's fault-tolerance machinery has nothing to recover onto.
Cluster orchestrationOther pages in this map that name Training frameworks as something they cannot do without.
What each company supplies at this step, and — where a public figure exists — its share of this specific market — with what that share measures, the period it covers and who published it. Some rows also show the company’s own reported revenue for the segment covering this step, which is a different thing: it says how much this business matters to that company, not how much of the market it holds. Not a ranking and not a recommendation.
Created and still funds PyTorch, which nearly all training code is written in.
Supplies JAX, the framework its own frontier training runs on.
Supplies Megatron and NeMo, the parallelism layers used for large runs.
Supplies DeepSpeed, whose memory-sharding work made large runs fit on available hardware.
Supplies the training and fine-tuning libraries most teams use above the framework.
Commercialises Ray, the layer used to coordinate training with the data and serving work around it.
Maintains the training libraries from its MosaicML acquisition and runs them against customer data.
Supplies the distributed training service, and the compiler stack for its own accelerators.
Funds the framework work that lets the same training code run on its hardware unchanged.
Maintains MindSpore, the framework its own accelerators are trained through.
Maintains PaddlePaddle, the third framework by usage and the default one inside China.
Whether parallelism strategy selection becomes automatic rather than hand-tuned per cluster.
Whether fault tolerance improves enough that node failures stop costing a checkpoint interval.
Whether frameworks keep genuine multi-vendor support in their optimised paths.
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.
Plutux is not an investment adviser. Market data and AI-generated analysis are for information and education only, not investment advice. Disclaimer