Sub-system
Pre-training is the single largest expense in building a frontier model, and the reason anyone commits to it is that the result is predictable. Model quality improves smoothly with compute, parameters and data along relationships stable enough to plan a nine-figure budget around.
In one sentence
Pre-training is the initial phase in which a model learns from a very large corpus by predicting the next token; scaling laws are the empirical relationships between compute, parameters, data volume and the resulting loss.
The practical finding is that for a fixed compute budget there is an optimal balance between how large the model is and how many tokens it sees. Early large models were substantially over-parameterised and under-trained; correcting that produced smaller models that outperformed larger ones at the same cost, and reset how runs are planned.
There is a second consideration that pure compute-optimality ignores: a model is trained once and served for a long time. Training a smaller model on more data than compute-optimality suggests costs more up front and less every hour thereafter, which for a widely deployed model is usually the better trade.
Loss falls as a smooth power-law function of compute, parameters and data, with diminishing returns but no visible cliff over the ranges studied. It is an empirical regularity, not a theorem, and it says nothing directly about capability — only about how well the model predicts the next token, which has turned out to correlate usefully with what the model can do.
A frontier run occupies thousands of accelerators continuously. Nodes fail; the run must survive that through periodic checkpoints and restart. Numerical instabilities appear that were invisible at small scale. A large fraction of the engineering is not the model at all — it is keeping an enormous distributed job healthy long enough to finish.
The binding limit has shifted over time between compute, high-quality data, and power. Compute-optimal recipes demand tokens in proportion to parameters, and the supply of high-quality text is finite, which is what makes data curation, licensing and synthetic generation strategic rather than clerical.
4 of these are marked as a chokepoint: a handful of qualified suppliers, a multi-year lead time, or a single geography.
The run is defined by how many accelerator-hours can be kept working in parallel.
AI computeCompute-optimal training needs data in proportion to model size, and the readily available high-quality supply is finite.
Training corporaAt this scale hardware failures during a run are certain, so the ability to save and resume state is a prerequisite, not an optimisation.
AI storageA frontier run is a sustained multi-tens-of-megawatts load; securing that capacity is now part of planning a training programme.
PowerA run of this size exists only as a parallelism strategy across thousands of devices, with sharded optimiser state and recomputation to make it fit. Change the framework and the maximum model size changes with it.
Training frameworksSomething fails during any run of this length. Without a scheduler that detects it, replaces the node and restarts from the last checkpoint quickly, the fleet spends a material fraction of the run idle.
Cluster orchestrationCompute has to arrive as installed, cabled and validated racks before it can be trained on. The lead time on those systems, not the chip design, usually sets when a run can start.
AI server systemsOther pages in this map that name Scaling and pre-training as something they cannot do without.
What each company supplies at this step, and — where a public figure exists — its share of this specific market — with what that share measures, the period it covers and who published it. Some rows also show the company’s own reported revenue for the segment covering this step, which is a different thing: it says how much this business matters to that company, not how much of the market it holds. Not a ranking and not a recommendation.
Runs frontier-scale pretraining and published the original scaling relationships.
Runs frontier-scale pretraining.
Runs frontier pretraining on its own TPU fleet, the only lab not buying its main training silicon.
Runs frontier-scale pretraining and releases the resulting weights.
Built one of the largest single training clusters to run frontier pretraining.
Supplies the accelerators, interconnect and training software nearly all of these runs execute on.
Operates the fleet a large share of frontier pretraining runs on, and runs smaller ones of its own.
Built a cluster of its own accelerators specifically to carry a partner lab's pretraining — the first at that scale not on merchant GPUs.
Published the training-cost accounting that reset what the field assumed a frontier run had to cost.
Runs the largest pretraining programme in China and releases most of the results under open weights.
Runs frontier-scale pretraining and publishes the optimiser and stability work behind it.
Rents the clusters several of these runs execute on, rather than training models itself.
Whether the observed scaling relationships continue to hold as runs grow, or bend.
Whether high-quality data supply, rather than compute, becomes the binding constraint on frontier runs.
How much of the capability gain shifts from pre-training to post-training and to compute spent at answer time.
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.
Plutux is not an investment adviser. Market data and AI-generated analysis are for information and education only, not investment advice. Disclaimer