Sub-system
Before a training run there is a much less glamorous computation: reading petabytes of raw text, filtering most of it away, deduplicating what remains and writing it out in a form a cluster can stream. That runs on ordinary compute with general-purpose distributed frameworks, and it is a real cost line of its own.
In one sentence
Distributed data processing frameworks execute transformations over datasets too large for one machine, splitting work across a cluster and handling partitioning, shuffling and failure recovery.
The characteristic operation is the shuffle: redistributing records across the cluster so that everything sharing a key ends up in one place. Deduplication needs it, grouping needs it, and it is where these jobs spend most of their time and all of their network bandwidth. Framework design is largely about making shuffles cheaper or avoiding them.
The corpus work is a sequence of such passes — parse, normalise, language-identify, score for quality, deduplicate near-matches, mix according to weights, tokenise, shard. Each is a full pass over petabytes, and at that scale the difference between a well-planned pipeline and a naive one is measured in weeks of cluster time.
Exact duplicates are easy. Documents that differ by a boilerplate header are not, and they are far more common. The standard approach hashes overlapping shingles of each document so that similar documents collide, then compares only within collisions — which turns an impossible all-pairs comparison into a large but tractable shuffle.
Corpus preparation is batch, but the same engines now express streaming pipelines with the same operators, which matters for continuously updated retrieval indexes and for feature pipelines. Teams increasingly run one framework for both rather than maintaining separate systems.
Almost none of it uses accelerators. It is processor-bound and network-bound on ordinary cloud machines, which is why data preparation cost is invisible in accelerator-hour accounting and shows up separately — and why it is frequently underestimated when a training budget is drawn up.
Technology dependencies are solved by engineering; supply dependencies are solved by building something, which takes years.
Corpus jobs run on ordinary cloud instances against object storage, not on the training cluster.
Data centresThese jobs are scheduled on the same shared infrastructure as everything else and compete for it.
Container platformsEngines interoperate because they read and write the same on-disk formats. Without them a corpus prepared in one framework would have to be rewritten to be read by another.
Other pages in this map that name Data processing frameworks as something they cannot do without.
What each company supplies at this step, and — where a public figure exists — its share of this specific market — with what that share measures, the period it covers and who published it. Some rows also show the company’s own reported revenue for the segment covering this step, which is a different thing: it says how much this business matters to that company, not how much of the market it holds. Not a ranking and not a recommendation.
Provides managed large-scale data processing and storage platforms.
Operates managed batch and streaming data processing services.
Operates managed distributed processing and object storage services.
Commercialises the dominant open-source distributed processing engine.
Runs a managed analytics and processing stack, and is the largest hosted home for the open engines.
Commercialises the distributed compute framework most of these pipelines use to fan work across a cluster.
Maintains the on-premises distribution of the batch stack for operators who cannot move the data.
Supplies the dataframe and loader libraries that move parts of this work off the CPU and onto the accelerator.
Runs the warehouse a large share of enterprise corpora are already sitting in before any of this starts.
Operates the batch and stream platform most Chinese pipelines run on, and maintains the stream engine underneath it.
Supplies the dataset and deduplication tooling most open corpora are built with.
Supplies the on-premises data platform, and the format work behind open table storage.
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.
Plutux is not an investment adviser. Market data and AI-generated analysis are for information and education only, not investment advice. Disclaimer