Sub-system
Every claim about a model rests on a measurement, and measuring an open-ended generator is genuinely hard. Evaluation has become its own discipline and its own product category, and it is the machinery both training and deployment decisions depend on.
In one sentence
Model evaluation is the systematic measurement of a model's behaviour — against fixed benchmarks, against human or model judgement, or against programmatic verifiers that can check an answer automatically.
Three families of measurement are used, and they fail in different ways. Fixed benchmarks with known answers are cheap and repeatable, and they leak into training corpora and stop discriminating. Human preference comparison measures what people actually want and is slow, expensive and noisy. Model-as-judge scales the second and inherits the judge's biases, including a documented preference for its own style of output.
The fourth and increasingly important case is verifiable: tasks where a program can decide correctness — code that compiles and passes tests, a mathematical result, a structured output that validates against a schema. That removes the human throughput limit entirely, which is why it has become the backbone of both evaluation and reinforcement learning in reasoning-heavy domains.
A benchmark published on the internet ends up in the next model's training data, after which a high score means the model memorised the answers. Detecting it is imperfect, and the responses — held-out private sets, continuously refreshed questions, contamination checks against the corpus — are mitigations rather than fixes.
Verifying generated code means executing it, which means running untrusted output at scale with isolation, resource limits and no network access. That sandbox is real infrastructure with real security requirements, and its throughput bounds how much verifiable training signal can be generated.
A benchmark says whether a model is good in general; a deployment needs to know whether it is good at this task, on this distribution, after this prompt change. That has produced a tooling category around capturing production traces, curating them into task-specific suites, and regression-testing every change — which is closer to software testing than to research benchmarking.
1 of these is marked as a chokepoint: a handful of qualified suppliers, a multi-year lead time, or a single geography.
Anything not machine-checkable needs qualified people to judge it, and at frontier capability that means practitioners.
Human data operationsRunning generated code to check it requires sandboxes with hard isolation and resource limits, at scale.
Container platformsAn evaluation is thousands of generations against a checkpoint, repeated for every variant considered. Without a serving path fast enough to do that in hours, evaluation becomes the slow step in the research loop.
Serving enginesOther pages in this map that name Model evaluation as something they cannot do without.
What each company supplies at this step, and — where a public figure exists — its share of this specific market — with what that share measures, the period it covers and who published it. Some rows also show the company’s own reported revenue for the segment covering this step, which is a different thing: it says how much this business matters to that company, not how much of the market it holds. Not a ranking and not a recommendation.
Provides evaluation, red-teaming and human judgement services to model developers.
Operates evaluation and monitoring tooling within its AI platform.
Develops evaluation methodology and platform tooling for its own and hosted models.
Provides evaluation and monitoring tooling for deployed models.
Hosts the public leaderboards and benchmark datasets that most published comparisons are run against.
Supplies the trained human raters behind preference and correctness judgements that no automated metric replaces.
Sells the tracing and evaluation tooling teams use to score their own applications rather than the base model.
Sells the production monitoring side of the same problem — what the model did on live traffic, not on a benchmark.
Publishes the benchmarks and pre-deployment evaluations its own releases are measured against.
Publishes the evaluation and model-card methodology much of the field's safety testing is patterned on.
Runs the blind pairwise vote most public claims about which model is better are ultimately citing.
Builds benchmarks held out of training data, and tracks the compute behind each run.
Measures price, latency and quality across hosted endpoints, so the comparison is of what is served rather than what was published.
Owns Weights & Biases, where most teams' internal evaluation history actually lives.
Whether private held-out evaluation becomes standard as public benchmarks saturate and leak.
Whether model-as-judge is shown reliable enough to replace human comparison at scale.
Whether verifiable-task evaluation extends beyond domains with automatic checkers.
Each page explains one technology in plain language, states what it depends on, and names companies by what they supply at that step. Company roles are described qualitatively and deliberately carry no market shares, revenue figures or rankings — those change faster than an explainer can, and a stale number is worse than none. Ticker links point at company pages on this site and are provided for reference only.
Nothing here is investment advice, a recommendation, or a forecast. A company named on a page about a technology is not thereby a good investment, and the chokepoints described are structural facts about supply chains rather than predictions about prices. Technology moves; where a page describes something as unresolved or in development, that was true when it was written.
Plutux is not an investment adviser. Market data and AI-generated analysis are for information and education only, not investment advice. Disclaimer