Synthetic Data Won't Collapse Your Model — Sloppy Data Will
Synthetic data — text and examples generated by one model to train another — has become a core input at frontier AI labs as the supply of fresh, high-quality human text runs thin. Researchers are split between two camps: synthetic data as the fix for the looming data wall, and model collapse as the hidden cost of models increasingly training on machine-generated output. The split matters now because the labs pulling ahead aren't avoiding synthetic data, they're building verification layers around it, while smaller players risk quietly degrading their models by skipping that step. For PMs, the real exposure isn't in the lab's training pipeline — it's in whether you're watching your own vendor's outputs for the early signs of homogenization.
Ask yourself this: if your model vendor's next update was trained partly on data generated by its own previous model, would you even notice if quality quietly regressed on the edge cases your product depends on? Most PMs would not. You track uptime, latency, and maybe a handful of eval scores your vendor publishes. You don't track whether rare-but-correct answers are quietly disappearing in favor of blander, more "average" outputs — which is exactly what unverified synthetic training produces first. This week, do one concrete thing: pull 90 days of logs for your two or three lowest-confidence use cases, the ones where correctness depends on unusual inputs rather than common ones, and snapshot the outputs. Diff that snapshot against the next model version your vendor ships. You're looking for narrowing, not just errors — answers converging toward the same safe phrasing, tail-case handling getting worse even as benchmark scores stay flat or improve. To be fair, the major labs are more careful about this than the discourse suggests; nobody wants to ship a visibly worse model. But you don't have visibility into their training pipeline, only their outputs, and no lab publishes an independent collapse audit for you to check. Build the regression habit now. The alternative is finding out about a quality regression from your own users, in production, after it's already your incident.
Frontier labs including Microsoft (Phi) and Meta (Llama) already train on large volumes of synthetic data by design — the live debate is about verification, not whether to use it at all.
Picture a frontier lab in mid-2026, six weeks from a flagship model launch. Compute isn't the bottleneck. Headcount isn't the bottleneck.
The bottleneck is finding a few trillion more tokens of text that no other lab has already scraped and, increasingly, that no other model already wrote.
That's the real backdrop to the synthetic data debate, and here's my verdict: synthetic data is neither the crisis some researchers describe nor the free lunch labs would like it to be. It's a tool that works when it's verified and mixed with a growing pool of real data, and fails when it's used to cut corners. It's not synthetic versus real that decides the outcome; it's curated versus uncurated.
The Wall Every Lab Hit at Roughly the Same Time
Scaling laws reward more tokens, but the internet's stock of high-quality, human-authored text is finite, and labs have been pulling from largely the same crawlable and licensable corpus for years. Researchers at Epoch AI, in an analysis that's become a standard reference point, projected the usable stock of public human-generated text could be effectively exhausted sometime between the late 2020s and early 2030s at current scaling rates.
That's the "data wall": not that text runs out overnight, but that the marginal ton of fresh, high-quality, never-before-trained-on human text gets scarcer and more expensive every year. Labs responded the way any resource-constrained industry does. They started manufacturing the input they couldn't buy enough of, generating training examples with existing models rather than waiting for humans to write more.
Two Narratives, One Real Mechanism
The optimistic case: a strong model can generate textbook-quality explanations, worked math problems, or code with unit tests, faster and cheaper than sourcing equivalent human content, and at far greater volume. Microsoft's Phi model line built its whole pretraining pitch on this — small models punching above their weight because their diet is curated "textbook quality" synthetic content rather than raw web scrape, a strategy Microsoft has documented publicly since Phi-1.
The pessimistic case has a name and a mechanism: model collapse, formalized in a widely cited 2024 Nature paper by Shumailov and colleagues, which showed that models trained recursively on their own outputs across generations progressively lose the rare, tail-end patterns in the original distribution and converge toward blander, more repetitive output. Think of it like reducing a sauce using only the sauce itself as liquid, generation after generation, with no fresh stock added. You don't get a new flavor.
You get a more concentrated version of whatever was already in the pot, minus anything delicate that couldn't survive the reduction.
To be fair to the collapse researchers, the mechanism is real and reproducible in controlled experiments where synthetic output fully replaces, rather than supplements, real data across generations. That's not a strawman. It's a legitimate finding that any lab quietly training generation N+1 on generation N's uncurated output should take seriously.
But the cleanest rebuttal to "collapse is inevitable" is also the most practical one: later studies found that when synthetic data accumulates alongside an ever-growing pool of real data — rather than replacing it — the degeneration slows dramatically or doesn't show up at all over the ranges tested. That's the difference between reducing the sauce forever and adding a splash of fresh stock every time you reduce it. The dish keeps its character.
It might even get better.
Why the Frontier Labs Treat This as Mostly Solved
Here's what the collapse-versus-solution framing misses: nearly every frontier lab already uses synthetic data at meaningful scale, and has said so. Meta's Llama 3 technical report disclosed heavy reliance on synthetic data in post-training, generated and then filtered through reward models and rejection sampling rather than used raw. The reasoning-model wave — OpenAI's o-series, and comparable efforts at Google and Anthropic — leans on synthetic chains of thought verified against ground truth in domains where correctness is checkable by a compiler or a proof checker, not just plausible-sounding.
That's the pattern worth naming: the labs pulling ahead aren't the ones avoiding synthetic data. They're the ones treating verification as the actual product. A math problem's synthetic solution is worth training on if you ran it and got the right answer.
An essay a model wrote about itself, with no external check, is worth much less — and if you keep feeding those back in, you're the lab reducing the sauce with nothing new added.
The Next 12 to 24 Months
Here's my falsifiable prediction: by the end of 2027, no frontier lab will report a flagship training run meaningfully bottlenecked by "we ran out of real data." Synthetic pipelines with verification — code execution, formal proof checkers, human preference data, reward models — will have made the data wall a solved engineering problem for the handful of labs with the resources to build that verification layer well. Model collapse won't disappear as a phenomenon, but it will show up mainly in smaller, cost-constrained, or open-source efforts that scrape synthetic content indiscriminately without an equivalent verification budget.
The historical parallel is the shift from hand-tuned assembly to optimizing compilers: nobody worried that a compiler compiling itself would degrade output, because correctness was checked against an external spec at every generation, not against the compiler's own opinion of itself. Watch for the tell: if a top-tier 2027 flagship model ships with measurably worse calibration or factuality than its predecessor and the cause traces to synthetic contamination, the pessimists win the argument. I don't expect that.
I expect the casualties to be undifferentiated smaller models trained on scraped synthetic exhaust with no verification step at all.
The harder question isn't whether synthetic data works. It's whether the verification layer — human raters, reward models, executable checkers — quietly becomes more valuable than the base model or the compute underneath it, and who ends up owning that layer.
Get this in your inbox. AI Rundown Daily delivers original briefings every morning — free. Subscribe →
Frequently Asked Questions
You're right to be skeptical, and no lab publishes an independent collapse audit you can check before a release. What's known is the mechanism: recursive training on unverified, self-generated data degrades tail knowledge, per the widely cited 2024 Nature study. In practice, major labs mitigate this by mixing synthetic with an accumulating real-data pool and verifying synthetic examples against ground truth in checkable domains like math and code, which is a materially different setup than the pure-replacement scenario collapse research warns about. Your best defense isn't trusting the label — it's running your own before-and-after regression checks on releases that matter to your product.
Indirectly, yes. Cheaper synthetic data generation lowers labs' variable costs for building instruction-tuning and evaluation datasets, which is one contributor among several to the broader decline in cost-per-token seen across the market. If you're fine-tuning your own model, generating synthetic instruction or eval data in-house is now a legitimate, low-cost option — but only if you pair it with a verification step or a real-data holdout set, otherwise you inherit the same narrowing risk at a smaller scale. Budget for the verification tooling, not just the generation.
Without diversity checks, you get a smaller-scale version of model collapse: outputs narrow toward the most probable phrasing, and edge cases your synthetic generator rarely produced get worse or disappear. This is easy to miss because your benchmark scores can hold steady while your worst-case behavior quietly degrades. The fix is straightforward if unglamorous — keep accumulating real user data rather than swapping it out, hold out a real-data eval set the synthetic pipeline never touches, and watch for calibration drift over successive fine-tuning rounds, not just accuracy.
