Data Curation and Dedup: Why Less Data Beats More of It
Data curation and deduplication — not architecture tweaks — are what separate a model that generalizes from one that memorizes noise. FineWeb and Llama 3 both converged on filtering, near-duplicate removal, and deliberate domain mixing as the real quality lever behind their training corpora. The signal: labs are spending their engineering hours on editorial pipelines, not just bigger scrapes. For PMs, the same three levers — quality scoring, dedup, and mixing ratio — apply directly to whatever fine-tuning dataset your team is building right now.
Ask yourself this: when was the last time anyone on your team actually looked at your fine-tuning data, row by row, instead of just measuring how many rows it has? If the honest answer is never, you have a curation gap, not a data gap. This week's action: pull a 200-row sample of whatever corpus you're using for fine-tuning or RAG and manually check for near-duplicates — same support ticket rephrased, same doc chunk repeated across pages, same boilerplate template with the names swapped. If you find more than 10-15% duplication in that sample, assume it's systemic across the full set, and run a dedup pass before you spend another dollar on training compute or eval runs. To be fair, this isn't free. Someone has to build or buy the tooling, and it's the kind of work that never shows up in a demo, so it's easy to deprioritize behind a shinier feature. But every dollar spent training on duplicate content is a dollar not spent on genuinely new signal, and every hour saved re-running evals because your corpus was clean is an hour your team gets back. Do this before your next fine-tuning cycle, not after the eval scores come back flat and someone asks why.
FineWeb's public corpus uses MinHash deduplication with 112 hash functions across 96 CommonCrawl snapshots, yielding roughly 15-18.5 trillion clean tokens from a far larger raw crawl.
Hugging Face's FineWeb dataset draws from 96 CommonCrawl snapshots stretching back to 2013. After filtering and deduplication, what's left is roughly 15 trillion tokens — later expanded past 18.5 trillion as the pipeline improved. That's the number labs actually train on.
The raw crawl behind it is far larger and mostly unusable.
- boilerplate
- spam
- mirrored press releases
- pages that say nothing twice
Here's the verdict: your model's ceiling isn't set by parameter count or which attention variant your team debated for a quarter. It's set by what you let through the filter before a single gradient update runs. Data curation and deduplication are boring, invisible in a demo, and they move eval numbers more than most architecture decisions ever will.
If you're building a fine-tuning dataset and skipping this step, you're leaving quality on the table you'll never get back by adding a bigger base model.
The Filter Is the Product, Not the Cleanup Step
Meta's own account of Llama 3's pretraining data makes the point plainly: the team built text-quality classifiers, trained in part using Llama 2 itself to label examples, then layered heuristic filters and NSFW detection on top before ever touching the mixing ratios. That's not preprocessing. That's the actual engineering.
Think about it like a hiring funnel. Feeding a model an unfiltered web scrape is like handing a resume pipeline every email that lands in the inbox — spam, duplicates, and all — and expecting it to get sharper at spotting good candidates. It doesn't get sharper.
It gets slower and more confused, because now the signal has to fight through noise nobody bothered to sort.
The lesson from FineWeb, from Llama 3, from basically every serious pretraining writeup published in the last two years is consistent: it's not a data-volume problem, it's an editorial problem. More raw tokens without a filter just means more garbage at higher throughput.
Deduplication Is Where the Money Actually Goes
Web crawls are full of near-duplicate content — mirrored articles, templated boilerplate, the same press release syndicated across a dozen sites. Exact-match dedup catches almost none of that, because the copies aren't identical, just close.
FineWeb's public documentation gets specific about the fix: MinHash deduplication using 5-grams and 112 hash functions, with document signatures split into buckets so near-duplicate pages get caught even when the text isn't byte-for-byte the same, per Hugging Face's dataset writeup. This is the unglamorous machinery — locality-sensitive hashing, bucket comparisons, signature matching — that determines whether your model memorizes the same paragraph forty times or actually generalizes from forty distinct ones.
Skip it and you pay twice. Compute gets spent re-training on content the model already saw, and evaluation benchmarks risk contamination when duplicate training text overlaps with test sets. Neither shows up on a roadmap slide.
Both show up in your eval scores three months later, and by then it's a re-training bill, not a code review comment.
The Mixing Ratio Nobody Talks About
Even a perfectly deduplicated, perfectly filtered corpus still has a knob left: how much of it should be code versus books versus web text versus Wikipedia. Get that ratio wrong and you can out-tune your way into a model that's great at one thing and mediocre everywhere else.
This is where DoReMi, a technique from Xie et al. published at NeurIPS, earns its keep. It trains a small proxy model to find better domain weights through group distributionally robust optimization, then applies those weights to the full-size run. Reported results: roughly 6.5 points of average few-shot accuracy improvement over default domain weights, and matching baseline accuracy in 2.6x fewer training steps — using a proxy just 1/30th the size of the final model.
To be fair to the scale-first camp: no amount of curation turns a small corpus into a frontier model. Compute and token count still set the outer bound, and a perfectly filtered ten-billion-token dataset won't out-argue a trillion-token one. Curation raises the ceiling of what your data can do; it doesn't replace the data.
What This Means for Your Fine-Tuning Dataset
You're probably not training a frontier model from scratch. You're fine-tuning on support tickets, internal docs, or a scraped corpus of domain content — and the same three levers apply at your scale. Score examples for quality before you dump them in a training run.
Deduplicate your support transcripts; the same customer complaint copy-pasted across forty tickets is not forty training examples, it's one example forty times. And check your domain mix deliberately instead of letting whatever you happened to scrape most of dominate the ratio.
None of this requires FineWeb-scale infrastructure. A basic MinHash implementation, a handful of open-source Python libraries, and an afternoon of scripting will catch the bulk of near-duplicate rows in a corpus of a few hundred thousand examples. The quality-classifier step can start even simpler: a short rubric and a sample review, not a trained model, is often enough to catch the worst offenders before they ever reach a training run.
The point isn't sophistication for its own sake. It's making sure someone actually looked.
The harder question for your team isn't whether curation matters. It's whether you're willing to spend real engineering hours on the boring 80% — filtering, dedup, quality scoring — instead of the exciting 20% everyone wants to work on, like prompt templates and fine-tuning hyperparameters. The labs that treat curation as the product, not the cleanup step, are the ones shipping models that actually hold up outside the demo.
Get this in your inbox. AI Rundown Daily delivers original briefings every morning — free. Subscribe →
Frequently Asked Questions
Big labs solved it for their own web-scale corpora, not for your support tickets, internal docs, or scraped niche content. Your data has its own duplication patterns and quality problems that no open dataset pipeline touches. The techniques — near-duplicate detection, quality scoring, deliberate domain mixing — transfer down to a 10,000-row fine-tuning set even though the tooling has to be built at your scale, not theirs.
For a corpus in the thousands-to-low-millions of rows, expect engineering time more than infrastructure spend: someone building a dedup script using existing open-source hashing libraries, plus a lightweight quality classifier or even a rules-based filter. That's days, not months, for most teams. The larger cost is organizational — someone has to own data quality as a real workstream instead of an afterthought squeezed in before a training run.
Over-filtering can strip out legitimate edge cases and minority-pattern examples your model needs to handle real-world variance, leaving you with a corpus that's clean but narrow. Aggressive semantic dedup can also collapse genuinely different examples that just share vocabulary. Test filter thresholds against a held-out eval set before committing, and keep a raw, unfiltered copy of your source data so you can walk back a bad filtering decision.
