Curation beats volume: 63 methods show LLM data work shifting from corpus recipes to learned, model-aware selection
Asked:
“get me list of LLM Training Data: Cleaning, Deduplication, Mixing, and Selection (2022–2026)
all sota and github link and models (LLMS) where it was used .and thoughts”
A best-effort state-of-the-art landscape current to 2026-09-01, not a mathematically exhaustive list: a diversified web sweep across 42 search angles and thousands of result pages, consolidated to 63 high-signal methods, open pipelines, benchmarks, and corpus recipes spanning 2022–2026 — corpus recipes, curation pipelines, deduplication, mixture optimization, pretraining selection, and instruction-data selection.
Timeline of the field, 2022–2026
filled dot = verified GitHub repositoryhollow dot = no official repo verifieddot size = source mentions (discovery support, not quality)
SlimPajama drew the widest discovery support of any single item (90 source mentions), yet mentions measure visibility, not quality — cerebras.ai
2024 is the peak year with 24 of the 63 items, led by open corpus recipes (Dolma, FineWeb, DCLM) and a first wave of learned selection (MATES, QuRating, LESS) — alphaxiv.org
Mixture optimization moved from fixed heuristics (DoReMi, 2023) to regression and scaling laws (RegMix, Data Mixing Laws) and online control (DeMix, Olmix) by 2025–26 — github.com
2025–26 entries turn model-aware, multilingual and scale-aware — Group-MATES, MuRating, JQL, AutoScale, BLISS — but all five 2026 items are emerging and thinly documented — arxiv.org
Analyst thoughts
There is no universal winner. A strong 2026 practical stack layers deterministic extraction and normalization, exact + MinHash/LSH near-deduplication, selective semantic deduplication, language/PII/safety filters, learned quality scoring, benchmark decontamination, then mixture optimization.
The strongest established starting points are FineWeb/DataTrove, Dolma, DCLM, NeMo Curator, SemDeDup/FED, DoReMi/DoGE/RegMix, DSIR/QuRating/MATES, and LESS/DEITA for instruction tuning; choose by stage and budget rather than one global leaderboard.
Learned quality filters often produce larger gains than adding more unfiltered tokens, but optimize toward the judge's taste and can erase minority languages, informal registers, or novel domains. Preserve raw lineage and audit acceptance rates by language/domain.
Deduplication is multi-objective: it improves efficiency and reduces memorization/contamination, but aggressive semantic dedup can remove legitimate recurring facts and rare formulations. Measure both retained diversity and downstream loss.
Mixture optimization is moving from fixed heuristics to proxy models, regression/scaling laws, bilevel gradients, and online control. The main unresolved risk is proxy-to-target transfer across model size, token budget, tokenizer, and training stage.
"SOTA" should mean Pareto performance across quality, compute, reproducibility, multilingual coverage, legal provenance, privacy, and downstream targets — not only benchmark accuracy.
Recommended evaluation protocol: fixed architecture and token budget; ablate each curation stage; report tokens removed and compute spent; audit language/domain/license distributions; use held-out clean benchmarks; test memorization and contamination; repeat proxy-selected mixtures at target scale.
All 63 items
"Models used / evaluated (reported)" mixes two evidence types because public documentation is uneven: for corpus and pipeline rows it often names a model documented as trained with the corpus or tool; for research-method rows it usually names the models used in the original experiments. It is not proof of secret or production adoption; a blank cell means the source did not establish a model. A blank GitHub cell means no official repository was verified in the collected sources.
Name
Year
Area
GitHub
Models used / evaluated (reported)
What it does
Mentions
Source
Curated shortlist of 63 methods, pipelines, corpora and benchmarks for LLM training-data cleaning, deduplication, mixing and selection, 2022–2026, consolidated from a diversified web sweep current to 2026-09-01. Year is the best-supported first paper or release year and may differ between preprint, conference, and software release; 2026 entries are especially emerging. Source mentions count discovery support across the sweep, not quality — do not rank methods by mentions alone. Assessment text is shown in tooltips and shortened from row notes for space.