Spend compute on more good tokens first — the rest is stabilizers, checkpoints, and a few advanced options
Asked:
“tricks to train small transformers”
Eleven evidence-backed techniques for training small transformer language models, one per primary paper claim, distilled from a scan of 2,379 deduplicated search results across 12 queries and 404 extracted candidate rows. All shortlist sources are arXiv-hosted papers; no single URL supplies more than 18.2% of the entries.
The priority ladder: five tiers, top three carry the return
Core, do by defaultAdvanced or conditional — adopt only when the condition appliesHover or tap a technique for its action and caveat
What the ladder cannot say
A 400-model sweep (70M–16B parameters, 5–500B tokens) shows size and tokens should scale in equal proportion — prior large models were significantly undertrained. ar5iv.labs.arxiv.org
TinyStories gets sub-10M-parameter models to fluent, grammatical stories — at the cost of a 3–4-year-old’s vocabulary and knowledge. arxiv.org
When inference demand is heavy, training a smaller model on more tokens than Chinchilla-optimal lowers lifetime training-plus-inference cost — but you must estimate demand before training. arxiv.org
Overtrained models are stronger under repeated-sampling test-time compute, yet harder to fine-tune afterwards — a real trade, not a free win. arxiv.org
Starter recipe, straight from the shortlist
Pick the smallest model your task allows, then scale tokens with parameters in equal proportion; a quality web corpus like FineWeb at a Chinchilla-optimal budget is the reference point. If you expect heavy inference, shift smaller-and-longer.
Keep the decoder-only architecture conventional and stabilize it: AdamW (β1 0.9, β2 0.95), weight decay 0.1, gradient clipping 1.0, warmup of 1,000,000 tokens, learning rate 4e-4 decayed to 4e-5 — the reproducible small-Llama configuration in the shortlist. It was tuned for reproducibility under constrained compute, not benchmark records.
Track validation loss at token intervals and stop at the best observed point rather than exhausting the budget; fix capability gaps with multi-stage data rebalancing instead of new full runs.
Only then consider advanced scale-transfer methods (u-µP, Muon learning-rate transfer, overtraining for test-time scaling) and deployment tactics (QLoRA — a fine-tuning and memory tactic, not from-scratch pretraining).
All 11 techniques
Tier
Technique
Do this
Why it helps
Caveat
Source
Shortlist of 11 techniques for training small transformer language models, one row per technique with action, rationale, caveat, and primary-paper URL; distilled November-scan pipeline: 2,379 URL-deduplicated results from 12 queries, 404 extracted candidate rows, normalized and filtered to 11. Sources span 2 URL hosts (both arXiv mirrors) across multiple distinct papers; largest single-URL share 18.2%. Long paper titles shortened in the ladder; full titles in the table.