There is no universally best encoder: ModernBERT is the recipe, but sample efficiency, compute-optimal training and honest benchmarks each tell a different winner
Asked (summary):
Important papers from the last three years on how to build a proper modern encoder model like ModernBERT or similar, and how encoders are compared against each other.
A curated reading guide of 26 papers published 2023-12-29 through 2026-07-04, spanning four threads: architecture & MLM pretraining (8 papers), embedding training (9), decoder-to-encoder alternatives (5) and evaluation benchmarks (4). Sources span 8 hosts — arXiv, ACL Anthology, AlphaXiv, Hugging Face, Jina, and others; one secondary article backs two rows (Gecko and NV-Embed), every other URL backs one paper.
Hover or tap a marker for the paper's headline finding; every paper is also linked in the table below.
What the timeline cannot say
OptiBERT-239M, trained compute-optimally on 41B tokens, beats both NeoBERT and ModernBERT on MTEB with 50× less pretraining compute — scale is not the only lever. aclanthology.org
A controlled study pretraining ModernBERT on the same data as CamemBERTaV2 finds DeBERTaV3 still more sample-efficient; ModernBERT's real advantages are 8,192-token context and speed. alphaxiv.org
Ettin's paired encoder/decoder suite (17M–1B, 2T open tokens) gives the first clean controls: Ettin-Enc-1B hits 91.6 GLUE vs 90.4 for ModernBERT-large. alphaxiv.org
PosIR shows retrieval on documents over 1,536 tokens correlates poorly with MMTEB, and position bias grows with document length — short-text averages hide long-context failure. alphaxiv.org
How to compare encoders fairly
Match the budget
Compare at matched parameter count, pretraining FLOPs/tokens, data and fine-tuning recipe — OptiBERT and the ModernBERT-vs-DeBERTaV3 study show conclusions flip when these are controlled.
Report the costs
Always state latency, throughput, VRAM, context length and embedding dimension alongside quality; ModernBERT's headline strengths are efficiency ones.
Separate the regimes
Frozen probes, full fine-tuning and contrastive embedding tuning rank models differently — NeoBERT's MTEB win holds under identical fine-tuning, not out of the box.
Break the average apart
Use per-task and per-language results, not one MTEB average; MMTEB's Borda ranking rewards consistency across 500+ tasks and 250+ languages.
Test true long documents
Use target-position controls (PosIR) and reasoning-intensive, out-of-domain retrieval (MAIR); short-text scores predict neither.
Check reproducibility
Prefer papers with open data, checkpoints and code (Ettin, NeoBERT, Nomic), and watch for contamination and licensing limits.
Recommended reading order
Core encoder construction — MosaicBERT (throughput-oriented pretraining) → ModernBERT (modern components, 2T tokens, native 8,192 context) → NeoBERT → OptiBERT (compute-optimal MLM) → CrossBERT (objective redesign); add mmBERT for multilingual pretraining and Ettin for paired encoder/decoder controls.
Embedding specialization — Nomic Embed, BGE-M3/M3-Embedding, mGTE, jina-embeddings v2/v3/v5, sparse-MoE Nomic Embed v2, plus LLM distillation (Gecko); the shared recipe is contrastive training with hard and in-batch negatives, heavy data curation or LLM-generated data, long-context positional design, and task adapters or Matryoshka objectives.
Decoder-derived encoders — LLM2Vec (bidirectional attention + MNTP + contrastive), GritLM (one model for generation and embedding), NV-Embed (latent attention, two-stage tuning), BGE-Multilingual-Gemma2, LLM2Vec-Gen.
Evaluation — MMTEB → LongEmbed-style long-context probes via PosIR → MAIR (instructed retrieval) → HAKARI-Bench (cheap proxy, Spearman > 0.97 with full suites, but not a replacement).
All 26 papers
Date
Artifact
Paper
Headline finding or warning
Source
Curated reading guide: 26 papers on encoder-only Transformer construction, embedding training, decoder-derived encoders and evaluation, dated 2023-12-29 to 2026-07-04, one row per paper across 8 source hosts; dates are publication or latest-revision dates from each source. Findings are quoted or condensed from each paper's own claims; recipe details and baseline lists are shortened for space in the table — hover the timeline for headline findings.