BM25 Beats Dense Retrieval at Scale, by Nearly 20 Points

The standard assumption in production RAG design is that dense retrieval beats lexical search once the corpus gets large enough and semantics matter. This study runs 28 nested corpus tiers spanning a 450-fold size range and finds the opposite: BM25 becomes the dominant retrieval method as corpus size grows, not despite scale, but because of it.

The mechanism is a scale-dependent crossover, not a static ranking. At the smallest corpus tiers, a File-System Agent wins: it explores documents sequentially, reasoning its way through a small search space. That strategy works when the space is bounded. As the corpus expands, sequential exploration becomes a liability. The agent spends 39 times more query tokens at the bedrock tier compared to BM25, and its accuracy degrades as the search space widens. BM25, by contrast, performs global candidate ranking across the entire corpus in a single pass. That architectural difference, local sequential search versus global lexical ranking, is what drives the crossover.

Dense retrieval sits in the middle: more efficient than agentic search at large scale, but consistently less accurate than BM25 at shared tiers. Graph-based RAG hits construction cost walls before reaching deployment scale, and its scalable variants stay below BM25 across every shared tier in the study. Around 10 million corpus tokens, BM25 takes the lead and holds it, with the accuracy margin approaching 20 points at full scale. For ML infrastructure teams building or re-evaluating RAG pipelines, the takeaway is direct: if the corpus is large, embedding infrastructure is not buying accuracy, it is buying cost.

We're thinking: We read this as a quiet indictment of a widespread infrastructure assumption. Teams that adopted dense retrieval because it felt more modern than BM25 now have a controlled, 28-tier scaling study to weigh against that choice. The result is not that dense retrieval is wrong at small scale. It is that the accuracy crossover happens well before most production corpora reach their ceiling, meaning the embedding pipeline, the indexing cost, and the query latency are all overhead for a retrieval method that is already losing to BM25 by the time the corpus is meaningfully large. The more actionable read: agentic reasoning belongs after ranked retrieval, not instead of it.

Key takeaways:

• BM25's global candidate ranking scales with corpus growth in a way that sequential agentic exploration and dense vector similarity do not, producing a structural accuracy advantage above 10 million corpus tokens.
• BM25 leads every shared tier above the crossover point, with a margin approaching 20 accuracy points at full scale, across 28 strictly nested corpus tiers and a 450-fold size range; the caveat is that the study holds the reader model and judging protocol fixed, so results may shift with different LLM backbones.
• Teams running dense retrieval at corpus sizes above 10 million tokens should benchmark BM25 directly against their current pipeline before the next infrastructure renewal cycle.

Source: BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms