A new paper (arXiv 2607.26497) argues there is no unconditional best way to do retrieval for RAG: the winner depends on how big the corpus is. Dense embeddings and clever agents lead on small collections, but as the corpus grows the plain old keyword baseline, BM25, catches and passes them, for free. We rebuilt the core of that experiment on a single CPU in an evening, and the crossover showed up exactly where the paper said it would.
What we did
The paper's method is a controlled scaling study: hold the questions and a fixed bedrock of relevant documents constant, then grow the corpus around them across 28 nested tiers spanning roughly 450 times, and measure how each retrieval paradigm holds up. We reproduced the mechanism at a smaller, honest scale. Our questions were the 300 SciFact test queries from the BEIR benchmark, and their 283 gold documents are the bedrock, present at every single tier. We then grew the corpus by adding distractors in a fixed shuffled order, 34,000 out-of-domain FiQA financial documents plus the SciFact non-gold set, so each tier is a strict superset of the one before it. Seven tiers span 65,000 to 5.6 million word tokens, about 87 times.
At every tier we ran two retrievers over the same documents: BM25 (lexical keyword matching) and a dense retriever (the all-MiniLM-L6-v2 embedding model, cosine similarity). We embedded all 39,183 documents once and cached them, then measured answerable accuracy at 10, nDCG at 10, recall and latency. Four scripts, all seeded and deterministic.
Why it was worth doing
Because the finding held, and it is counter to the reflex. At the smallest tier dense retrieval led BM25 by 4 points of accuracy (0.950 against 0.910) and 3 points of nDCG. That is the reflex: reach for embeddings, they win. But as the corpus grew, the dense lead eroded monotonically. By 5.6 million tokens the accuracy gap had collapsed to 0.003, a dead heat, and on nDCG at 10 BM25 actually crossed over and led at 3 million tokens (0.698 against 0.694). Same questions, same gold documents, the only thing that changed was how much hay we buried the needles in, and that alone flipped the ranking.
The reason is the cost side, which is the paper's real point. BM25 needs no construction: no model, no embeddings, no tokens spent building an index. Our dense arm spent 580 seconds embedding the corpus once; BM25 built in under a second and cost nothing. So BM25 does not merely catch dense at scale, it does so from the cheap end of the Pareto frontier. When we add the agentic paradigm's cost curve from the paper (a file-system agent whose query-token cost starts at 39 times BM25's and climbs with the search space), the picture is complete: the fancy methods win small and lose big, on both accuracy and cost.
What's still off
This is a scaled-down reproduction, and we are saying so plainly. Our 87 times range reaches the crossover in ranking quality and sits right at the cusp in raw hit-rate; the paper's full crossover lands nearer 10 million tokens, just above where our single-CPU corpus tops out. We measured BM25 against dense honestly; the agentic arm is the paper's cost model plotted alongside, not something we re-ran ourselves, and it is labelled as such. One more caveat worth flagging: our BM25 query latency grew to 217ms at the top tier, but that is an artefact of a naive Python implementation with no inverted index. A production BM25 (Lucene, Tantivy) answers in well under a millisecond, so the latency line is not a mark against the method.
What's now in the stack
- A four-script harness (prep, embed, eval, chart) that reproduces the BM25-versus-dense scaling crossover on BEIR data on one CPU, seeded and rerunnable.
- A number for our own retrieval work: for keyword-heavy search over a large, heterogeneous index, a strong BM25 baseline is a Pareto-optimal default, not a fallback.
- A rule of thumb for the memory and document indexes we build: expect the dense lead to erode as the index grows and the distractor tail lengthens, and price the embedding construction in before you reach for it.