Your Retrieval Pipeline's Biggest Bottleneck Is the Part You Never Tune

The standard retrieval improvement playbook runs in a predictable order: swap in a dense retriever, add a reranker, tune embedding dimensions, repeat. The assumption underneath all of it is that the documents themselves are a fixed input. AutoIndex runs systematic experiments across eight heterogeneous retrieval tasks and finds that assumption is the bottleneck, not the retriever.

The core insight is architectural. Instead of treating document preprocessing as a one-time engineering decision made before retrieval begins, AutoIndex treats it as an optimization target with its own search space. The framework generates executable programs that transform raw documents before indexing: slicing long documents into targeted chunks, enriching content with synthesized context, normalizing inconsistent formatting, reweighting fields by retrieval signal, or reorganizing structure entirely. Think of it as a compiler pass for your corpus, where the compiler is guided by retrieval feedback rather than human intuition.

Each iteration of AutoIndex runs a validation-guided search loop. Agents inspect where the current program fails on a held-out validation set, propose candidate modifications, and keep only the changes that produce measurable recall gains under the updated index. BM25 stays fixed throughout. The retriever never changes. Only the representation of the documents changes, and that representation is now a first-class artifact that the system learns rather than the practitioner guesses.

The numbers are hard to dismiss. Across all eight tasks on the CRUMB benchmark, learned programs improve over a static full-document BM25 baseline. Average gains land at +8.4% in Recall@100 and +8.3% in nDCG@10. On the hardest tasks, the ceiling lifts substantially: +30.5% in Recall@100 and +43.6% in nDCG@10. These are not marginal improvements from hyperparameter nudges. They come from changing what the retriever sees, not how it sees it. For teams running BM25 or any lexical retrieval system in production, the takeaway is direct: before adding retrieval infrastructure, audit whether your document representation was ever actually optimized for retrieval.

We're thinking: We find the framing here more consequential than any individual benchmark number. The retrieval community has spent years optimizing the retriever while treating document preprocessing as a solved or at least settled problem. AutoIndex makes the case that this ordering is backwards, or at minimum incomplete. The practical implication for teams is uncomfortable: every chunking strategy, every normalization decision, every field-weighting choice you made at index-build time was probably made once and never revisited against retrieval signal. AutoIndex's agent-driven search loop is one approach to fixing that. The broader point is that document representation is a design surface that compounds across the entire retrieval stack, and most teams are leaving it on autopilot.

Key takeaways:

• AutoIndex replaces static document preprocessing with iteratively learned executable programs (slice, enrich, normalize, reweight, reorganize) that are optimized directly against retrieval quality, with the retriever held constant.
• Across all 8 CRUMB benchmark tasks, learned programs beat static full-document BM25 by an average of +8.4% Recall@100 and +8.3% nDCG@10, with peak gains of +30.5% Recall@100 and +43.6% nDCG@10; caveat is that BM25 is the only retriever evaluated, so dense retrieval interaction remains untested.
• Teams running lexical retrieval pipelines should treat their indexing and preprocessing stage as an optimization target and test AutoIndex before adding retriever or reranker complexity.

Source: AutoIndex: Learning Representation Programs for Retrieval