The Retrieval Metric Scoring Your RAG Pipeline Into Failure

Standard retrieval evaluation scores each document independently, then aggregates. That design choice made sense when humans read search results one at a time. It makes no sense when an LLM consumes the entire retrieved set as a single context window.

The problem is not that current rerankers are weak in isolation. It is that the evaluation framework cannot see what actually determines downstream generation quality: whether the documents in a set contradict each other, repeat the same information, or cover complementary angles that together answer a query. A retrieval system can score well on nDCG while returning a set where three of five documents conflict on a key fact and two are near-duplicates. The metric never notices. The language model does.

SetwiseEvalKit makes this structural gap measurable. The benchmark organizes document set quality across three levels and nine dimensions, covering both short-form and long-form retrieval scenarios, with roughly 28,000 high-quality evaluation rubrics. Those dimensions include cross-document coordination: redundancy, conflict, and complementarity, the properties that nDCG is blind to by design. Running 12 rerankers through this framework produces a clear finding: the best-performing method reaches only 45% coverage. Cross-document coordination dimensions are the weakest area across all 12 systems, universally. No single method holds top position across both short-form and long-form settings.

The coverage ceiling is not a minor gap to close with better training data. It signals that rerankers optimized against relevance signals are solving a different problem than set-level quality for generation.

Rubric4Setwise closes the loop without requiring retraining. It converts the rubric-based evaluation criteria directly into selection signals, treating the diagnostic output of SetwiseEvalKit as an optimization input. The mechanism is training-free: rubrics that define what a good document set looks like get repurposed to score candidate sets during retrieval, not just evaluate them after the fact. That reuse is what makes it practical. Teams do not need a new model. They need a different selection objective applied to the same candidates their existing pipeline already retrieves.

The downstream result is better generation performance with fewer documents and fewer search rounds. Rubric4Setwise is the only method in the benchmark that holds state-of-the-art results across both short-form and long-form scenarios simultaneously. For teams building RAG pipelines where generation quality is the actual product metric, the takeaway is direct: the reranker you are evaluating with nDCG may be actively degrading the set-level coherence your LLM depends on.

We're thinking: We find the 45% coverage ceiling more damning than it first appears. It means that even after years of reranker development and fine-tuning on relevance labels, the entire field has been optimizing for a proxy that does not track the thing that matters for LLM consumers. The deeper issue is that nDCG encodes a document-independence assumption that was always a simplification, but became actively harmful once LLMs replaced humans as the primary retrieval consumer. Rubric4Setwise is a practical fix, but the more consequential shift is in evaluation design: teams that adopt SetwiseEvalKit-style rubric benchmarks will start seeing failure modes in their pipelines that their current metrics cannot surface.

Key takeaways:

• Rubric-based set evaluation exposes three dimensions nDCG cannot score: redundancy, conflict, and complementarity between documents in the retrieved set, and converting those rubrics into selection signals produces a training-free reranking method.
• Across 12 rerankers on 28,000 rubrics, the best system reaches 45% coverage; cross-document coordination is the weakest dimension for every method tested, and no system leads on both short-form and long-form tasks.
• Teams running RAG pipelines should audit their retrieval eval against set-level dimensions, not just per-document relevance scores; a reranker that wins on nDCG can still return a set that degrades generation quality through internal conflict or redundancy.

Source: Beyond Relevance-Centric Retrieval: Rubric-Oriented Document Set Selection and Ranking