DistilVDR: A 524M Visual Document Retriever That Indexes 15.6x Smaller

Prior compression work on visual document retrieval solved the wrong half of the problem. Distilling only the query encoder makes inference cheaper at serving time, but the document encoder stays large, corpus indexing stays slow, and the index stays fat. For any team running retrieval at scale, that's the bottleneck that actually blocks deployment.

DistilVDR takes a different route: bilateral distillation, compressing both the query and document encoders simultaneously from a single frozen 8B vision-language teacher. The student is asymmetric by design. The document side gets the bulk of the visual capacity, because documents are image-heavy and visually complex. The query side is held to 70M parameters, because text queries are cheap to encode. That asymmetry mirrors the actual input structure of VDR systems, where one query matches against millions of indexed documents. All training signal comes from the teacher's embedding space under a pointwise cosine alignment loss, which means no relevance labels, no negative sampling, and no contrastive term. The frozen teacher was itself trained with relevance supervision, so that signal is already baked into the embedding geometry the student is learning to replicate.

Two variants ship under the same encoder pair, differing only in the visual-tile budget for the document encoder. DistilVDR-HiRes scores 61.74 average NDCG@5 across ViDoRe v1, v2, and v3, reaching 86.9% of the 8B teacher's performance at roughly 6% of its parameter count. It leads every reproduced sub-1B baseline on the resolution-sensitive v3 benchmark. DistilVDR-Fast, running a 3x smaller visual-token budget, scores 59.98. Both variants store one million documents in an index 15.6x smaller than the strongest sub-1B multi-vector baseline and complete corpus indexing an order of magnitude faster. For ML infrastructure teams running document retrieval at corpus scale, the takeaway is direct: a single-vector 524M retriever now exists that closes most of the quality gap against billion-plus-parameter systems while eliminating the indexing cost that made those systems impractical to deploy.

We're thinking: The specific move that prior VDR compression missed was bilateral distillation, and DistilVDR makes that gap concrete. Query-only distillation was always a partial fix: it reduced online serving cost but left the offline indexing pipeline, the one that runs continuously as corpora update, just as expensive as before. What we find worth watching here is the label-free training objective. Distilling from a frozen teacher's embedding space, with no relevance labels or negative mining, means teams can adapt this to new document domains without annotation pipelines. That may be more practically valuable than the parameter count itself, especially for enterprise retrieval where labeled query-document pairs are scarce and corpora are constantly changing.

Key takeaways:

• Bilateral distillation compresses both query and document encoders into a 524M asymmetric student, concentrating visual capacity on the document side while holding the query encoder to 70M parameters, with all supervision derived from a frozen 8B teacher's embedding geometry.
• DistilVDR-HiRes reaches 61.74 NDCG@5 on ViDoRe v1+v2+v3 (86.9% of the 8B teacher), produces an index 15.6x smaller than the strongest sub-1B multi-vector baseline, and indexes a million documents an order of magnitude faster; the main caveat is that the 13.1% quality gap to the teacher remains real on the most resolution-demanding tasks.
• Teams building or scaling visual document retrieval pipelines should evaluate DistilVDR before committing to multi-vector or full-scale 8B deployments, particularly if corpus size or indexing frequency is the binding constraint.

Source: DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation