Beyond Query-Only Distillation: Toward Dual-Tower Representation Alignment for Visual Document Retrieval

TL;DR

Visual document retrieval embeds page images directly—no OCR in the loop. The catch is that the models worth using are 2B–8B parameter VLMs, and building an index means running one of them over every page you own.

We froze an 8B teacher, cached its page embeddings once, and trained a 457M student to reproduce them with a single cosine loss. No relevance labels, no negative mining, no contrastive objective. Paired with the 70M text-only query tower from our previous release, the complete retriever is 527M parameters and retains 86.9% of the teacher's NDCG@5 across all 22 ViDoRe datasets—leading the strongest sub-1B baseline we reproduced by 8.73 average points.

Here is what indexing costs on a single H200:

Today we are releasing:

• NanoVDR-D-HiRes and NanoVDR-D-Fast, the two document towers, loadable through both transformers and sentence-transformers.
• The full training and evaluation code at github.com/Ryenhails/NanoVDR, including the packaging script and the verifier we use to confirm that a released checkpoint reproduces our internal numbers.
• The 1.20M-image training mixture manifest and its deduplication artifacts.
• Fourteen ablations—including the five that did not work.

from sentence_transformers import SentenceTransformer      # >= 5.4 for the doc tower

docs = SentenceTransformer("nanovdr/NanoVDR-D-HiRes-Qwen3VL8B-4096", trust_remote_code=True)
queries = SentenceTransformer("nanovdr/NanoVDR-Q-DistilBERT-Qwen3VL8B-4096-ML")

doc_emb = docs.encode(pages)                                  # PIL pages -> (N, 4096)
query_emb = queries.encode(["what was the revenue growth in Q3 2024?"])

scores = query_emb @ doc_emb.T

Table of Contents

• Why the document side
• Teacher and data
• Architecture and loss
• Fast and HiRes
• Performance
• What the ablations said
• Usage
• What we would do differently

Why the Document Side

Our previous release, NanoVDR, replaced the query tower of a visual document retriever with a 70M text-only encoder. The recipe was about as simple as distillation gets: cache a frozen VLM teacher's query embeddings, then train a small student to reproduce them under

loss = 1 − cos(student, teacher)

It worked. A 70M DistilBERT that has never seen an image can land squarely in a vision-language model's embedding space—close enough to retrieve against it—and it encodes a query in a few milliseconds on a single CPU thread.

But queries are cheap and there are few of them. Documents are expensive and there can be millions, and after that first release you still needed the full 8B model to build an index. So this time we ask: does the same objective work on the document side?

That is not a given. The input is now a page image rather than a short sentence. The student has to resolve fine-grained text, table cells, and chart labels rather than parse a query. And there is no pretrained text encoder to bootstrap from—one cosine term against a cached vector is thin supervision for learning to read.

It also turns out to be the bigger half of the gap. If we take the finished system and swap each side back to the teacher:

Teacher and Data

Teacher: Qwen3-VL-Embedding-8B, frozen, 4096-d output.

We chose it for two reasons. First, it was the strongest single-vector retriever in our reproduction, scoring 71.05 average NDCG@5. Second, it is trained with Matryoshka representation learning, so any leading prefix of its output is itself a valid embedding. That turns the student's output width into a tuneable knob rather than a hard architecture decision: aligning to the first 768 dimensions is a principled compression, not a random projection. We measure the cost of this truncation further down.

Teacher targets are precomputed once over the entire mixture and stored as HDF5. After that, the teacher never runs again. This is what lets the document and query towers train independently—and in parallel. It also happens to be the single largest cost in the project at 99.5 H200-GPU-hours, more than either student's training.

Data: 1.20M unique page images, all public and permissively licensed.

Everything is pHash-deduplicated against itself and against all three ViDoRe evaluation corpora before training. We release the mixture manifest and the dedup artifacts rather than re-hosting images that already live on the Hub under their own licences.

The two finance-specific sources at the bottom were added because retention was worst on financial pages, and we assumed more in-domain data would help. It didn't. What actually fixed those domains was resolution—the first finding in the ablations section.

Architecture and Loss

The document tower is 457M parameters and emits a single L2-normalised 4096-d vector per page. Retrieval against a query vector is a dot product.

page ─► dynamic tiling ─► InternViT-300M ─► Linear(1024→768) ─► ModernBERT-base ─► mean pool ─► Linear(768→4096) ─► L2-norm
        aspect-ratio-      per-tile patch                       bidirectional,
        matched 448px      tokens, all tiles                    zero text tokens
        tiles + thumbnail  concatenated

Three design choices carry it.

Tiling before the visual encoder. The page is cut into aspect-ratio-matched 448×448 crops at the encoder's native resolution, plus one whole-page thumbnail for global context. This follows the InternVL-V2 partition rule. A single 448px view of an A4 page renders 10-point body text at roughly two pixels per stroke—barely legible. When the model struggled with financial tables, it wasn't failing to understand finance; it was failing to resolve the glyphs.

A language encoder over patch tokens. ModernBERT re-encodes the projected visual tokens with bidirectional attention. There is no tokenizer and no text input; the text model serves purely as a deep contextual aggregator over patches. The intuition is that "this page is an annual report whose revenue table shows a decline" is a statement about relationships between distant patches, and mean-pooling a ViT throws exactly that away. We chose ModernBERT-base for its 8192-token context, which absorbs the worst case of 7 × 1024 = 7168 tokens without truncation.

One linear head to the teacher's full width. Mean-pool, one linear layer to 4096-d, L2-normalise. No MLP, no bottleneck.

And the loss, in its entirety:

loss = 1 - F.cosine_similarity(student_emb, teacher_emb).mean()

Training recipe: AdamW, one-cycle schedule with 3% warmup, peak LR 1e-3, effective batch size 256, 3 epochs, 2× H200. The visual encoder trains at 3.3% of the base learning rate—it arrives already pretrained on document imagery and is the component most easily damaged by aggressive updates.

Fast and HiRes

Fast and HiRes share the same architecture, the same data, and the same recipe. They differ in a single number.

Think of the tile budget as a deployment knob rather than a quality tier. Fast is the right choice when you are indexing millions of pages; HiRes is what you want when those pages are dense professional reports. The next section prices both.

Performance

Retrieval Quality

Setup. All 22 datasets from ViDoRe v1, v2, and v3, scored by NDCG@5 under a single evaluation protocol. Every baseline was reproduced locally rather than quoted from its own paper, so these numbers are comparable to each other but not directly to published tables. The two bold rows are fully distilled end-to-end: the document tower paired with the 70M query tower distilled from the same teacher, 527M total, with nothing multi-billion running at deployment.

HiRes leads the strongest sub-1B baseline we reproduced (colSmol-500M) by 8.73 average points; Fast leads it by 6.97.

The two variants separate most clearly on v3—the hardest benchmark, built on long professional reports with dense, small text: 47.07 vs. 43.66. On v1 and v2 they stay within 1.5 points of each other. If your corpus is slides and forms, take Fast; if it is annual reports, the extra tiles are doing real work.

Compared with larger models, HiRes surpasses DSE-Qwen2 (2.2B) and ColPali v1.3 (2.9B) while being four to six times smaller. It trails Qwen3-VL-Embedding-2B, and sits 6.95–9.80 points behind the 4–8B multi-vector models. Overall, HiRes retains 86.9% and Fast 84.4% of the teacher's quality.

Where the Student Holds Up, and Where It Does Not

That 86.9% is one number over a benchmark that spans eight professional domains and six languages, and it hides a great deal. Per-dataset retention for HiRes runs from 57.9% to 97.8%, a spread of forty points:

_{End-to-end student x student. Retention is this model's NDCG@5 over the teacher's on the same dataset. The 86.9% headline is the ratio of the two averaged scores; the unweighted mean of the column above is 82.1%, because the datasets the student handles worst are also ones where the teacher scores low.}

The pattern is consistent. Synthetic and government documents are close to lossless, TabFQuAD and the SyntheticDocQA family all above 92%. Dense professional reports are where the student gives ground: industrial at 57.9% and ESG at 60.3% are the two worst, and both are long PDFs with small text, dense tables and figures that carry the argument. Finance splits by language, 72.8% in English against 70.6% in French, which is the multilingual gap showing up on the document side rather than the query side.

If you are deciding whether to deploy this, the average is the wrong number to plan against. Find the row closest to your corpus.

Efficiency

Setup. Single H200, batch size 8, bf16, each encoder under its best-supported flash-attention backend. Query latency is measured at batch size 1 including tokenisation. Index size is per million documents in float32. "Score 10K" is the time to score one query against 10,000 candidates on a single CPU thread—where single-vector and multi-vector retrieval fundamentally part company.

Fast indexes at 99 pages per second in 2.10 GB—roughly 18× the teacher's throughput and an order of magnitude beyond every multi-vector baseline at any scale. HiRes trades some of that speed for its larger tile budget, yet still runs at 7× the teacher.

It is worth noting that the four 250–500M baselines use the uncapped Idefics3 image splitter, which emits more than ten sub-images per page, whereas we cap at seven. Part of the throughput gap is therefore budget, not architecture.

The scoring column reveals the structural difference. One 4096-d vector per page occupies 16.4 GB per million pages and takes 9.6 ms to score 10,000 candidates. The multi-vector retrievers in the same parameter class need 256 GB and over a second for the same query—a 15.6× index inflation and two orders of magnitude more scoring work. Multi-vector retrieval buys quality, but it pays with storage and latency that a single-vector model simply doesn't spend.

What the Ablations Said

Each block varies one factor while holding the rest fixed. Blocks (a) and (b) are end-to-end evaluations; (c) is document-side in isolation, scored against teacher-encoded queries, so its absolute values are not comparable across blocks.

What Worked

(a) Tile budget. Re-trained at each budget, not just re-evaluated.

Resolution was the largest single lever we found, and by a wide margin. Tiling at all is worth 3.36 average points over a single view, and going from two tiles to six adds another 1.76—or 3.41 on v3 alone. There is a cheaper version of this finding: take a single-view checkpoint, retrain nothing, and simply swap the preprocessor to tile at inference. That alone gains 3–5 points on every benchmark, with tabfquad jumping by +30.6.

(b) Training data scale. Uniform subsamples of the 1.20M mixture.

The gains are monotonic but saturate above 75%. The full mixture is worth roughly 5 points over a quarter of it, yet only 0.06 over three quarters. If you are reproducing this on a budget, 900K images gets you essentially all of it.

(c) Output dimension. Document side in isolation.

A 5.3× smaller index costs 3.19 average points (4.77 on v3). We ship 4096-d by default and treat 768-d as the option for hard storage constraints.

This result reversed an earlier conclusion of ours. Under a 2B teacher at 2048-d with an MLP projector trained on 711K images, we measured the student's embedding geometry and found it hollow: effective rank 636 out of 2048, with all the variance packed into the first 768 dimensions. Truncating cost nothing, and we concluded that aligning at the backbone's native width was the way to go. That was true for that teacher. A stronger teacher, five times more data, and a plain linear head reverse the finding.

(d) Visual encoder. Same recipe, 711K images, 2B teacher at 2048-d.

Two takeaways. First, OCR-oriented pretraining is non-negotiable: InternViT-300M beats SigLIP2-base by roughly 30 points of retention with the same head attached, and SAM never got off the ground because it is a segmentation encoder that has never been asked to read. Second, a ViT alone is not enough: the same encoder with nothing on top reaches 61.7% retention versus roughly 80% with ModernBERT over it—a component that never sees a single text token.

What Did Not Work

Contrastive supervision on top. Starting from a cosine-distilled checkpoint, we ran one further epoch of joint refinement under L = L_d + L_q + γ·L_rank, with γ ∈ {0.5, 1.0, 2.0}, using both InfoNCE and KL divergence over the teacher's in-batch score distribution. KL stayed flat within ±0.14; InfoNCE drifted downward from −0.12 to −0.50 as γ grew. The refinement budget was about a third of the document encoder's training budget—enough that a real contribution should have been visible. Pointwise alignment appears to preserve the teacher's ranking implicitly, leaving little for an explicit ranking objective to teach. A good deal of the distillation literature assumes the opposite.

Reusing the teacher's own vision encoder. QwenViT is the visual half of the model we are distilling, so it should already produce compatible features. Instead, it stalled above 0.6 loss at every configuration we tried—worse than a 95M SigLIP2. In Qwen3-VL, those features are consumed by an LLM decoder that does all the semantic heavy-lifting; detached from it, they were never asked to mean anything on their own. A teacher's sub-component is not a pretrained student.

More in-domain data. Adding 32K finance images at their natural 2.7% proportion moved v1 by +0.45, v2 by −0.29, and v3 by +0.34. Noise, and finance_fr actually regressed. We had the wrong hypothesis; resolution fixed those same domains.

EOS pooling. The teacher pools at its last token, so matching that strategy seemed principled. A learnable [EOS] on the document encoder plus last-token pooling on the query side consistently lost 0.5–0.8 points. A DistilBERT [SEP] is not an instruction-tuned Qwen EOS.

Post-hoc cross-alignment. Freezing the query tower and fine-tuning the document tower to align with it directly—rather than with the teacher—across six loss variants, all saturated within one epoch at under 0.2 points of change. When both encoders already live in the teacher's space, pulling them toward each other has nothing left to fix.

Domain-balanced sampling. Weighting a 5K domain equally against a 700K base collapsed v3 to 8.7. Concatenate-and-upsample works; aggressive rebalancing does not.

Usage

Through sentence-transformers (>= 5.4 required for the document tower):

from sentence_transformers import SentenceTransformer

docs = SentenceTransformer("nanovdr/NanoVDR-D-HiRes-Qwen3VL8B-4096", trust_remote_code=True)
queries = SentenceTransformer("nanovdr/NanoVDR-Q-DistilBERT-Qwen3VL8B-4096-ML")

scores = queries.encode(["what was the revenue growth in Q3 2024?"]) @ docs.encode(pages).T

Through transformers, which has no version floor and returns identical vectors:

from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained("nanovdr/NanoVDR-D-HiRes-Qwen3VL8B-4096", trust_remote_code=True).eval()
proc = AutoImageProcessor.from_pretrained("nanovdr/NanoVDR-D-HiRes-Qwen3VL8B-4096", trust_remote_code=True)

doc_emb = model.encode(pages, proc, batch_size=4)

The repo ships a NanoVDRDocImageProcessor that handles tiling and returns pixel_values and tile_mask together, so processor(images=pages) gives the model exactly what it expects. The two paths above agree to the bit.

Mixing and matching. Both towers target the same teacher space at the same width, so either one is a drop-in replacement for the corresponding half of the teacher. All four combinations are valid retrieval systems, letting you trade quality against what you no longer have to run:

Reproducibility. Re-encoding the 500-page ViDoRe arxivqa corpus with the released package yields cosine 0.7959 against the teacher and NDCG@5 of 83.07 vs. the teacher's 86.91—a 95.6% retention that matches our internal evaluation. packaging/verify_doc_package.py in the repo is the verification script, and it is the same one we run before every upload.

What We Would Do Differently

Two findings carry forward to the next project. Input resolution mattered more than any architectural choice we tried. And adding a ranking objective on top of pointwise alignment bought nothing at any weight we tested.

ESG and industrial pages still sit below 70% retention, and no ready-made Hub dataset covers SASB-style reports or USAF technical orders. The resolution lever is nearly exhausted; the data lever for those domains is not, but it will require scraping rather than downloading.

We would also run the visual-encoder comparison first. It turned out to be the experiment that mattered most, and we ran it only after building two full systems on a backbone we had not yet justified.

A multi-vector query tower is in progress and will ship as NanoVDR-v2.

Links

Paper: arXiv:2608.10636 · Models: nanovdr · Code: github.com/Ryenhails/NanoVDR · Demo: NanoVDR-Demo

Citation

If you find this work useful, please cite:

@article{liu2025distilvdr,
  title   = {DistilVDR: A Compact End-to-End Visual Document Retriever via Dual-Student Distillation},
  author  = {Zhuchenyang Liu and Ziyi Wang and Yao Zhang and Yu Xiao},
  year    = {2026},
  journal = {arXiv preprint arXiv:2608.10636}
}

The paper is on arXiv: arXiv:2608.10636.