Multimodal Reranking Is a System One Decision
TL;DR Highlight
One multiple-choice question to a VLM over 16 candidates reranks them, beating a dedicated 2B reranker.
Who Should Read
Backend developer improving image or PDF-page retrieval in a RAG system who finds running a separate reranker model costly.
Core Mechanics
- Per-candidate yes/no backfires
- Asking yes/no for each candidate zero-shot scored below the first-stage retriever.
- Putting 16 candidates under labels A–P and asking which fits best works better.
- Choice needs one forward pass
- Read the next-token probabilities of the labels A–P and renormalize them to get the ranking, with no text generation.
- It needs no training and one model call per query.
- General 27B beat a dedicated 2B reranker
- Zero-shot Qwen3.8-27B beat the supervised Qwen3-VL-Reranker-2B on all three benchmarks.
- It came within 1.6 points of the 8B dedicated reranker on each.
- Small models don't work
- Gemma 4 E2B and E4B fell below the first stage on all three benchmarks.
- Models of 12B and up improved page retrieval.
- Text-only tuning helps pages a bit
- JEV-27B-VL, tuned on text decisions only, improved page reranking by 0.6 to 1.1 points.
- Natural images were unchanged.
- Image-only queries get worse
- All nine image tasks with text in the query improved.
- The three tasks with image-only queries dropped.
Evidence
- Choice vs yes/no on MMEB-V2 image retrieval
- Choice scored 5.4 to 7.3 hit@1 above per-candidate yes/no.
- Qwen3.8-27B: choice 76.76, yes/no 69.50, first stage 74.85.
- Qwen3.8-27B vs dedicated rerankers
- It beat Reranker-2B by 2.3 hit@1, 2.1 nDCG@5 and 1.8 nDCG@10.
- It tied Reranker-8B on MMEB-V2 visdoc, 84.79 vs 84.78 nDCG@5.
- JEV-27B-VL page reranking gain
- Over Qwen3.8-27B: +0.63 nDCG@5 on MMEB-V2 visdoc and +1.09 nDCG@10 on ViDoRe V3.
- Sign tests gave p = 0.017 and 0.008.
- Latency per ViDoRe V3 query on one H200
- Qwen3.8-27B choice took 3.97 s, 1.1x Reranker-8B's 3.67 s.
- That is 1.4x Reranker-2B (2.75 s), and 1.5x Reranker-8B with batched scoring.
How to Apply
- Switch your rerank prompt to multiple choice
- If you score candidates one by one with yes/no today, put the top 16 under labels A–P and pick in one call.
- Compare hit@1 or nDCG against the first stage on the same eval set.
- Reuse your large VLM as the reranker
- If you already serve a large VLM for RAG answers, rerank top candidates with the same model.
- Check whether you can drop a separate reranker, and measure latency.
- Check small models against the first stage
- Models under 12B may be worse than the first stage, so always compare against it.
- Check candidate order and query type
- Candidates go in first-stage order only, so shuffle them and see whether the pick changes.
- For image-only queries, compare keeping the first-stage order instead of reranking.
Try It
# Paste into Claude or ChatGPT with image/page candidates attached (simplified version of the paper's method: returns the answer letter rather than logits)
Instruction: {{search instruction}}
Query: {{question}}
Candidates:
Candidate A: {{candidate 1}}
Candidate B: {{candidate 2}}
Candidate C: {{candidate 3}}
...
Candidate P: {{candidate 16}}
Which candidate best matches the query, following the instruction? Answer with the letter of the best candidate only (A-P).
# Next, try this: shuffle the candidate order and ask again to see whether it picks the same letter.Terminology
Related Papers
You could have invented transformers from memory
Viewing linear, MLP and attention layers as memory stores: capacity goes ~64 to ~2,700 to over a million
Pointing AI at archives found a forgotten meteorite, lost rhinos, and more
Embedding search plus a cheap filter model surfaced unrecorded events in 400 years of archives
EmbeddingGemma 2: An open, lightweight multimodal embedding model
One open 740M model embeds text, images, audio and video into a single shared vector space
RAG-NAROK: Retrieval-Aware Knowledge Corpus Poisoning in RAG with Source-specific Refutation
Refutation documents targeting retrieved sources took top rank and evaded anomaly detectors up to 88.37%.
Text, Pixels, or Both? Evaluating Input Representations for Multimodal Document QA
Page images beat OCR text on accuracy but cost more; a question-only router cut latency 30%.
Microsoft exec called AI scraping 'the largest theft of labor in human history'
NYT lawsuit unredacted filings reveal Microsoft and OpenAI internal views on scraping
Harnessing the Universal Geometry of Embeddings
Translating embeddings between models without paired data exposes vector database security risks
Better Together: Complementary Query Rewriting Under a Strong RAG Baseline
Combining query rewrites, run only on low-confidence queries, gets half the gain at about 40% of the cost
Related Resources
Original Abstract (Expand)
Multimodal rerankers are usually dedicated models: a vision-language model (VLM) is trained on retrieval data to score the candidates of a query. Scoring options is not specific to reranking. System One decision models such as Jev map a state and a typed question to one probability per option without generating text, and general models can make such decisions with little or no training. Reranking is one such decision, with the query and its candidates as the state and the candidates as the options. We therefore argue that a general VLM can serve as a multimodal reranker through System One decisions, and that decision tuning can improve its reranking. We ask a VLM one choice question over the top 16 candidates of a first-stage retriever and rank the candidates by the probabilities of their labels, read from one forward pass. On MMEB-V2 image retrieval, this choice scores 5.4 to 7.3 hit@1 above a yes/no question per candidate, and the yes/no question falls below the first stage. With the choice, zero-shot Qwen3.8-27B beats the supervised Qwen3-VL-Reranker-2B on all three benchmarks and comes within 1.6 points of Reranker-8B on each. It matches Reranker-8B on MMEB-V2 visual documents (84.79 vs. 84.78 nDCG@5), at 1.1 to 1.5 times its latency. JEV-27B-VL, the same backbone tuned on text decisions only, improves page reranking by 0.6 to 1.1 points, with gains on most tasks (sign test, p <= 0.017), and leaves natural images unchanged. The zero-shot model is larger than the supervised rerankers and of a newer generation, so these comparisons are confounded; we discuss this and other alternative views, and the experiments that would test the position.