NVIDIA Nemotron 3.5 Embed, Rerank and Parse for Agent Retrieval
Agentic AINVIDIA says adding its new Nemotron 3.5 Rerank model to an agent’s retrieval pipeline cuts agent token cost per query by about 14% while improving retrieval accuracy. That is the pitch behind three new open, commercially available model families released today: Nemotron 3.5 Embed, Rerank and Parse. For teams building agents over PDFs, charts and tables, the release matters less as a leaderboard win and more as a claim that better retrieval is now a cost lever, not only a quality one.
What Happened
In NVIDIA’s announcement on the Hugging Face blog, the company expands its Nemotron Retriever collection with five models:
| Model | Role |
|---|---|
| Nemotron 3.5 Embed 8B BF16 | Text and image retrieval, quality-focused |
| Nemotron 3.5 Embed 1B BF16 | Distilled from the 8B for a smaller footprint |
| Nemotron 3.5 Embed 1B NVFP4 | Quantized 1B for high-throughput indexing |
| Nemotron 3.5 Rerank 8B BF16 | Cross-encoder that re-orders candidates |
| Nemotron 3.5 Parse | 631M-parameter document-to-structure model |
The embedding models search document images, including tables, charts, infographics, datasheets and schematics, alongside text. NVIDIA reports text retrieval evaluated across 34 languages.
Why It Matters
NVIDIA’s argument is that weak retrieval makes agents search again and read more documents, which burns inference tokens. In its tests across ViDoRe v3, BRIGHT and BrowseComp-Plus, adding the reranker cut agent token cost per query by approximately 14% with the 8B embedder and 13% with the 1B embedder. These are NVIDIA’s own evaluations.
On the leaderboards NVIDIA cites, as of October 20, 2026:
- Embed 8B scores 61.17 nDCG@10 on ViDoRe v3.1, first among dense retrieval models and 8.94 points above NVIDIA’s earlier Llama Nemotron Embed VL 1B. The distilled 1B keeps 95.4% of that score.
- Rerank 8B scores 60.49 on the new ViDoRe v3.1 reranking leaderboard with combined image and text inputs, against 59.17 for Qwen3-VL-Reranker-8B.
- Parse ranks first on ParseBench among models under 1 billion parameters, scoring 75.20 at 10.5 pages per second on one H100 in its most accurate triple-pass mode, or 16.1 pages per second in single-pass mode at 72.74.
One detail matters for teams with existing indexes. NVIDIA kept the language component frozen during training, so text-only embeddings are expected to stay unchanged from the matching Nemotron 3 Embed models. Teams can add image retrieval without re-embedding their text corpus.
What This Means for Agent Builders
The deployment options span a wide range. The NVFP4 1B model delivers 27.1% higher throughput than the BF16 version on a GB200, at 99.85% of its accuracy. Served through NVIDIA NIM on an RTX PRO 6000 Blackwell Server Edition, the embedder handled 6,031 query requests per second, about 2.9 times a comparable vLLM setup. Fine-tuning recipes are on GitHub; in NVIDIA’s chart and infographic test, one epoch lifted the 1B model’s nDCG@10 by 18% relative and the 8B model’s by 3%.
Stack Archive has argued that retrieval quality is the defining challenge in agent architecture, and covered Gemini Embedding 2’s multimodal approach as one alternative. Nemotron 3.5 is the open-weight option on the same path.
The So What
Teams running agents over visually rich documents should benchmark Nemotron 3.5 Rerank on their own queries first, since it is the piece tied to the token savings claim, and measure tokens per completed task rather than nDCG alone. Teams already on Nemotron 3 Embed can trial the 3.5 embedder without rebuilding text indexes, and anyone parsing scanned PDFs should test Parse in single-pass mode before paying for the slower passes.
Source: Hugging Face Blog
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →