How we trained a reranker small enough for a laptop Eight times smaller than the hosted model, trained for $9, and it answered as many questions right in our agent: 132 of 150 against 133. No GPU needed.
- The problem: our search ends with a reranker, a model that reads each candidate passage beside the question and orders them. Ours, Nemotron, is hosted on GPUs. Many clients want everything on their own machines, and few want a GPU server to look after.
- What we did: we generated 49,713 questions from documents the test never uses, had Nemotron score each one's candidates, and trained a small multilingual model (118 million parameters) to reproduce its ordering.
- Cost: about $7.90 of model calls to make the training data, under $1 of rented GPU time to train.
- In the agent it tied: 132 of 150 FinanceBench questions right against 133 and 132 for two runs with Nemotron, 11 wrong against 10 and 12; and 28 of 32 on a mixed French and English library against 28.
- Cost at par, time not: a median of 1 step and 0.7 to 0.8 cents per question with either reranker, but a median of 58 s per answer against 8 to 9 s, about six times slower, on 4 older shared cores. Faster machines would cut that.
- On its own it ranks lower: the answer page reached its top 10 for 79 FinanceBench questions, against 95 for Nemotron. The agent makes up the difference by searching again.
Same agent, same search, same tools and limits; only the reranker changed. Answers graded by Claude against the reference answers, as in our published results.
The last step of search was the one that needed a GPU.
VectoGrid finds passages in two steps. A first search, by meaning and by keywords, gathers up to 80 candidates. A reranker then reads each candidate together with the question and puts the best first. It is the most accurate step and the heaviest: Nemotron Rerank, the model we use, has a billion parameters and runs on hosted GPUs.
That is fine for a cloud service. It is friction for a client who wants everything on their own server: a new vendor to approve, or a GPU machine to buy and keep running. A model small enough for a CPU removes that, if it is good enough.
Teach a small model to copy the large one.
Out of the box, a small multilingual reranker (mMiniLM, 118 million parameters) was far behind: with 80 candidates, it put the answer page in its top 10 for 65 FinanceBench questions, where Nemotron did for 95. So we trained it to imitate Nemotron, a method called distillation.
- Questions from documents the test never sees. FinanceBench's 150 questions point at 84 of its 368 reports. We took passages only from the other 284, and had a hosted model (Mistral Small) write questions about them: 73,329 questions from 25,000 passages.
- Candidates found the way the product finds them. For each question, the first search's top 40 by meaning and top 40 by keywords: about 69 candidates each.
- The teacher's scores as labels. Nemotron scored every candidate, 3.4 million pairs in all. No human labels, and nothing from the benchmark.
- Train on the order, not the numbers. The small model learned to reproduce Nemotron's ranking within each question's candidates, which is all the product uses. Three passes over 47,151 questions on one rented RTX 3090, about an hour; a held-out set of training documents chose the best pass.
| Step | Cost |
|---|---|
| Writing the questions (Mistral Small 3.2, hosted) | $1.36 |
| Scoring the candidates (Nemotron, hosted) | $6.53 |
| Training (GPUs rented by the hour; a pilot on 20,000 questions first) | under $1 |
On its own, it still ranks below its teacher.
We measured ranking first: the 150 FinanceBench questions, the candidates our first search finds, and how often the page people marked as the answer reaches the top.
| 80 candidates | Top 3 | Top 5 | Top 10 | Top 20 |
|---|---|---|---|---|
| Nemotron (hosted, 1 billion parameters) | 64 | 81 | 95 | 107 |
| Small model, trained on 49,713 questions | 49 | 65 | 79 | 100 |
| Small model, trained on 19,881 questions | 45 | 60 | 74 | 94 |
| Small model, untrained | 33 | 39 | 65 | — |
Training gained 14 to 26 questions at each cut-off, and more data kept helping. The gap to Nemotron remains: the small model needs its top 20 to find what Nemotron finds in its top 10.
Where it counts, the gap closed.
Our agent does not stop at one search. It reads the 5 passages it gets, searches again in other words when they fall short, and can open whole pages. A reranker that puts the answer 8th instead of 2nd costs it one more search, not the answer. So we ran the same agent with each reranker, nothing else changed.
| FinanceBench, 150 questions | Right | Partial | Wrong | Median steps | Median cost per question | Median time per question |
|---|---|---|---|---|---|---|
| Nemotron, run 1 | 133 | 7 | 10 | 1 | 0.78¢ | 8 s |
| Nemotron, run 2 | 132 | 6 | 12 | 1 | 0.76¢ | 9 s |
| Small model on a CPU | 132 | 7 | 11 | 1 | 0.74¢ | 58 s |
Quality and cost are at par: the hosted reranker's fee disappears, and the agent's own calls cost about the same. Time is not: the small model ran on 4 older shared cores, 4 questions at a time, and each answer took about six times as long.
Question by question, it differed from each Nemotron run on 5 to 8 questions, split almost evenly: the difference is what two runs of the same setup also show.
| Library, 32 questions over 383 French and English documents | Right of 32 |
|---|---|
| Nemotron | 28 |
| Small model, trained | 28 |
| Small model, untrained | 26.5 |
On the library, all three were judged in one pass with their labels hidden. The trained model scored the same as Nemotron on every one of the 32 questions.
The agent forgives a weaker reranker. What it cannot forgive is a passage that never reaches the candidates.
Seconds per search, no GPU.
| Machine | 80 candidates | 20 candidates |
|---|---|---|
| 8 dedicated server cores | 3.0 s | about 0.8 s |
| 4 older shared server cores | 5.3 s | 1.4 s |
In the agent run above, 4 questions at a time shared the 4 older cores, so searches queued: a median of 58 s per answer, against 9 s with hosted Nemotron. Eight dedicated cores rerank in 3.0 s instead of 5.3 s and would cut that, though we did not run the agent on them. A larger model of the same family, bge-reranker-v2-m3 with 568 million parameters, ranked better untrained but took 11.8 s for just 20 candidates on 8 cores: too slow for a CPU.
What this says, and what it does not.
- Two benchmarks, one run each with the small model. Ties within the run-to-run variation of the same setup, not proof of equality everywhere.
- It holds in an agent that searches again. A pipeline that searches once and answers from its top results would feel the ranking gap; there, sending 20 passages instead of 10 makes up for it, at more reading per question.
- The teacher sets the ceiling. The small model learned Nemotron's judgements, including its mistakes.
- The library is partly ours. Its questions were written by us; the FinanceBench test documents were never used in training.
- Speed depends on the machine and the load. The per-search times were measured one at a time; the agent run had 4 questions sharing 4 older cores. More users on the same CPU means longer waits.
How the test was run
- Student: cross-encoder/mmarco-mMiniLMv2-L12-H384-v1, trained with a listwise loss toward Nemotron Rerank VL 1B's scores over groups of 16 candidates; three epochs; the best on a held-out 5% of training documents.
- Agent: the Agentic RAG setup of our grep against RAG study, qwen3.6-27b, 5 passages per search, up to 25 steps; the reranker served from a CPU machine.
- Grading: Claude against FinanceBench's references, with the same six references corrected from the documents as in our published results.
For what the CPU reranker joins, see A smaller vector that finds more.
Want document search that runs entirely on your own machines?
Published 8 November 2026 · Philippe Vergnet, VectoGrid