What is the biggest RAG ever built? Search engines with hundreds of billions of vectors, a model that retrieved from 2 trillion tokens, and companies that split billions of vectors into millions of small indexes. Bigger retrieval cuts hallucinations. No size removes them.
- The biggest retrieval systems are search engines. Microsoft says its SPANN index serves vector search over hundreds of billions of vectors in Bing.
- The biggest retrieval model on record is DeepMind's RETRO (2021). Retrieval was built into the model itself, over a database of 2 trillion tokens, and a model 25 times smaller matched GPT-3.
- Nobody builds models that way now. Plain RAG won: put the passages in the prompt of any model, and change the documents, the search or the model on their own.
- The largest business systems split rather than grow. Notion keeps over 10 billion vectors in over a million separate indexes. Uber Eats filters billions of candidates by place before searching, and trimmed its vectors to 256 numbers for half the storage.
- Retrieval reduces hallucinations, never to zero. Part of RETRO's gain was copying. Commercial legal research tools built on RAG still gave false answers to 17% to 33% of questions in a Stanford study.
From published papers, engineering blogs and company statements, listed at the end. Vendor and company figures are their own claims, not audited.
Web search got there first.
RAG means retrieving passages and giving them to a model to answer from. The retrieval half existed long before chatbots: it is what search engines do. So the largest retrieval systems are the ones behind web search.
Microsoft published two of the methods it uses. SPANN keeps a small part of the index in memory and the large lists of vectors on disk, and its authors state it has been deployed in Bing "to support hundreds of billions scale vector search". DiskANN, a graph method from the same research group, is described by its authors as the first able to index a billion points on one ordinary machine with under 64 GB of memory.
AI search products such as Perplexity are RAG on top of this kind of index: a web search, then a model writing an answer from the results. The retrieval is the hard part, and it was built for search engines first.
RETRO built retrieval into the model.
In December 2021 DeepMind published RETRO (Retrieval-Enhanced Transformer). Most RAG today runs a search and pastes the results into the prompt. RETRO went further: retrieval was part of the model's architecture, and the model learned to use it during training.
- The database: about 2 trillion tokens of text, cut into short chunks. Each chunk was stored with a key made by BERT, an earlier Google model that turns text into numbers, used frozen.
- While writing: for each chunk of text so far, the model looked up its nearest neighbours in the database, and dedicated layers inside the model read them as it wrote the next chunk.
- The result: RETRO matched GPT-3 and Jurassic-1 on the Pile benchmark with 25 times fewer parameters. The model could look facts up instead of memorising them.
NVIDIA took the idea further. Its researchers continued training a 43-billion-parameter model on 100 billion more tokens, retrieving from 1.2 trillion, and released InstructRetro in 2023. It beat its plain counterpart by 7% on average across eight short-answer question sets and 10% across four long-form ones.
Plain RAG won.
RETRO-style models stayed in research. We know of no major model in production built this way. The papers do not say why, so the reasons below are our reading.
- A RETRO model must be trained with its retrieval. Plain RAG works with any model, including one you cannot retrain, such as the latest from a provider.
- Everything can change on its own. With plain RAG you swap the documents, the search engine or the model independently. A company's documents change daily, and its preferred model every few months.
- Prompts got long. Models now read tens or hundreds of thousands of tokens, so pasting twenty passages costs little.
- NVIDIA's own result points the same way. In InstructRetro, the authors found they could remove the retrieval encoder after instruction tuning and use the rest of the model directly, with comparable results. Retrieval during training helped; the special machinery at answer time mattered less.
The idea that survived is RETRO's main lesson: a smaller model with good search can match a much larger one without it. Every RAG system rests on it.
The largest systems split instead of growing.
Companies rarely publish the size of their internal search. Those that do describe something other than one giant index.
- Notion keeps over 10 billion vectors on turbopuffer, in over a million separate indexes: one per workspace. Cursor stores billions of vectors there, one index per codebase. Turbopuffer's founder speaks of "trillions of objects" across customers. These figures come from case-study summaries and interviews, not audits.
- Uber Eats searches billions of candidate items, but filters them first by area, city and delivery type, so the vector search runs on a small slice. Its embedding model is trained so that the first numbers of each vector carry most of the meaning (Matryoshka embeddings). Keeping 256 numbers instead of the full vector lost under 0.3% of recall in English and Spanish and cut storage by nearly half. Uber also compresses the vectors further, and retrains the model and rebuilds the index every two weeks.
- Vector database vendors claim more. Milvus is described as running beyond 10 billion vectors, and Pinecone as scaling to over 100 billion per cluster. We found no published recall or cost behind those figures.
At billions of documents, the winning move is to narrow the search before it starts, not to search everything better.
That fits how people ask. A question about one customer, one region or one workspace should only ever touch that slice, and access rules usually force the same split anyway.
Bigger retrieval is not the same as truth.
If the documents hold the knowledge and the model only reasons, a model should stop inventing facts. Retrieval does help. It does not finish the job.
- RETRO was more factual, with limits. NVIDIA's comparison found RETRO made fewer hallucinations than an equivalent GPT model on one factuality test, with a smaller margin on TruthfulQA, whose questions are built to mislead.
- Part of the gain was copying. A 2023 reproduction found that RETRO's improvement came largely from text that also appeared in the database: when the model's recent words matched a retrieved chunk, it went into "copy mode". Retrieval worked best when the answer was already written down nearly word for word.
- Commercial RAG still invents. Stanford and Yale researchers tested legal research tools sold as reliable because they retrieve real sources. Lexis+ AI and Ask Practical Law AI hallucinated on about 17% of questions, and Westlaw AI-Assisted Research on about 33%. LexisNexis disputed the findings.
- Our own test agrees. On 150 FinanceBench questions, letting the model search again halved wrong answers, from 25 to 11, but 11 remained (Is RAG really dead?).
The reasons are the same at every size. When search misses, the model guesses from memory. When it finds the passage, it can still read the wrong row, year or company. And it sometimes prefers what it learned in training to what the document says.
Retrieval makes an answer more often right. A source on every value makes the wrong ones easy to catch.
What carries over.
- Scope first. Uber filters by place, Notion by workspace. A large company's documents split naturally by entity, owner, type and access.
- Compress the vectors. Shorter vectors (Uber) and 1-bit vectors with rescoring cut memory by a large factor, at a small and measurable cost in recall.
- Prove every answer. At any size, some answers will be wrong. A source a person can open is what turns a wrong answer into a caught one.
For how we would build document search over a billion documents, see Document AI over a billion documents, and for growing from one machine to many, From one report to millions.
What this does not say
- "Biggest" has no audited answer. Search engines and large companies do not publish full figures, and vendor numbers are claims.
- Some sources are secondary. The Notion and Cursor figures come from summaries of case studies and interviews, not from the companies' own technical papers.
- Why RETRO-style models faded is our reading, not a finding of any paper.
Sources
- Chen et al., SPANN: Highly-efficient Billion-scale Approximate Nearest Neighbor Search, NeurIPS 2021 (Microsoft).
- Subramanya et al., DiskANN (NeurIPS 2019), described in a Microsoft Research talk.
- Borgeaud et al., Improving language models by retrieving from trillions of tokens (RETRO), DeepMind, 2021, and DeepMind's blog post.
- Wang et al., Shall We Pretrain Autoregressive Language Models with Retrieval?, NVIDIA, 2023.
- Wang et al., InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining, NVIDIA, ICML 2024.
- Norlund et al., On the Generalization Ability of Retrieval-Enhanced Transformers, Findings of EACL 2023.
- Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, Journal of Empirical Legal Studies, 2025, and Stanford HAI's summary.
- Uber, Evolution and Scale of Uber's Delivery Search Platform, and Scaling Multilingual Semantic Search in Uber Eats Delivery, 2026.
- Sacra, turbopuffer research note (Notion), Morph comparison (Cursor), and an interview with turbopuffer's founder.
- MarkTechPost, vector database comparison (vendor scale claims).
Planning search over a very large library?
Published 7 November 2026 · Philippe Vergnet, VectoGrid