How RAG Has Changed Over the Years

A GenAI Newsletter by Raj

When generative AI moved from demos into production, RAG was the route most of it took. Companies didn't need a model that knew everything. They needed one that could answer from their own documents, and RAG was the most practical way to get there.

Six years on, that job hasn't changed. Almost everything about how it's done has. This post walks through what RAG is, why it became the default for enterprise AI, and each stage it has gone through: vector search, knowledge graphs, memory, and now agents that do their own reading. Across all of them, the biggest change is who does the job.

I've worked on retrieval for most of my career: search ranking on the founding team of Bing, grounding for Microsoft 365 Copilot Chat, and now agents at AgentNava. Each stage below is one I've seen up close.

Six years of RAG: a documents thread and a memory thread converge into agents that do their own reading

What RAG Is, and Why It Matters

RAG is retrieval-augmented generation: before the model answers, find the relevant information and put it in front of the model.

RAG turns a closed-book exam into an open-book one. That is why most enterprise AI in production is some form of RAG: knowledge assistants, support, contract review, coding assistants. A general model gives every company the same intelligence. A company's own data is what makes it useful, and RAG is how that data reaches the model.

Why not just fine-tune the model on your documents? Many teams tried that in 2023: supervised fine-tuning (SFT) on company manuals, hoping the model would absorb them. It mostly didn't work, and the research explains why:

Fine-tuning teaches behavior, not facts. The LIMA paper (Meta, 2023) argued that a model's knowledge comes almost entirely from pretraining; fine-tuning mainly teaches it how to respond.
New facts are learned slowly, and they raise hallucination. A Google study (Gekhman et al., 2024) found models absorb new facts through fine-tuning much more slowly than familiar ones, and that once they do, they hallucinate more.
Retrieval wins head to head. A Microsoft study (Ovadia et al., 2023) compared the two for injecting knowledge, and RAG consistently beat fine-tuning.

Fine-tuning still has a job: tone, format, and domain style. But for knowledge, retrieval wins on four counts: answers reflect a document the moment it changes, every answer can cite its page, the user only sees what they're allowed to see, and nothing needs retraining. Fresh, private, cited, permission-aware. That's why RAG keeps getting reinvented rather than replaced.

RAG Does Not Mean Vector Database

The R is retrieval, and retrieval is any way of finding the right information. Bing Chat, the first RAG product most people used, grounded its answers in a web search engine. It can also be a SQL query or an API call. This post is about the most common case, knowledge that lives in documents, where the main methods are:

Method	How it finds things	Best for	Weak at
Traditional keyword search (BM25, "sparse")	Matches the query's words, weighted by rarity	Names, codes, exact terms	Synonyms and paraphrase
Semantic search (embeddings, "dense")	Matches meaning via embeddings	Paraphrased, fuzzy questions	Exact terms, IDs, numbers
Hybrid	Runs both, merges the rankings	Most document search	Two systems to tune
Knowledge graph	Follows links between entities	Multi-hop, "what is affected" questions	Building and updating the graph
Agentic (`ls`, `grep`, read)	The model searches and reads files itself	Hundreds to thousands of documents	Millions of documents

Keyword search was never the weak option. BM25, the ranking formula behind Lucene and Elasticsearch, dates to the 1990s and has run enterprise search for decades. It scores a document by how often the query's words appear, how rare those words are, and how long the document is. When the BEIR benchmark (2021) tested retrievers on 18 datasets they weren't trained on, BM25 stayed one of the strongest baselines and beat many neural retrievers.

So why did vector databases become the default? Not search quality:

Embeddings match meaning. "Time off" finds "leave policy." That was the real gain.
App teams weren't search teams. A good search stack existed, but tuning one was a specialty. A vector database was one API call.
The tutorials used one. So everyone did.

RAG and vector databases became synonyms, and many teams dropped keyword search that would have served their questions better.

The teams that already had a search stack built hybrid from the start. Microsoft 365 Copilot is the clearest example: it grounds answers in years of Microsoft Search infrastructure, a lexical and semantic index that combines keyword matching with vectors, personalization, and the permissions every enterprise file already carries. Microsoft added a vector index to a search engine that already worked.

Organizing Knowledge Is the Real Work

RAG is a retrieval problem, and retrieval only finds what was organized to be found. Every era of RAG was, underneath, a different way to organize knowledge:

Era	How knowledge was organized	What it made findable
Search engines	An inverted index of words	Exact terms
Vector databases	Chunks placed by meaning	Paraphrases
RAPTOR	A tree of summaries	Chapters and themes
Knowledge graphs	Entities and their links	Connections
Memory	Facts written down over time	What happened before
Agents	Folders, file names, headings	Whatever a careful reader would find

People have always worked this way. A library is only useful because someone catalogued it. A well-run team keeps a shared drive where folder names mean something.

It starts with ingestion, the half nobody demos. Bad data in, bad retrieval out: parse PDFs, scans, and tables without flattening them, strip the boilerplate, normalize dates and IDs, retire old versions, and remember that a policy PDF is only a snapshot of the live system.

Then organize it so it can be found, and traced back. Finding the right page is only half the goal. The other half is being able to show where an answer came from, because an answer nobody can cite is an answer nobody should trust.

For finding, name files for what they contain (refund-policy-2026.md, not doc7_final_v2.pdf), keep one topic per file with real headings, and put dates and versions where they can be seen. For tracing, every converted or summarized file should keep a link to its original, and paths and headings should stay stable, so a citation points to the exact section rather than "somewhere in the PDF."

This is the oldest form of retrieval engineering, and every stage below inherits whatever it gets wrong.

2020: The Paper

Lewis et al. at Facebook AI Research coined the term: a retriever pulled Wikipedia passages, and a generator answered with them in view. It stayed in research until ChatGPT made every company ask: can it answer from our documents?

2022-2024: The Vector Database Boom, Then the Patches

The recipe: chunk documents, embed each chunk, store the vectors, fetch the nearest chunks, paste them into the prompt. LangChain made it a ten-line tutorial.

The context window shaped all of it. In late 2022, a model could read about 4K tokens at once, roughly six pages. A 300-page policy had to be cut into small pieces, and only a handful could be shown to the model. Retrieval had to pick those few pieces perfectly, and chunking is where it failed most often:

The exception gets separated from the rule. The limit is on page 40; "except for enterprise customers" is the next paragraph, in another chunk.
References lose their target. "This policy" or "the table above" means nothing once cut loose. Half a table looks complete and isn't.
Size is a no-win trade. Small chunks match precisely but carry no context. Large chunks carry context but blur the embedding.

The industry tried every way of cutting:

Approach	How it cuts	What it still misses
Fixed size	Every N tokens, with overlap	Ignores meaning entirely
Semantic	Where the topic shifts	Rule and exception share a topic, so they still split
Document-aware	Along headings, sections, tables	Cross-references between sections
Parent-child	Search small pieces, return their parent section	Parent may still be the wrong section
Contextual	Prefix each chunk with where it came from (Anthropic)	A model call per chunk at ingestion
No chunks	The agent reads whole files	Needs search first, and limits on file size

Each row fixes part of the problem. Only the last one removes it.

Then every other box in the recipe got a patch: hybrid search, rerankers (fetch 50, keep 5), RAPTOR's tree of summaries, and loops like Self-RAG and Corrective RAG that grade the evidence and search again. As windows grew through 2023 (32K, then 100K and 200K tokens), chunks got bigger and more of them fit, which gave rerankers room to work. Each patch helped. Together, they turned a four-step recipe into a dozen tunable parts, and nearly every patch was making a fixed pipeline act a bit more like a careful reader.

2023-2025: Knowledge Graphs and Memory

Some questions need connections, not passages. "If we close this bridge, which projects are affected?" No chunk holds that answer. In 2024, models started building knowledge graphs from plain text: GraphRAG (Microsoft) extracts entities and relations and summarizes clusters of them, HippoRAG follows multi-hop links, LightRAG makes graphs cheaper to update.

This is where most teams got lost, developers and non-developers alike:

A new vocabulary, all at once. Entities, ontologies, triples, community detection, graph databases, a new query language.
Expensive to build. GraphRAG runs a model over every chunk to build the index. Microsoft later released LazyGraphRAG to cut that cost.
Hard to debug. A wrong answer could come from extraction, the graph, a summary, or the query.
Adopted because it sounded advanced. Teams built graphs for questions that were simple lookups.

With chunking strategies, embedding models, rerankers, and now graphs piled up, RAG started to feel too complex for many teams to build themselves. Some bought a platform, some stalled at the prototype, and a lot of "chat with your documents" projects never reached production. Graphs earn their place only when the answer lives in relationships, and most enterprise questions don't.

At the same time, a second thread began. Memory is RAG over the agent's own history instead of company documents. Generative Agents (Stanford, 2023) ranked memories by recency, importance, and relevance. MemGPT treated the prompt as RAM and an outside store as disk, with the model paging facts in and out. ChatGPT memory, Mem0, and Zep brought it to products, and Zep stores memory as a graph with time attached, so it knows an old address is no longer true.

Memory brought its own failure mode: it is written by the model, and goes stale. A wrong memory is worse than none, because it's retrieved with confidence every time.

2022-2025: Context Windows Grow 250 Times

The context window is how much text a model can read at once. Its growth changed what RAG had to do:

When	Typical window	About	What it meant for RAG
Late 2022	4K tokens	6 pages	A few small chunks. Retrieval had to be precise.
2023	32K to 200K	50 to 300 pages	Bigger chunks, more of them. Rerankers had room.
2024	1M (Gemini 1.5)	1,500 pages	Whole documents fit. "RAG is dead" takes everywhere.
2025-26	1M is common	1,500 pages	Room for an agent to search, read, and reason.

Bigger windows didn't kill RAG (Context Windows Keep Growing): they're slow and costly when full, models still lose the middle (Lost in the Middle), and a company's documents run far past 1,500 pages. What they changed: chunking matters less, retrieval can fetch ten candidates instead of three, and the agent loop became possible. At 4K tokens there was no room to loop. At 1M there is.

2025: Context Engineering

Finding the right evidence is half the job. The other half is deciding what reaches the model, and in what order. In 2025 this work got a name: context engineering. Say a doctor asks a question, and retrieval returns four documents: a medication guide, a contraindication warning, the hospital policy, and a clinical guideline. All four are right. The answer can still be wrong:

If the warning lands in the middle of a long context, the model may skim past it.
If the guideline appears twice, it crowds out the policy.
If nothing says which policy is current, the model may blend this year's with last year's.

Context engineering prevents this. Put the critical evidence where the model will notice it. Remove duplicates and boilerplate. Label every piece with its source, date, and version. Keep the whole thing within budget, because evidence competes with instructions for the same window.

Retrieval decides what the model can know. Context engineering decides what it uses.

2025-2026: Agents Do the Reading

The biggest shift of the last two years is that the model now does the searching. Coding agents went first: the Claude Code team has said it dropped its vector index because letting the agent search with ordinary tools worked better. The agent lists folders, runs grep, opens promising files, reads them whole, and searches again if it missed.

As the vector database moves out of the center, so does the embedding model. In 2023, teams spent weeks comparing embedding models on public leaderboards. When the agent reads whole files and matches exact terms itself, which embedding model you picked stops deciding the quality of the answer. Embeddings stay useful for hybrid search and for collections too large to browse, as one tool among several.

The loop is the difference. Classic RAG gets one shot: one query, one set of chunks, one answer. If retrieval misses, nothing notices. An agent gets as many shots as it needs:

Failure	One-shot RAG	Agent loop
First search misses	Answers from the wrong chunks	Tries other words, other folders
Answer spans documents	Gets whichever chunks scored highest	Reads one, follows the reference to the next
Exception on the next page	Never sees it	Keeps reading
Two versions disagree	Blends them	Checks dates, picks the current one
Answer isn't there	Makes something up	Searches, finds nothing, says so
Easy vs hard question	Same pipeline for both	One search for easy, ten for hard

Retrieval stopped being a step before the answer and became part of the reasoning. The agent even does its own context engineering, choosing what to open, keep, and summarize away.

From a pipeline to a loop: a 2023 fixed pipeline with a setting to tune at every step, versus a 2026 agent loop that searches, reads, decides, verifies, and searches again

We made this choice at AgentNava. Our agents don't get a vector database. They get a read-only folder of the customer's documents and the same shell tools a person would use. Nothing to keep in sync when a file changes, and when an answer is wrong, the agent's search history shows exactly what it looked at.

Memory meets retrieval here too: agents keep notes where they search, from project instruction files to wikis the model maintains itself, like Karpathy's LLM Wiki idea.

The loop has costs.

Cost	One-shot RAG	Agent loop
Latency	One retrieval, one model call: a second or two	Every search and every file read is another model round trip. Ten steps can mean tens of seconds.
Tokens	A few chunks per question	Whole files, plus the agent's own reasoning at every step. Cost per question can be many times higher.
Corpus size	An index answers in milliseconds over millions of documents	`grep` over a folder is fast for thousands of files, slow for millions
Concurrency	Thousands of users share one index	Every user runs their own loop, each holding its own context
Predictability	Same path every time	A different path every time, which makes response times harder to promise

For a support chat or a voice agent, tens of seconds is too slow, however good the answer. For a contract review or a research question, it is an acceptable wait.

In practice, the agent should use indexes as tools:

Route by difficulty. Answer simple lookups in one shot; send only the hard questions into the loop.
Use an index for the first hop. At millions of documents, a keyword or semantic index narrows the search to a few folders, and the agent reads from there.
Use small, fast models for the search steps, and save the large model for reading and answering.
Run searches in parallel, not one after another.
Let notes do the remembering. An agent that wrote down where the refund policy lives doesn't search for it again.

Then there's control. A pipeline does exactly what it was built to do: you know which index it searched, how many chunks it fetched, and what reached the model. An agent decides for itself, and an enterprise has to be able to bound those decisions:

What it can see. Read-only access, scoped to the folders this user is allowed to open. The system enforces permissions; the prompt does not.
What it can do. An allowed list of tools and commands. Searching and reading, yes; writing, deleting, or calling out to the internet, no.
How much it can spend. A cap on steps, tokens, and time per question, with a clear answer when the cap is hit.
What it read. A full log of every search and every file opened, so any answer can be audited after the fact.
What it trusts. Documents can contain instructions too. An agent that reads whole files must treat their contents as data, never as commands.

These are the same controls you'd put around a new employee with access to company files. A pipeline had most of them by construction. With an agent, they have to be designed in from the start.

The rest of the limits are about the knowledge itself: it has to be well organized, huge files must be searched before they're read, and notes must keep their sources.

Evaluation: Not an Afterthought

Most teams treat evaluation as an afterthought: they build the RAG system first and add evaluation once it's running. By then, the decisions that matter most are already made: how documents are split, which retrieval method, which index, how answers are grounded. Evaluation can only tune what's left, so it delivers incremental improvements to an architecture it never got to question.

Start evaluating before those decisions are locked, and it can tell you which architecture to build. Much of my work on Copilot was metric design, and the main lesson carries over: score retrieval and the answer separately. One blended "quality" number hides which half broke.

What to check	The question it answers
Retrieval recall	Did the right passage come back at all?
Faithfulness	Does the answer say only what the sources say?
Answer quality	Does it answer the question asked?
Citation	Does the cited page support the claim?
Refusal	When the documents don't have it, does it say so?
Access	Does it ever surface a file this user can't open?

Build the test set from real user questions, including ones the documents can't answer, and rerun it on every change. With agents, also evaluate the path: what it searched, what it read, how many tries it took, and what that cost.

Where This Is Going

Look at the whole arc. Chunking strategies, rerankers, knowledge graphs, memory stores, context engineering: every one of them was an engineer approximating, ahead of time, what a careful reader does when they look something up. We spent six years engineering around a reader that couldn't read. Now it can.

So my prediction: RAG converges on how a skilled person finds information, writes an answer, and checks it, with far more powerful tools.

A skilled person	An agent
Knows where to look	Lists folders, reads the index, picks a tool
Skims, then reads closely	`grep` across thousands of files, then reads whole documents
Follows a reference	Opens the next file, as many times as needed
Notices two sources disagree	Compares dates and versions, keeps the current one
Checks every claim	Verifies each claim against its source
Says "I don't know"	Reports what it searched and that nothing was found

If that's right, a few things most teams believe today will flip:

Your RAG is only as good as your shared drive. Retrieval quality stops being an infrastructure problem and becomes a knowledge-management problem. The edge moves from who has the best retrieval stack to who has the best-organized knowledge.
Documentation becomes infrastructure. The policy owners, technical writers, and operations leads who maintain a company's documents become central to its AI system, more so than whoever tunes the embeddings.
Trust comes from citations. Enterprises will pick the agent that shows the exact passage behind every claim, and says "I couldn't find it" when it couldn't, over the one with the best benchmark score.
"Which vector database?" will sound like "which ORM?" Indexes don't disappear. They become what a catalog is to a librarian: one tool among several, reached for when the collection is too big to browse.

RAG didn't die. It grew up into the way a careful person works: search, read, answer, verify, remember. Only faster, and at the scale of everything a company knows.

One question to ask of any RAG project today: if you handed your documents to a brilliant new hire with no context, could they find the answer?

Further Reading:

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: The original RAG paper (Lewis et al., 2020)
Fine-Tuning or Retrieval?: Why retrieval beat fine-tuning for adding knowledge
BEIR: The benchmark where BM25 held its own against neural retrievers
GraphRAG: Knowledge graphs built by a model, from plain text
MemGPT: Memory the model manages itself
Lost in the Middle: Why a bigger context window isn't the whole answer