How is RAG used in chatbots?

In a RAG-powered chatbot, the retriever doesn't just run once at the start of a conversation — it runs again on nearly every message, because what the user is actually asking keeps shifting as the conversation goes on. Each turn, the bot re-reads the relevant documents before it replies.

What happens on each message?

Mechanically, a single turn runs through three steps:

• Query rewriting: before searching, the system folds your latest message together with recent conversation history into one self-contained search query. That's why "what about the enterprise plan?" as a follow-up doesn't confuse the retriever — it knows you're still asking about pricing from three messages ago.
• Re-retrieval on follow-ups: each new question can pull a completely different slice of the knowledge base. Ask about refunds, then shipping, then a specific SKU, and the bot fetches three different sets of documents rather than reusing whatever it found first.
• Context injection: the retrieved snippets get stuffed into the prompt alongside the chat history, so the model answers with your actual return policy or product spec in front of it, not just what it remembers from training.

Why do RAG chatbots hallucinate less?

Because the model is being asked to summarize a document it can see, rather than recall a fact it half-learned during training. On support questions especially, that shift — from memory to reading — is what keeps a RAG chatbot anchored to real policy instead of inventing a plausible-sounding answer. It can still get things wrong, but it's working from evidence rather than guesswork.

What are the tradeoffs?

They're real. Every turn now costs an extra search step, which adds latency you'll notice in a live chat. And the whole system is only as good as the knowledge base behind it — if support docs go stale, the chatbot will confidently repeat outdated policy word for word.

Good retrieval hides these costs; sloppy retrieval makes them obvious, and the fix is almost always better documents rather than a bigger model.