RAG vs a regular chatbot: why your AI keeps giving wrong answers
Your AI assistant tells members the wrong refund window with total confidence. Here is why a plain chatbot guesses, and how RAG makes it look things up.

A member asks Lena’s AI assistant how long she has to ask for a refund. It says 60 days, warmly and without a flicker of doubt. Lena’s policy says 14. (Lena is a made-up example: a course creator with about 400 members in her coaching program.)
Here is the short answer to why. A plain chatbot answers from memory, and its memory is the public internet up to a training date. It has never read Lena’s refund page, so it guesses what a refund window usually looks like. RAG (short for retrieval-augmented generation) fixes that by looking the answer up in her own documents before it replies.
There is a catch, though. Even tools built this way, and sold on accuracy, still get up to a third of answers wrong. The difference sits in the parts nobody shows you in a demo.
Your chatbot has never read your program
This isn’t a malfunction. It is how these models are made. A language model learns from a huge pile of public text, then gets frozen at a cutoff date. Three things follow from that, and each one gives Lena’s members wrong answers.
- It knows nothing recent. If Lena moved her live calls from Tuesday to Thursday last month, the model has no way to know.
- It has never seen her business. Her refund policy, her module outline and her community rules were never in its training data.
- It is built to always answer. A model predicts the next likely word, and “I don’t have that” is rarely the likeliest next word.
That last one is the surprise. A confident wrong answer and a confident right answer come out of the exact same machinery. The model can’t tell plausible from correct. So it can’t warn the member either.
Even tools sold on accuracy get up to a third wrong
Wrong answers show up often enough to matter. In 2024, Stanford researchers tested AI legal research tools from LexisNexis and Thomson Reuters. Lawyers buy these products for one reason: accuracy.
The tools still gave incorrect or misleading answers on between 17% and 33% of queries. Here is the part people miss. Those tools already looked things up before answering.
Now picture Lena’s assistant doing that badly. One member question in every three to six would get a wrong answer. And each one would arrive in the same friendly, sure voice as the right ones.
RAG means the assistant looks it up first
The idea is simpler than the name. Instead of answering from memory, the assistant gets a library card.
Here is how it works. A member’s question comes in. The system searches a private collection of Lena’s documents for the passages that match it best. Those passages go into the prompt (the text the model reads before it answers), right next to the question.
Now the model answers from her refund page. Its memory of refund pages in general stays out of it.
The idea isn’t new. A 2020 research paper from Meta’s AI lab gave it the name. In 2026 it is the default design for any serious business AI.
The surprise is what it does to fixing mistakes. When the assistant gets something wrong, Lena edits a document. Nobody retrains anything.
Sources make wrong answers easy to catch
RAG makes the model guess less, because the answer is already sitting in front of it. Everyone expects that part. The part teams underrate is what happens to the errors that remain.
An answer that cites its source can be checked in seconds. Say Lena’s assistant still says 60 days, and links to an old launch page she forgot to delete. She opens the link. Now she knows exactly what to fix.
An answer with no source can only be trusted or not. That is the real gain. RAG turns your assistant into something you can check.
If your AI can’t say “I don’t know,” it isn’t reading your facts. It is improvising, and improvising is where wrong answers come from.
A RAG demo takes an afternoon, and that is the trap
Basic retrieval takes an afternoon to wire up, and it demos beautifully. Then real members arrive.
They type questions half spelled. They ask things that span three documents. They ask about things Lena’s program doesn’t cover at all. The afternoon version starts missing, and it misses quietly.
Remember the Stanford tools. They were shipping products that already used retrieval, and they still got up to a third of answers wrong. It is one more case of the reason most AI projects fail: the demo was never the hard part.
Three pieces of engineering separate a demo from something you can put in front of paying members.
- Chunking means cutting documents into passages so they can be searched. Cut too small, and Lena’s refund rule loses the exception in the next sentence. Cut too big, and the right sentence gets buried.
- Hybrid retrieval pairs search by meaning with plain keyword search. Search by meaning quietly misses exact terms, like a module number or a coach’s name. Keyword search catches them.
- Reranking and evaluation come last. A second pass sorts the passages so the best ones reach the model first. Then a graded set of real questions runs on every change, so a slip shows up before a member finds it.
None of this is exotic. It is the gap between something that worked in a meeting and something that holds up on a busy Tuesday.
Sometimes you don’t need RAG at all
RAG earns its keep when the knowledge is yours, sizeable and changing. For Lena, that means her course library, her policies, her live call schedule and the answers her coaches already wrote in the community.
It won’t fix everything. If the assistant sounds nothing like Lena, that is a behavior problem. The usual fix there is fine-tuning (training the model further on examples of her style). If the job is pure writing with no facts to look up, retrieval adds nothing.
And if the whole program fits on a few pages that never change, skip it. Put those pages straight into the prompt.
How we build it for coaching businesses
We build retrieval the production way. Chunking is tuned to your actual documents. Search is hybrid, with a reranking pass. Every answer cites its source, and an evaluation set catches a slip before your members do.
For coaching businesses, this is the base layer of the AI practice coach we build. A coach can only help your people practice your method if it has read your method. Outside coaching, the same work powers a chatbot that answers from your facts and shows its sources.
It starts with a free call. If a build makes sense, we run a free audit of where your current assistant goes wrong and tell you plainly whether retrieval fixes it. Then you get a fixed quote in writing. You own the code and the data.
Back to Lena’s 60 days. With retrieval, her assistant pulls up her refund page, says 14 days and links to it. When she changes the policy next spring, she edits one page.
A chatbot that gets your facts wrong was never shown your facts. If yours is giving members confident wrong answers, book the free call and bring three of the worst ones.
Sources
- Magesh, Surani, Dahl, Suzgun, Manning and Ho (Stanford), Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools, 2024: https://arxiv.org/abs/2405.20362
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020: https://arxiv.org/abs/2005.11401

