The example business is Alder & Finch Accounting, a made-up five-person firm with three files behind its website bot: a client FAQ, a fee sheet and an engagement policy.
What are the most common retrieval bot mistakes?
| # | Mistake | Fix |
|---|---|---|
| 1 | Old prices or policies left in a file | Remove them before you upload |
| 2 | Editing a file but not uploading it again | Re-upload after every change |
| 3 | Scanned PDFs with no selectable text | Run OCR first, or use a text file |
| 4 | Two files that disagree | Keep each answer in one current file |
| 5 | No "answer only from the sources" rule | Put it in the instructions |
| 6 | No fallback answer for weak matches | Give one that points to a person |
| 7 | Testing only questions the bot can answer | Add one question it shouldn't know |
Why do the files cause most bad answers?
1. Old prices or policies in a file. Retrieval reads your files on every question, and the bot quotes whatever is there. If last year's fee sheet is still in the knowledge base, the bot can find it and quote it. Remove old prices and policies before you upload.
2. Editing without re-uploading. When you add a file, the tool cuts it into chunks and keeps them in a search index. Change a price in your copy of the file and the index still holds the old chunk. Upload it again, or the bot keeps the old answer.
3. Scanned PDFs. A scanned PDF is a picture of text. The bot sees nothing in it. Use files with selectable text, or run OCR first. The OCR and document parsing FAQ explains the difference.
4. Files that disagree. If the FAQ says one thing and the policy says another, either chunk can match a question. Before you upload, list the questions customers ask most and check each answer is written down in one current file.
Try it free, step by step
How AI Answers from Your Documents: Retrieval (RAG) Explained. Follow one customer question through a support bot: files cut into chunks, a search by meaning, the AI told to answer only from what it found, and a safe fallback when nothing matches. You do every step yourself in a practice copy of Stepthrough, it checks your work as you go, and “Do it for me” finishes any step you get stuck on. Free, and nothing touches your real accounts.
Start the free lessonWhich instruction mistakes let the AI make things up?
5. No source rule. The AI receives three things: instructions, the matching chunks and the question. The instruction "answer only from the sources, say so if they don't cover it" is the whole point. Without it, the AI fills gaps from memory, and that is where wrong prices come from. Grounding and hallucination explain why.
6. No fallback answer. In the lesson, a visitor asks about taxes for a crypto mining rig. None of the three files covers it, and every match is weak (0.31, 0.27 and 0.22). The bot says it isn't sure, doesn't want to guess, and gives Nora's email, phone and office hours. Weak matches are the signal to stop. A plain chatbot with no documents would have answered confidently anyway. The support bot guardrails FAQ shows how to write the fallback and the rules around it.
What testing mistake hides all the others?
7. Testing only the easy questions. A bot that answers your five most common questions well can still guess on the sixth. Before a bot goes live, ask five real questions and one it shouldn't be able to answer. If it guesses on that one, tighten its rules before anyone else sees it.
A related mix-up is thinking the bot was "trained" on your files. It wasn't. Training (fine-tuning) changes the model and is slow to update. Retrieval reads your current files on every question. That is good news: you fix an answer by fixing the file.
Key takeaways
- Bad answers usually start with the files. Fix the source first, then the instructions.
- Remove old prices, re-upload after every edit, and use files with selectable text.
- Keep each answer in one current file so files don't disagree.
- "Answer only from the sources" plus a fallback answer stops made-up replies.
- Always test one question the bot shouldn't know.
More answers are in the retrieval FAQ. To build a real one, see Chatbase questions answered.