How to tune your knowledge base when the agent gives wrong answers
When your agent gives a wrong answer, the instinct is often to edit the prompt — add more instructions, make the guardrails stricter, tell the model to be more careful. That instinct is usually wrong. The majority of wrong answers in production are retrieval problems, not model problems. Here's how to tell the difference and fix the right thing.
Two kinds of wrong answers
The first kind: the agent says something false. The answer is confident, specific, and incorrect. This is usually a KB problem — the article covering this topic is outdated, or there's no article at all and the model is filling the gap with inference.
The second kind: the agent says something irrelevant. The answer is plausible but doesn't address what the customer actually asked. This is almost always a retrieval failure — the embedding similarity matched on surface-level keywords rather than on meaning, and the wrong chunks were passed to the model. The model then answered faithfully based on content that wasn't relevant to the question.
Treating both kinds the same way wastes effort. Identify which one you're looking at before you start fixing anything.
Root cause 1: outdated documentation
This is the most common cause of false answers. Your product shipped an update; your KB didn't. The agent retrieves the old article and confidently answers based on how the product used to work. From the agent's perspective, it found a relevant article and answered correctly. The article is just wrong.
The fix is a doc review cadence. For fast-moving products, weekly. For stable products, monthly. The trigger event for every product change should include a KB review task, the same way it includes a changelog entry.
When you update an article that has significantly changed in structure, delete the old version rather than editing in place. Embedding indexes often cache chunk vectors from the previous version until the next crawl. An article you've substantially revised can still serve stale chunks if the old vectors haven't been purged. A clean delete-and-recreate ensures the retrieval system starts fresh.
Root cause 2: the chunk doesn't answer the question
Your article about password reset has a paragraph that also mentions changing your email address. A customer asks “how do I change my email?” The retrieval system returns the password reset article because it contains both concepts. The model reads that chunk and gives a partial, confused answer — or gives the password reset instructions instead.
This is a one-concept-per-chunk problem. The retrieval system is doing its job: it found a chunk that mentions the topic. The chunk just contains too many topics for retrieval to be reliable.
The fix is splitting. If an article covers multiple distinct tasks, each task becomes its own article or its own clearly bounded section that can be chunked independently. The test: can this chunk answer the target question completely, without requiring the reader to have read anything else? If not, it needs to stand alone before it will retrieve reliably.
Root cause 3: the KB doesn't cover it at all
The agent was asked about something not in the KB. If your escalation guardrails are configured correctly, it escalates. If not, it either refuses to answer or, worse, fabricates something plausible.
Pull your escalation logs and look for conversations where the agent escalated but no KB article would have covered the question. That list is your content gap. For each item, make one of two decisions: this should be answered by the agent (write the article), or this should always escalate regardless of KB coverage (add an explicit escalation signal for this topic category).
Billing disputes and account security questions often belong in the second category — even if an article technically covers the topic, the right answer depends on context that a KB can't fully capture, and the cost of a wrong answer is high enough that you want a human involved.
The diagnosis workflow
When a wrong answer is reported: pull the conversation ID. Look at what KB chunks were actually retrieved for that turn — your platform should surface this in the conversation inspector. Then ask one question: does the right chunk exist?
If the right chunk exists but wasn't retrieved, you have a retrieval tuning problem. If the right chunk doesn't exist, you have a content gap. These require different fixes, and conflating them leads to a lot of KB editing that doesn't change retrieval behavior.
Retrieval tuning when the right chunk exists
Embedding models retrieve better when the chunk text matches the query style. A procedural article written in the third person (“Users can cancel by navigating to Settings”) doesn't embed the same way as a question a customer would actually ask (“how do I cancel my subscription?”).
The fix is adding a Q&A variant to the article. A literal question-and-answer pair that mirrors natural customer phrasing, placed at the top of the article or as a distinct chunk. Something like: “Q: How do I cancel my subscription? A: Go to Settings → Billing → Cancel plan, then confirm.” That format retrieves reliably because the query and the chunk are semantically similar in the way the embedding model measures.
You don't need to add Q&A variants to every article — only the ones where the right chunk exists but isn't being retrieved for the obvious customer query. Target the diagnosis before the fix.
When to patch vs. when to rebuild
Most KB problems should be patched at the specific gap. You found a stale article, you updated it. You found a chunking problem, you split the article. You found a retrieval miss, you added a Q&A variant. These are surgical fixes and they work.
Rebuild when you've made targeted patches to more than 20% of your articles in a short period. That pattern usually means the underlying structure of the KB is wrong — articles are organized by how your team thinks about the product, not by the questions customers actually ask. Rebuilding around customer questions rather than product features tends to reduce retrieval failures more durably than continuing to patch a structure that doesn't fit the query patterns.
Numbers in this post that cite no source are rules of thumb, not Velor measurements. Velor has no customer deployment to measure yet.