A Step-by-Step Framework for Debugging Hallucinations in RAG

One of my AI Engineer interview questions with Philips started with this broken RAG system👇 And here is how you answer it. Your RAG system gives hallucinated answers. Where would you debug first? There are actually quite a few places where things could have gone wrong before the information reached the LLM. So I would start by tracing what happened to between the user query and what information eventually made it into the context. → Did the system call the right tools? → Any metadata filtering issues? → Anything wrong with query rewriting, for example adding the wrong time constraints? → Did retrieval actually fetch the right chunks? → Is the chunk quality good? What is your chunking strategy, and how are you evaluating chunk quality (e. using chunk attribution)? → Do you retrieve all relevant chunks? Look at 𝐫𝐞𝐜𝐚𝐥𝐥: 𝘰𝘶𝘵 𝘰𝘧 𝘢𝘭𝘭 𝘤𝘩𝘶𝘯𝘬𝘴 𝘳𝘦𝘭𝘦𝘷𝘢𝘯𝘵 𝘵𝘰 𝘵𝘩𝘦 𝘲𝘶𝘦𝘳𝘺, 𝘸𝘩𝘢𝘵 𝘱𝘦𝘳𝘤𝘦𝘯𝘵𝘢𝘨𝘦 𝘪𝘴 𝘳𝘦𝘵𝘳𝘪𝘦𝘷𝘦𝘥? → Did we retrieve too much irrelevant context? Look at context 𝐩𝐫𝐞𝐜𝐢𝐬𝐢𝐨𝐧: 𝘰𝘶𝘵 𝘰𝘧 𝘢𝘭𝘭 𝘵𝘩𝘦 𝘥𝘰𝘤𝘶𝘮𝘦𝘯𝘵𝘴 𝘳𝘦𝘵𝘳𝘪𝘦𝘷𝘦𝘥, 𝘸𝘩𝘢𝘵 𝘱𝘦𝘳𝘤𝘦𝘯𝘵𝘢𝘨𝘦 𝘪𝘴 𝘳𝘦𝘭𝘦𝘷𝘢𝘯𝘵 𝘵𝘰 𝘵𝘩𝘦 𝘲𝘶𝘦𝘳𝘺? Now suppose all of that is working. We get the right chunks, but the answer is still wrong. Then the debugging path changes. → Was there conflicting information, for example information from different timelines with completely different views? → Were the prompt instructions clear? → How are we constructing the context? Does the metadata stay together with the chunk? If you just dump chunks together, you may not give the LLM enough context to understand what that text represents. Things like source, timestamp, or other metadata can give the LLM a better world view of the retrieved information. The solution depends on the failure mode. If user queries are ambiguous or underspecified: → Maybe query rewriting. If metadata filtering is wrong: → Check the filtering logic. If retrieval misses exact terms used in the user query: → Hybrid search may help. Although it is often already the default. If retrieved results are relevant but noisy: → Reranking One interesting suggestion I got from the interviewer was about where you place the most relevant chunks after reranking. For example, if you retrieve 5 chunks, instead of simply ordering them from most to least relevant, what if you put the most important ones at top #1 and bottom #5? The reasoning was the “lost in the middle” problem. And this is exactly why observability is non-negotiable. At this point, we have implemented so many things: guardrails, query rewriting, metadata filtering, retrieval, LLM calls, tool calls, tool arguments etc. We need end-to-end traces and span-level visibility. If you only see the final answer, you don’t know where things went wrong. What other RAG failure modes have you run into. Any debugging tip you would share? 💾 Save this for your next interview. ♻️ Repost if you found this useful 💚

When a RAG system produces hallucinated answers, debugging should start by tracing everything that happens between the user query and the information that ultimately reaches the LLM's context — checking whether the right tools were called, whether metadata filtering or query rewriting introduced errors, and whether retrieval actually fetched the correct chunks. Retrieval quality can be evaluated using recall, the percentage of all relevant chunks that were successfully retrieved, and precision, the percentage of retrieved chunks that are actually relevant to the query. If retrieval is working correctly but the answer is still wrong, the failure likely lies in generation — potential causes include conflicting information across sources, unclear prompt instructions, or context construction that strips useful metadata like source and timestamp away from the chunk text, leaving the LLM without enough context to properly interpret it. Different failure modes call for different fixes: query rewriting for ambiguous queries, corrected filtering logic for metadata issues, hybrid search for missed exact-term matches, and reranking for relevant-but-noisy results. One additional consideration is chunk placement after reranking — rather than ordering strictly from most to least relevant, placing the most important chunks at both the top and bottom of the context can help counter the "lost in the middle" effect, a documented phenomenon where LLMs tend to underweight information placed in the middle of a long context window. Because a RAG pipeline involves many interacting components — guardrails, query rewriting, metadata filtering, retrieval, and LLM and tool calls — end-to-end tracing and span-level observability are necessary to identify where a failure actually originated, rather than relying only on the final answer.