The Retrieval Problem Nobody Asks About in GenAI Interviews
Interview: Two retrieved documents contain different answers to the same question. One is from last year, while the other was updated yesterday.
Question:
How would you design the retrieval and ranking pipeline to identify the authoritative and most recent information?
Explanation:
Metadata Filtering: Store document version, effective date, source, department, and authority level as metadata during ingestion.
Retrieval: Use hybrid search to retrieve semantically relevant documents, then apply metadata filters where appropriate.
Authority Ranking: Assign higher ranking to trusted sources, such as official policies, over drafts or user-generated documents.
Recency Ranking: Boost documents with newer effective dates, not simply upload timestamps.
Reranking: Use a cross-encoder/reranker combining relevance, authority, version, and freshness signals.
Conflict Detection: If two high-confidence sources still conflict, trigger a conflict-resolution step rather than blindly selecting one.
Generation: Instruct the LLM to prioritize the authoritative, latest effective document and provide citations.
Evaluation: Measure retrieval accuracy, freshness, source correctness, and conflict-resolution accuracy.
Metadata Filtering — ✅ Accurate.
Storing version, effective date, source, department, and authority level as metadata at ingestion time is standard practice in enterprise RAG systems. The distinction between effective date and upload timestamp is a real and important one — most junior implementations get this wrong. Correct to call it out here.
Hybrid Search — ✅ Accurate.
Hybrid search (dense vector + sparse BM25/keyword) is well-established as superior to pure semantic search for retrieval recall, especially when documents share similar meaning but differ in recency or provenance. Applying metadata filters after retrieval (not before) is the right sequence — pre-filtering can silently drop relevant documents.
Authority Ranking — ✅ Accurate.
Weighting official policies over drafts or user-generated content is a legitimate and practiced design pattern. In production systems this is often implemented as a numeric authority score at ingestion, then used as a ranking signal. Nothing overstated here.
Recency Ranking — boosting effective date, not upload timestamp — ✅ Accurate and an underrated distinction.
A document uploaded yesterday with an effective date of 2019 should rank lower than one uploaded last year with an effective date of last week. This is a genuine implementation trap and the explanation correctly identifies it.
Reranking with a cross-encoder — ✅ Accurate.
Cross-encoders (e.g., fine-tuned BERT-based rerankers) evaluating query-document pairs jointly are the standard approach for reranking. Combining relevance, authority, version, and freshness as signals into a reranker is valid — though in practice this is often done as a weighted linear combination of scores rather than a single cross-encoder that natively understands "authority." Minor compression here, not an error.
Conflict Detection — ✅ Accurate, and the most sophisticated point in the explanation.
Most RAG pipelines skip this entirely and just take the top-ranked chunk. Explicitly triggering a conflict-resolution step when two high-confidence sources disagree is correct system design. This could mean flagging for human review, generating a hedged response, or routing to a specialized resolution prompt. The explanation doesn't overspecify — that's fine.
Generation instruction — ✅ Accurate.
Instructing the LLM via system prompt to prioritize the authoritative, latest-effective document and cite sources is standard. Citations matter here because they make the authority signal visible to the end user, not just the system.
Evaluation metrics — ✅ Accurate.
Retrieval accuracy, freshness, source correctness, and conflict-resolution accuracy are all valid eval dimensions. This goes beyond the typical "just measure answer quality" approach most candidates give.
What's slightly compressed but not wrong:
Reranker design — Real production rerankers combining heterogeneous signals (semantic relevance + metadata scores) are usually hybrid scoring functions, not a single cross-encoder end-to-end. The answer implies a unified reranker, which is an idealization. Practically fine for an interview answer, but worth knowing the implementation is usually more modular.
Conflict detection threshold — "Two high-confidence sources still conflict" needs a definition of confidence in practice. How do you measure that two sources are genuinely conflicting vs. complementary? The answer skips this, which is acceptable at interview level but a real design question in production.
Cold start / sparse metadata — The whole pipeline assumes clean, structured metadata exists. In practice, effective dates and authority levels are often missing, inconsistent, or wrong. The answer doesn't address metadata quality, which is a real failure mode.
What's missing (not incorrect, just absent):
- Feedback loops — User corrections or downstream signals (e.g., "this answer was wrong") should feed back into authority scores over time. Static ranking doesn't adapt.
- Chunking strategy interaction — How you chunk documents affects whether the effective date metadata propagates correctly to each chunk. A document updated yesterday where only section 3 changed is a real edge case.
- LLM hallucination risk — Even with the right document retrieved and cited, the LLM can misrepresent it. The eval section should include faithfulness/grounding metrics, not just retrieval accuracy.