How Booking.com Builds Judge-LLMs: A Practical Framework for Evaluating GenAI in Production

Booking.com released a guide on how they evaluate every single AI app they build.
Evaluating LLMs in prod is a different game than evaluating traditional ML models.
- Golden datasets matter: human-annotated data is still the foundation for building trustworthy judge-LLMs. Without reliable labels, automated evaluation breaks down.
- Annotation protocols are key: whether you go with a single annotator (basic) or multiple with consensus/weights (advanced), the consistency of annotation directly impacts evaluation quality.
- Judge-LLM can't be the same as target-LLM: a stronger LLM can be used to evaluate the outputs of another, allowing scalable and automated monitoring of GenAI systems.
- Pointwise vs. comparative judges: pointwise scoring works for production monitoring, but comparative evaluation (A vs. B) often provides stronger signals for ranking and system improvement.
- Automation + synthetic data are emerging directions: auto-prompt pipelines and synthetic golden datasets could significantly reduce the time and cost of judge-LLM development.
♻️ Share it with anyone who might benefit :)
I regularly share AI Agents and RAG projects on my newsletter 𝑨𝑰 𝑬𝒏𝒈𝒊𝒏𝒆𝒆𝒓𝒊𝒏𝒈 𝑾𝒊𝒕𝒉 𝑺𝒂𝒓𝒕𝒉𝒂𝒌: https://lnkd.in/gaJTcZBR
Link to full article: https://lnkd.in/g3-qWFhB

I found and read the actual source article — "LLM Evaluation: Practical Tips at Booking.com" by George Chouliaras, Zeno Belligoli, and Antonio Castelli, published on Booking.com's official ML & DS blog (Aug 12, 2025). The post is a solid, mostly accurate summary of a real piece.

• Golden datasets as the foundation — confirmed, this is the article's central thesis and its explicit "key takeaway."
• Basic (single annotator) vs. Advanced (multiple annotators with consensus/weighting) protocols — confirmed precisely. The article names them exactly this way: Basic uses one annotator; Advanced uses 3+ annotators with explicit strategies for disagreement — a "not sure" class, a confidence weight (n_selected_votes/n_total_votes), or discarding ambiguous examples.
• "Judge-LLM can't be the same as target-LLM" — slightly overstated. The article doesn't state this as a hard rule; it recommends using a strong/powerful model (GPT-4.1, Claude 4.0 Sonnet) as the judge backbone, and notes that after tuning, a weaker, cheaper model is often substituted for production monitoring. The spirit ("use a more capable evaluator") is right, but "can't be the same" phrases a best practice as an explicit prohibition that isn't in the source.
• Pointwise vs. comparative judges — confirmed accurately, including the nuance that pointwise judges serve dual purposes (ranking + production monitoring) while comparative judges give stronger ranking signals and suit development-phase use more.
• Automation + synthetic data as emerging directions — confirmed. The article describes an OPRO-inspired automated prompt-engineering pipeline and flags synthetic golden datasets as an active, unsolved area of work.
• One omission: the post doesn't mention the article also explicitly excludes agent evaluation as a separate, uncovered topic — not an error, just a scope note worth knowing.