Stop Evaluating Agents Like a Single LLM Call: AWS's 3-Layer Framework
Most teams still evaluate like it’s a single LLM call.
But agents aren’t just outputs anymore. They’re systems.
- multi-step reasoning
- tool selection + execution
- memory retrieval
- multi-agent coordination
If you only evaluate the final response, you miss why things break.
AWS just shared how they evaluate agents internally, and it’s a much more practical approach.
They break evaluation into 3 layers:
1. Bottom layer: model performance (latency, accuracy, cost)
2. Middle layer: components (intent detection, planning, tool use, memory)
3. Top layer: final outcome (task success, UX, safety)
Instead of asking “did it answer correctly?”, you can localize failure:
- planning score: was the task decomposition valid?
- tool selection accuracy: was the right capability invoked?
- tool call error rate: did execution fail or inputs break?
- grounding / faithfulness: did reasoning stay consistent with context?
- multi-turn coherence: did state drift over time?
Another key shift: evaluating trajectories, not just outputs.
Agent traces (reasoning steps, tool calls, intermediate states) become first-class evaluation artifacts.
- replay traces against new versions -- regression testing
- generate synthetic eval datasets from production logs
- benchmark tool-use sequences, not just answers
This is especially critical in multi-agent setups, where failure modes include:
- poor task decomposition by the orchestrator
- incorrect agent assignment
- communication breakdown between agents
- inconsistent aggregation of results
Finally, one thing that consistently shows up in production:
You need human-in-the-loop, not as a fallback, but as part of the eval loop.
- calibrate LLM-as-a-judge
- audit edge-case trajectories
- validate reasoning quality, not just correctness
♻️ Share it with anyone who is building AI agents in production :)
I share tutorials on how to build + improve AI apps and agents, on my newsletter 𝑨𝑰 𝑨𝒈𝒆𝒏𝒕 𝑬𝒏𝒈𝒊𝒏𝒆𝒆𝒓𝒊𝒏𝒈: https://lnkd.in/gaJTcZBR
Link to article: https://lnkd.in/eFT45Gsq
This one is unusually accurate — it's a faithful summary of a real AWS blog post, not a loose paraphrase of vibes.
Source confirmed: "Evaluating AI agents: Real-world lessons from building agentic systems at Amazon," published on the AWS Machine Learning blog (Feb 18, 2026) by four named AWS/Amazon authors (Yunfei Bai, Allie Colin, Kashif Imran, Winnie Xiong).
"Thousands of agents" — confirmed verbatim claim. The AWS post states directly: since 2025, there have been thousands of agents built across Amazon organizations. amazon
The 3-layer framework is accurately represented, with one small misplacement. AWS's actual layers: Bottom layer: Benchmarks multiple foundation models to select the appropriate models powering the AI agent and determine how different models impact the agent overall quality and latency. Middle layer: Evaluates the performance of the components of the agent, including intent detection, multi-turn conversation, memory, LLM reasoning and planning, tool-use, and others. Upper layer: Assesses the agent's final response, the task completion, and whether the agent meets the goal defined in the use case. It also covers overall responsibility and safety, the costs, and the customer experience impacts. The post puts "cost" in the bottom layer; AWS actually places cost in the top layer alongside safety and UX. Minor, but worth a correction if precision matters.
The specific metrics (tool selection accuracy, tool call error rate, faithfulness, grounding) are pulled directly from AWS's own metric library — Tool selection accuracy: Did the AI assistant choose the appropriate tool for a given situation. Tool call error rate: The frequency of failures when an AI assistant makes tool calls. Faithfulness score: Measures logical consistency across the reasoning process. These aren't paraphrased or invented — they're the real terminology.
One nuance: "planning score" is presented in the post as a general single-agent metric, but in AWS's article it's specifically introduced for multi-agent orchestration: Amazon teams use the metrics such as the planning score (successful subtask assignment to subagents), communication score (interagent communication messages for subtask completion), and collaboration success rate. Not wrong, just narrower in the source than the post implies.
Trajectory-based regression testing and synthetic eval datasets — confirmed, from the shopping assistant case study: Amazon teams created golden datasets for regression testing. The datasets are generated synthetically using LLMs from historical API invocation logs upon user queries.
Multi-agent failure modes and the HITL argument — confirmed, closely matching AWS's own language on why human review can't be automated away: HITL becomes critical because of the increased complexity and potential for unexpected emergent behaviors that automated metrics might fail to capture, used for calibration of LLM-as-a-judge in the automatic evaluator to align with human preferences.
Nothing here is fabricated or overstated — this is a genuinely solid, well-attributed summary.