Clear
Your agent needs to handle a request under a hard deadline, but it sometimes spends its whole budget on one sub-step and misses the deadline.
How would you design the system to manage a time budget across sub-steps?
General Medium

Answer:

  1. Allocate time per step: Divide the deadline across sub-steps, so no one step consumes it all.

  2. Enforce per-step timeouts: Bound each step, so overruns are cut off.

  3. Track remaining budget: Monitor time left as steps complete, so later steps adapt.

  4. Prioritize essential steps: Ensure critical steps get their time first, so the core is protected.

  5. Return best-available on deadline: Deliver what's ready when time runs out, so the deadline is met.

Your agent works, but its retrieval returns technically relevant results that don't actually help answer the specific question.
How would you design the system to retrieve for answerability, not just topical relevance?
General Medium

Answer:

  1. Target the specific question: Retrieve for what answers the query, not just the topic, so results are useful.

  2. Rerank for answer content: Prioritize chunks that contain answers, so answerability drives ranking.

  3. Decompose the query: Retrieve for each part of a complex question, so all needed content is found.

  4. Check answerability: Verify retrieved context can actually answer, so topically-relevant-but-useless results are caught.

  5. Re-retrieve on insufficiency: Search again when context can't answer, so gaps are filled.

Your agent needs to operate where a wrong action is far worse than inaction, but it's biased toward doing something.
How would you design the system to bias toward safe inaction when appropriate?
General Medium

Answer:

  1. Weight action costs: Factor the high cost of wrong action, so caution is favored.

  2. Require confidence to act: Set a high bar for taking action, so uncertain cases default to inaction.

  3. Prefer reversible steps: Choose actions that can be undone, so mistakes are less costly.

  4. Escalate instead of acting: Route uncertain high-stakes cases to humans, so risky action isn't automated.

  5. Make inaction a valid choice: Treat doing nothing as an acceptable outcome, so the agent isn't forced to act.

Your agent works, but its answers occasionally depend on a fact that was true when indexed but has since been corrected at the source.
How would you design the system to propagate source corrections to answers?
General Medium

Answer:

  1. Track source changes: Detect when source facts are corrected, so propagation can begin.

  2. Re-index corrected content: Update the index when sources correct, so retrieval reflects fixes.

  3. Invalidate affected answers: Clear cached answers built on corrected facts, so stale ones aren't served.

  4. Prefer current source for critical facts: Verify important facts against live sources, so corrections are honored.

  5. Notify on material corrections: Alert users whose prior answers are affected, so acting on corrected info is possible.

Your agent needs to handle a request where the ideal answer requires combining capabilities that individually work but have never been tested together.
How would you design the system to validate capability combinations?
General Medium

Answer:

  1. Test combinations, not just parts: Evaluate capabilities working together, so integration issues surface.

  2. Cover common interactions: Test how capabilities are actually combined, so real usage is validated.

  3. Watch for emergent behavior: Check whether combinations produce unexpected results, so surprises are caught.

  4. Isolate combination failures: Diagnose which interaction breaks, so it can be fixed.

  5. Expand combination coverage: Add newly used combinations to tests, so validation keeps pace.

Your agent works, but a change that improves the average user hurts your most important power users, and you didn't notice.
How would you design the system to protect high-value segments during changes?
General Medium

Answer:

  1. Segment evaluation: Measure changes on key segments separately, so their impact is visible.

  2. Set segment guardrails: Block changes that harm important segments, so they're protected.

  3. Weight high-value cases: Give critical segments extra weight in evals, so their needs aren't averaged away.

  4. Test on power-user scenarios: Include their use cases in testing, so regressions surface.

  5. Monitor segments post-change: Watch key segments after rollout, so harm is caught fast.

Your agent needs to make a call that depends on predicting a future outcome, but it states predictions as certainties.
How would you design the system to handle predictive uncertainty honestly?
General Medium

Answer:

  1. Express predictions as probabilities: Frame forecasts with uncertainty, so they aren't stated as fact.

  2. Ground predictions in evidence: Base them on actual data, so they're not speculation.

  3. Communicate confidence: Signal how certain a prediction is, so users weigh it appropriately.

  4. Avoid false precision: Don't state more certainty than warranted, so predictions aren't misleading.

  5. Note what could change it: Surface factors that would alter the prediction, so users understand its basis.

Your agent works, but its outputs are consumed by both humans and machines, and a format good for one is bad for the other.
How would you design the system to serve both human and machine consumers?
General Medium

Answer:

  1. Separate content from presentation: Generate core content, then render per consumer, so one logic serves both.

  2. Provide multiple formats: Offer machine-readable and human-readable versions, so each gets what it needs.

  3. Structure for machines: Include parseable structure, so machine consumers can rely on it.

  4. Format for humans: Present readable output for people, so it's usable.

  5. Detect the consumer: Serve the right format based on who's consuming, so each is well-served.

Your agent needs to reason over a document that mixes reliable primary content with unreliable user comments, and it treats both as authoritative.
How would you design the system to distinguish authoritative from unreliable content within a source?
General Medium

Answer:

  1. Segment by content type: Separate primary content from comments, so each is treated differently.

  2. Weight by authority: Trust primary content over user-generated parts, so unreliable content has less influence.

  3. Tag content reliability: Mark sections by trust level, so the model knows what to lean on.

  4. Ground answers in the reliable parts: Prefer authoritative content for claims, so answers rest on solid ground.

  5. Flag reliance on unreliable content: Signal when an answer draws on comments, so users judge accordingly.

Your agent is mature and reliable, but you have no framework for deciding when a new capability is worth the added complexity.
How would you design the process to evaluate whether new capabilities justify their complexity.
General Medium

Answer:

  1. Quantify the benefit: Estimate the value a capability adds, so it's weighed concretely.

  2. Assess added complexity: Evaluate the maintenance and risk it introduces, so cost is clear.

  3. Weigh against alternatives: Consider simpler ways to get most of the value, so you don't over-build.

  4. Consider the long-term burden: Factor ongoing cost, not just build cost, so complexity is fully priced.

  5. Decide against clear criteria: Judge on a value-versus-complexity threshold, so decisions are principled, not impulsive.

Your agent needs to serve a request, but doing so well requires capabilities split across systems you don't fully control.
How would you design the system to orchestrate capabilities across systems you don't own?
General Medium

Answer:

  1. Abstract external capabilities: Wrap third-party systems behind stable interfaces, so you're insulated from their internals.

  2. Handle their unreliability: Add timeouts, retries, and fallbacks, so external failures don't break you.

  3. Respect their constraints: Stay within their limits and terms, so integrations remain viable.

  4. Degrade when they're unavailable: Function partially without them, so you don't fully depend on systems you don't control.

  5. Monitor external dependencies: Track their health, so their problems surface as yours before users notice.

Your agent works, but it treats every source it retrieves as equally current, mixing fresh and outdated information.
How would you design the system to account for recency across mixed-age sources?
General Medium

Answer:

  1. Track source dates: Record when each source was published or updated, so age is known.

  2. Weight by recency: Favor fresher sources where currency matters, so outdated data doesn't dominate.

  3. Surface source age: Show when information is from, so users judge currency.

  4. Prefer current for volatile facts: Use the freshest source for fast-changing topics, so answers stay accurate.

  5. Flag potential staleness: Warn when relying on older sources, so users are aware.

Your agent needs to handle a task where the correct approach depends on the outcome of an earlier uncertain step.
How would you design the system to handle branching based on uncertain intermediate outcomes?
General Medium

Answer:

  1. Evaluate the intermediate outcome: Assess the earlier step's result, so branching is informed.

  2. Branch on the result: Choose the path based on actual outcome, so you adapt to what happened.

  3. Handle uncertain outcomes: Account for low-confidence intermediate results, so branching isn't overconfident.

  4. Verify before branching: Confirm the intermediate result when stakes are high, so you don't branch on a wrong basis.

  5. Support re-branching: Allow correcting course if the branch proves wrong, so mistakes are recoverable.

Your agent works, but its cost varies wildly month to month and you can't explain why.
How would you design the system to make cost variation explainable.
General Medium

Answer:

  1. Attribute cost granularly: Break spend down by feature, user, and request type, so drivers are visible.

  2. Track cost over time: Monitor spend trends, so variation is observable.

  3. Correlate with usage: Link cost changes to usage patterns, so causes are found.

  4. Identify anomalies: Flag unusual cost spikes, so surprises are explained.

  5. Report cost drivers: Surface what drove each period's spend, so variation is understood.

Your agent needs to handle requests that arrive with varying quality of input data, but it applies the same trust to all of it.
How would you design the system to calibrate trust to input data quality?
General Medium

Answer:

  1. Assess input quality: Evaluate incoming data's reliability, so trust can be calibrated.

  2. Weight by quality: Rely more on high-quality input, so poor data has less influence.

  3. Verify low-quality input: Cross-check questionable data, so it isn't taken at face value.

  4. Flag quality issues: Signal when answers rest on poor input, so users judge accordingly.

  5. Handle poor input gracefully: Degrade rather than confidently answer from bad data, so errors don't propagate.

Your agent works, but its behavior on a rare input caused a serious incident, and you want to ensure that exact case never recurs.
How would you design the process to permanently prevent a known failure from recurring.
General Medium

Answer:

  1. Add a regression test: Encode the exact case in evals, so it's checked on every change.

  2. Fix the root cause: Address why it failed, not just the symptom, so the class of failure is prevented.

  3. Add a guardrail: Put a deterministic check for the case, so it's caught even if the fix regresses.

  4. Verify the fix holds: Confirm the case now passes, so the fix works.

  5. Monitor for recurrence: Watch for the pattern in production, so any return is caught immediately.

Your agent needs to work under a constraint where certain operations must complete together or not at all, but a failure leaves some done.
How would you design the system to enforce all-or-nothing operation groups?
General Medium

Answer:

  1. Group into transactions: Treat the operations as one atomic unit, so partial completion can't happen.

  2. Stage before committing: Prepare all operations, then commit together, so nothing applies until all can.

  3. Compensate on failure: Undo completed operations when the group fails, so partial state is reversed.

  4. Make operations idempotent: Ensure safe retries, so recovery doesn't compound problems.

  5. Verify group completion: Confirm all-or-nothing held, so inconsistency is caught.

Your agent works, but its answers are only as good as the question, and users often don't know enough to ask well.
How would you design the system to help users ask better questions?
General Medium

Answer:

  1. Suggest refinements: Offer better-formed versions of vague questions, so users get more.

  2. Surface relevant dimensions: Point out aspects they might not have considered, so questions get richer.

  3. Ask guiding questions: Draw out what they actually need, so the real question emerges.

  4. Offer examples: Show well-formed questions, so users learn the pattern.

  5. Answer and expand: Address what they asked while surfacing what they didn't, so gaps get filled.

Your agent needs to reason about a request that references something visual the user described in words, but its mental model of it is wrong.
How would you design the system to handle imprecise verbal descriptions of concrete things?
General Medium

Answer:

  1. Confirm your understanding: Restate your interpretation, so wrong mental models get corrected early.

  2. Ask clarifying questions: Resolve ambiguous descriptions, so you don't reason from a wrong picture.

  3. Request the actual artifact: Ask for the image or data when words are insufficient, so precision replaces guessing.

  4. Surface assumptions: State what you're picturing, so mismatches are visible.

  5. Adapt on correction: Update quickly when the user clarifies, so errors don't persist.

Your agent is being used in ways you never anticipated, and some of those uses are outside what it was validated for.
How would you design the system to detect and handle unanticipated usage.
General Medium

Answer:

  1. Monitor usage patterns: Track how the agent is actually used, so unanticipated uses surface.

  2. Detect out-of-validation use: Flag requests outside tested scenarios, so unvalidated use is known.

  3. Handle unknown use safely: Degrade cautiously on unexpected use, so unvalidated paths don't cause harm.

  4. Assess emerging patterns: Evaluate whether new uses should be supported, so you respond deliberately.

  5. Expand validation: Add newly common uses to testing, so validation catches up to reality.