Clear
Your agent needs to handle a request where the user wants it to advocate for a position, but balanced treatment would serve them better.
How would you design the system to balance advocacy against balanced treatment?
General Medium

Answer:

  1. Clarify what they want: Determine if they want a case made or a fair overview, so you serve the real need.

  2. Advocate when asked: Make the requested case well, so you honor the explicit ask.

  3. Surface the other side: Note counterpoints even when advocating, so users aren't misled.

  4. Flag when balance serves better: Suggest a balanced view if it fits their goal, so they get what actually helps.

  5. Respect their choice: Give the advocacy they asked for if they still want it, so you don't override them.

Your agent works, but it can't recognize when its confident answer to one part should lower its confidence about a related part.
How would you design the system to propagate confidence across related answers?
General Medium

Answer:

  1. Model answer dependencies: Recognize when parts of an answer relate, so confidence can propagate.

  2. Propagate uncertainty: Let uncertainty in one part temper confidence in dependent parts, so overconfidence is avoided.

  3. Check internal consistency: Verify related answers cohere, so contradictions in confidence surface.

  4. Communicate compound confidence: Express overall reliability across related parts, so users judge the whole.

  5. Verify the linchpins: Firm up answers others depend on, so propagated uncertainty is reduced.

Your agent needs to handle a request where the user provides more constraints than any solution can satisfy, but doesn't realize it.
How would you design the system to handle unknowingly over-constrained requests?
General Medium

Answer:

  1. Detect the over-constraint: Recognize when constraints can't all be met, so it's caught.

  2. Show the user the conflict: Explain that the constraints are incompatible, so they understand.

  3. Identify the binding constraints: Point out which ones clash, so the problem is concrete.

  4. Propose which to relax: Suggest trade-offs to make it solvable, so a path forward exists.

  5. Offer the best compromise: Give the closest achievable solution, so users get value.

Your agent works, but it can't tell when a user asking "can you" wants a yes/no versus wants it to actually do the thing.
How would you design the system to interpret capability questions correctly?
General Medium

Answer:

  1. Read intent behind "can you": Judge whether they want confirmation or action, so you respond right.

  2. Use context: Interpret against the conversation, so ambiguous phrasing resolves.

  3. Default to helpful: Confirm and offer to do it, so both interpretations get value.

  4. Confirm before consequential action: Verify intent when acting has effects, so you don't act unwanted.

  5. Ask when unclear: Clarify if genuinely ambiguous, so you don't guess wrong.

Your agent needs to handle a request where the correct answer today may be seen as wrong once more is known, and users will blame it later.
How would you design the system to give answers that age well?
General Medium

Answer:

  1. Ground in current best knowledge: Base answers on what's known now, so they're defensible.

  2. Convey appropriate uncertainty: Signal where knowledge is provisional, so answers aren't overstated.

  3. Note what could change: Flag factors that might revise the answer, so users understand its basis.

  4. Timestamp the answer: Mark when it was given, so it's judged against what was known then.

  5. Avoid overclaiming: Don't assert more certainty than warranted, so answers don't age badly.

Your agent works, but it can't recognize when a request would be better served by a tool or resource outside its own capabilities.
How would you design the system to route to better external resources when appropriate?
General Medium

Answer:

  1. Recognize better-fit resources: Detect when another tool serves the need better, so you route well.

  2. Know your limits: Understand what the agent does poorly, so you defer appropriately.

  3. Recommend the right resource: Point users to the better option, so their need is served.

  4. Route seamlessly where integrated: Hand off directly if possible, so users don't have to switch manually.

  5. Still help where you can: Offer what value you provide, so routing isn't a pure deflection.

Your agent needs to handle a request where the user's emotional investment in a particular answer could bias how they receive the truth.
How would you design the system to deliver truth against emotional investment?
General Medium

Answer:

  1. Stay accurate: Give the honest answer despite emotional stakes, so users aren't misled.

  2. Acknowledge the investment: Recognize the user cares, so delivery is empathetic.

  3. Separate fact from preference: Distinguish what's true from what they hoped, so the truth is clear.

  4. Deliver with care: Frame hard truths considerately, so honesty isn't harsh.

  5. Offer constructive paths: Show what they can do given the truth, so the answer is useful.

Your agent works, but it can't tell when a small request is a probe testing whether it's worth trusting with a bigger one.
How would you design the system to earn trust through small interactions?
General Medium

Answer:

  1. Treat every request seriously: Handle small requests with full care, so trust is earned.

  2. Be consistently reliable: Deliver dependable quality on the small things, so bigger trust follows.

  3. Communicate limits honestly: Be upfront about what you can't do, so trust rests on accuracy.

  4. Demonstrate competence: Show quality on the probe, so users extend trust to larger tasks.

  5. Don't overpromise: Avoid overselling on small requests, so trust isn't undermined later.

Your agent needs to handle a request where the user wants a single number, but any honest answer is a range.
How would you design the system to communicate ranges when users want point estimates?
General Medium

Answer:

  1. Give a point with a range: Provide a best estimate alongside its uncertainty, so users get both.

  2. Explain the range: Show why the answer isn't a single number, so the range is credible.

  3. Offer the most likely value: Lead with the central estimate, so users have a usable figure.

  4. Convey confidence: Signal how wide the range is, so users gauge reliability.

  5. Avoid false precision: Don't collapse to a single number when it would mislead, so honesty holds.

Your agent is mature and successful, but you're concerned that optimizing purely for measured metrics is degrading things the metrics don't capture.
How would you design the process to protect unmeasured quality dimensions?
General Medium

Answer:

  1. Identify unmeasured dimensions: Name the quality aspects your metrics miss, so they're not ignored.

  2. Add measurement where possible: Bring important unmeasured dimensions into your metrics, so they're tracked.

  3. Use qualitative review: Regularly inspect quality beyond the numbers, so unmeasured degradation surfaces.

  4. Guard against metric myopia: Watch for metric gains that hurt unmeasured quality, so optimization doesn't distort.

  5. Balance measured and holistic judgment: Combine metrics with human assessment, so the whole is protected.

Your agent needs to handle a request where being maximally helpful now would create dependency that hurts the user long-term.
How would you design the system to balance immediate help against user long-term interest?
General Medium

Answer:

  1. Consider long-term impact: Weigh whether help now fosters unhealthy dependency, so you serve real interest.

  2. Build user capability: Help users learn to do it themselves where appropriate, so they grow more independent.

  3. Provide help while teaching: Solve the immediate need and explain how, so both are served.

  4. Avoid fostering dependency: Don't make users reliant when self-sufficiency serves them better, so you act in their interest.

  5. Respect user choice: Still help fully when that's what they want, so you don't paternalize.

Your agent works, but it can't tell when a user's question reveals a misunderstanding that will cause problems beyond the immediate answer.
How would you design the system to catch consequential misunderstandings?
General Medium

Answer:

  1. Detect underlying misunderstandings: Notice when a question reveals a deeper misconception, so it's caught.

  2. Assess the consequences: Judge whether the misunderstanding will cause broader problems, so you know to address it.

  3. Correct the misconception: Clarify the misunderstanding, not just answer the question, so the root issue is fixed.

  4. Explain the implications: Show why the misunderstanding matters, so the user grasps its significance.

  5. Answer the question too: Address what they asked, so the correction is additive, not a deflection.

Your agent needs to handle a request where the user wants confidence but the responsible answer is heavily caveated.
How would you design the system to balance user desire for confidence against necessary caveats?
General Medium

Answer:

  1. Lead with the substance: Give the actual answer clearly, so caveats don't bury it.

  2. Include only necessary caveats: Keep caveats to what genuinely matters, so you don't over-hedge.

  3. Distinguish firm from uncertain: Separate what you're confident about from what you're not, so users know which is which.

  4. Frame caveats usefully: Present them as decision-relevant, so they help rather than just disclaim.

  5. Don't fake confidence: Avoid stripping real uncertainty to please, so the answer stays honest.

Your agent works, but it treats a request identically whether it's the user's first attempt or their fifth after repeated failures.
How would you design the system to adapt when a user is repeatedly stuck?
General Medium

Answer:

  1. Recognize repeated attempts: Detect when a user keeps trying the same thing, so you adapt.

  2. Change the approach: Try a fundamentally different tack after failures, so you don't repeat what isn't working.

  3. Diagnose the sticking point: Identify why they're stuck, so you address the root cause.

  4. Simplify or break it down: Offer smaller steps when they're struggling, so progress becomes possible.

  5. Offer escalation: Suggest human help if the agent can't unstick them, so they aren't stranded.

Your agent needs to handle a request where committing to an answer now forecloses options the user might want later.
How would you design the system to preserve user optionality?
General Medium

Answer:

  1. Recognize foreclosing choices: Detect when an answer closes off future options, so it's flagged.

  2. Prefer reversible paths: Favor answers that keep options open, so flexibility is preserved.

  3. Surface what's foreclosed: Tell users what a choice rules out, so they decide informed.

  4. Offer staged commitment: Suggest low-commitment first steps, so users don't lock in prematurely.

  5. Let users weigh it: Present the optionality trade-off, so they choose deliberately.

Your agent works, but it can't recognize when the best help is to tell the user they're solving the wrong problem.
How would you design the system to recognize wrong-problem situations?
General Medium

Answer:

  1. Understand the real objective: Probe what the user ultimately wants, so you can spot a misaimed effort.

  2. Detect problem mismatches: Recognize when the stated problem won't achieve the goal, so it's caught.

  3. Raise it constructively: Point out the mismatch tactfully, so the user isn't dismissed.

  4. Suggest the right problem: Propose what they should be solving, so the redirection is actionable.

  5. Still help with what they asked: Address their request too, so you don't ignore their explicit ask.

Your agent needs to handle a request where the data supports several valid interpretations and it picks one without saying so.
How would you design the system to handle multiple valid interpretations of data?
General Medium

Answer:

  1. Detect multiple interpretations: Recognize when data supports several readings, so one isn't silently chosen.

  2. Present the alternatives: Lay out the valid interpretations, so users see the range.

  3. Explain the basis of each: Show what supports each reading, so users can judge.

  4. State your default: If you lead with one, say why, so the choice is transparent.

  5. Let context narrow it: Use the user's context to favor the fitting interpretation, so you're not needlessly ambiguous.

Your agent works, but it can't tell when a request is part of a larger goal it should help with proactively.
How would you design the system to recognize and support larger underlying goals?
General Medium

Answer:

  1. Infer the broader goal: Recognize what the request serves, so you can help with the bigger picture.

  2. Connect related requests: Link individual asks to the overarching goal, so you see the whole.

  3. Offer proactive help: Suggest next steps toward the larger goal, so users progress faster.

  4. Avoid overreach: Don't assume goals the user hasn't signaled, so proactivity isn't presumptuous.

  5. Confirm the direction: Verify the inferred goal before acting on it, so you stay aligned.

Your agent needs to handle a request where it must choose between a fast heuristic and a slow exact method, but it always uses the same one.
How would you design the system to choose between heuristic and exact methods per request?
General Medium

Answer:

  1. Assess accuracy needs: Judge whether the request needs exactness, so the right method is chosen.

  2. Use heuristics when tolerable: Apply fast approximations where precision isn't critical, so you save time.

  3. Reserve exact methods for critical cases: Use slow-but-precise methods where accuracy matters, so quality holds.

  4. Escalate on heuristic failure: Fall back to exact when the heuristic is insufficient, so mistakes are recoverable.

  5. Measure the trade-off: Track heuristic accuracy against exact cost, so the split stays optimal.

Your agent is mature, but you've realized that its metrics have all been optimized so much that they no longer leave room to distinguish good from great.
How would you design the process to keep metrics discriminating as performance improves.
General Medium

Answer:

  1. Raise the difficulty: Add harder cases as easy ones are mastered, so metrics stay discriminating.

  2. Retire saturated metrics: Drop measures everything now passes, so they don't waste attention.

  3. Introduce finer-grained metrics: Measure subtler quality dimensions, so good-versus-great is distinguishable.

  4. Focus on the failure frontier: Concentrate evals where the system still fails, so metrics track real gaps.

  5. Evolve metrics with performance: Update what you measure as the system improves, so metrics keep pace.