Clear
Your agent works, but its quality quietly depends on a specific model version, and you have no plan for when that version is deprecated.
How would you design the system to handle model deprecation gracefully?
General Medium

Answer:

  1. Track deprecation timelines: Monitor provider announcements, so you're not caught off guard.

  2. Abstract the model: Route through a gateway, so swapping is low-friction.

  3. Test replacements early: Evaluate successor models ahead of deprecation, so you're ready to switch.

  4. Re-tune for the new model: Adjust prompts for successor behavior, so quality is preserved.

  5. Migrate before forced: Switch proactively, so you're not scrambling at the deadline.

Your agent needs to make decisions that depend on the current state of an external system, but that state changes between when it reads and when it acts.
How would you design the system to handle state that changes between read and action?
General Medium

Answer:

  1. Re-check before acting: Verify state right before the action, so you act on current reality.

  2. Use optimistic locking: Detect if state changed since read, so stale-state actions are caught.

  3. Make actions conditional: Execute only if state still holds, so changes don't cause wrong actions.

  4. Handle conflicts: Retry or re-evaluate on state change, so you adapt rather than fail.

  5. Minimize read-to-action gap: Shorten the window, so less can change in between.

Your agent needs to handle a growing catalog of use cases, but adding each new one degrades performance on existing ones.
How would you design the system to add use cases without regressing existing ones?
General Medium

Answer:

  1. Isolate use cases: Keep them modular, so adding one doesn't affect others.

  2. Regression-test existing cases: Verify old use cases still work on every change, so regressions are caught.

  3. Route by use case: Direct requests to appropriate handling, so use cases don't interfere.

  4. Avoid shared-prompt bloat: Don't cram all cases into one prompt, so they don't dilute each other.

  5. Monitor per use case: Track quality by case, so regressions surface.

Your agent produces answers that are technically complete but miss the emotional context of a user who's clearly stressed or upset.
How would you design the system to respond appropriately to emotional context?
General Medium

Answer:

  1. Detect emotional signals: Recognize stress or upset in the user, so responses can adapt.

  2. Adjust tone: Respond with appropriate empathy, so the answer fits the emotional context.

  3. Address the underlying need: Attend to what's driving the emotion, so you help beyond the literal request.

  4. Avoid tone-deafness: Don't respond mechanically to distress, so the interaction feels human.

  5. Escalate genuine distress: Route serious cases to human support, so real needs get proper care.

Your agent needs to run reliably on infrastructure you don't fully control, where resources can be reclaimed at any moment.
How would you design the system to tolerate unreliable, preemptible infrastructure?
General Medium

Answer:

  1. Checkpoint frequently: Save progress often, so preemption doesn't lose much work.

  2. Make work resumable: Design tasks to continue after interruption, so preemption isn't fatal.

  3. Handle preemption signals: React to reclaim warnings, so you save state before shutdown.

  4. Distribute redundantly: Spread work across instances, so losing one doesn't halt everything.

  5. Retry preempted work: Automatically reschedule interrupted tasks, so they complete eventually.

Your agent's answers are good, but it never admits when a question is outside what it was designed for, wandering into unreliable territory.
How would you design the system to stay within its intended scope?
General Medium

Answer:

  1. Define scope explicitly: Specify what the agent is designed to handle, so boundaries are clear.

  2. Detect out-of-scope requests: Recognize when a question exceeds scope, so drift is caught.

  3. Decline gracefully: Redirect out-of-scope requests, so the agent doesn't wander into unreliability.

  4. Route to alternatives: Point users to better-suited resources, so out-of-scope needs are still served.

  5. Monitor scope creep: Track out-of-scope requests, so recurring needs inform expansion decisions.

Your agent needs to serve responses that comply with accessibility requirements, but its output format ignores them.
How would you design the system to produce accessible output?
General Medium

Answer:

  1. Structure for screen readers: Use proper semantic structure, so assistive tech can parse output.

  2. Provide text alternatives: Describe non-text content, so it's accessible.

  3. Use clear language: Favor plain, readable phrasing, so content is broadly understandable.

  4. Follow accessibility standards: Conform to guidelines like WCAG, so requirements are met.

  5. Test with assistive tools: Validate output with screen readers, so accessibility gaps surface.

Your agent's behavior needs to change based on business rules that non-engineers own, but every rule change requires a developer.
How would you design the system to let non-engineers manage business rules safely?
General Medium

Answer:

  1. Externalize rules as config: Store business rules outside code, so non-engineers can change them.

  2. Provide a safe interface: Give a controlled way to edit rules, so changes don't require code.

  3. Validate rule changes: Check edits against constraints, so bad rules are rejected.

  4. Test before applying: Run rules through evals, so changes are verified.

  5. Enable rollback: Let rule changes be reverted, so mistakes are recoverable.

Your agent needs to work with numerical data where precision matters, but the model rounds or approximates values in ways that corrupt results.
How would you design the system to preserve numerical precision?
General Medium

Answer:

  1. Keep numbers out of generation: Handle precise values in code, not the model, so they aren't approximated.

  2. Pass by reference: Use identifiers for values and compute externally, so the model doesn't retype numbers.

  3. Validate against source: Check output numbers match inputs, so corruption is caught.

  4. Preserve significant figures: Enforce precision requirements, so rounding doesn't lose accuracy.

  5. Compute with tools: Route calculations to exact tools, so precision is guaranteed.

Your agent needs to operate under a fixed monthly budget, but usage patterns mean it sometimes exhausts the budget before month-end.
How would you design the system to manage a fixed budget across a time period?
General Medium

Answer:

  1. Track spend against pace: Monitor burn rate versus time elapsed, so you see if you're on track.

  2. Set pacing targets: Allocate budget across the period, so spend is distributed.

  3. Throttle as budget depletes: Slow non-essential usage when running low, so you don't exhaust early.

  4. Prioritize under constraint: Reserve remaining budget for critical work, so essentials are protected.

  5. Alert on overpace: Flag when spend outpaces the plan, so you react before running out.

  6. Degrade gracefully near limits: Shift to cheaper handling as budget tightens, so service continues.

Your agent works well, but when it makes a mistake, users have no way to give feedback that actually improves it.
How would you design the system to turn user feedback into real improvement?
General Medium

Answer:

  1. Capture actionable feedback: Collect specific signals on what went wrong, so feedback is usable.

  2. Categorize feedback: Group issues by type, so patterns emerge.

  3. Feed into evals: Convert feedback into test cases, so fixes are validated and regressions prevented.

  4. Prioritize by frequency: Address common complaints first, so effort maximizes impact.

  5. Close the loop with users: Show that feedback led to change, so users keep providing it.

Your agent needs to handle sensitive user requests where even the metadata about what they asked is confidential.
How would you design the system to protect not just data but request metadata?
General Medium

Answer:

  1. Minimize metadata capture: Log only what's necessary, so sensitive metadata isn't accumulated.

  2. Redact in logs and traces: Mask sensitive request details, so metadata doesn't leak through observability.

  3. Encrypt metadata: Protect it at rest and in transit, so it's not exposed.

  4. Control access: Restrict who can see request metadata, so exposure is limited.

  5. Set retention limits: Expire metadata on schedule, so it doesn't persist unnecessarily.

Your agent's retrieval index is expensive to maintain, and much of it is content that's never actually retrieved.
How would you design the system to optimize index contents by actual usage?
General Medium

Answer:

  1. Track retrieval frequency: Measure what gets retrieved, so you know what's used.

  2. Prune dead content: Remove never-retrieved documents, so the index stays lean.

  3. Tier storage: Keep hot content fast and cold content cheap, so cost matches usage.

  4. Prioritize valuable content: Ensure high-value documents are well-indexed, so coverage serves real needs.

  5. Monitor coverage gaps: Watch for retrieval failures, so pruning doesn't remove needed content.

Your agent handles multi-part requests, but it often answers the first part thoroughly and neglects the rest.
How would you design the system to fully address multi-part requests?
General Medium

Answer:

  1. Decompose the request: Break it into distinct parts, so each is tracked.

  2. Address each part: Ensure every component gets answered, so none is neglected.

  3. Verify completeness: Check all parts are covered before responding, so gaps are caught.

  4. Structure the response: Organize by part, so coverage is clear to the user.

  5. Flag what's unanswerable: Note parts you can't address, so silence isn't mistaken for an answer.

Your agent needs to reason about cause and effect, but it confuses correlation with causation and draws wrong conclusions.
How would you design the system to handle causal reasoning more reliably?
General Medium

Answer:

  1. Distinguish correlation from causation: Prompt the model to separate them, so it doesn't conflate the two.

  2. Ground in evidence: Base causal claims on actual support, so conclusions aren't speculative.

  3. Surface assumptions: Make causal reasoning explicit, so flawed logic is visible.

  4. Express uncertainty: Flag when causation is inferred, not established, so users don't over-rely.

  5. Verify causal claims: Check reasoning against known facts, so wrong conclusions are caught.

Your agent needs to work with a legacy system that only accepts a rigid, brittle input format, and small deviations cause failures.
How would you design the system to reliably produce output for a rigid legacy format?
General Medium

Answer:

  1. Enforce the exact format: Constrain output to the legacy spec, so deviations don't occur.

  2. Validate strictly: Check output against the format before sending, so bad input is caught.

  3. Use templates: Generate within fixed scaffolding, so structure is guaranteed.

  4. Transform post-generation: Convert model output into the required format in code, so the model doesn't need perfect formatting.

  5. Handle rejections: Retry with correction when the legacy system rejects, so failures are recovered.

Your agent occasionally produces output that's correct but presented so confidently that users don't double-check when they should.
How would you design the system to calibrate expressed confidence to actual reliability?
General Medium

Answer:

  1. Estimate real confidence: Derive a genuine reliability signal, so expressed confidence has a basis.

  2. Match tone to certainty: Express uncertainty when confidence is low, so tone doesn't oversell.

  3. Flag double-check-worthy answers: Signal when verification is warranted, so users don't over-trust.

  4. Ground confidence in evidence: Tie certainty to support, so it reflects reality, not fluency.

  5. Validate calibration: Check that stated confidence matches accuracy, so calibration stays honest.

Your agent needs to make trade-offs a user would make differently, but it applies its own preferences instead of learning theirs.
How would you design the system to align trade-offs with individual user preferences?
General Medium

Answer:

  1. Learn user preferences: Capture how each user weighs trade-offs, so decisions reflect their priorities.

  2. Ask when it matters: Clarify preferences on significant trade-offs, so you don't assume.

  3. Apply stored preferences: Use known preferences where relevant, so past input shapes decisions.

  4. Explain trade-offs made: Surface the choices, so users can correct misaligned ones.

  5. Update from feedback: Refine preferences as users react, so alignment improves.

Your agent needs to handle requests that reference files or context the user assumes it has, but it doesn't actually have access.
How would you design the system to handle references to unavailable context?
General Medium

Answer:

  1. Detect missing references: Recognize when a request assumes context you lack, so gaps are identified.

  2. Ask for what's needed: Request the missing file or information, so you can actually help.

  3. Avoid guessing: Don't fabricate the missing context, so answers aren't built on invention.

  4. Search available sources: Check if the reference exists in accessible data, so you find it if possible.

  5. Communicate clearly: Tell the user what you're missing, so they can provide it.

Your agent is one of several that share a knowledge base, and one agent's bad writes are corrupting data the others rely on.
How would you design the system to protect shared knowledge from bad writes?
General Medium

Answer:

  1. Validate before writing: Check data quality on write, so bad data doesn't enter the shared store.

  2. Control write access: Restrict which agents can write what, so corruption sources are limited.

  3. Version writes: Track changes with history, so bad writes can be reverted.

  4. Isolate write paths: Separate write concerns, so one agent's errors don't cascade.

  5. Detect corruption: Monitor data integrity, so bad writes are caught early.

  6. Enable rollback: Restore prior good state, so corruption is recoverable.