H
pending
A Checkpoint Is Not Durable Execution — Here's the Difference
Grounded / Real
Inflated / Uruttu
Original Content
AI Engineer Interview Question:
"Your agent crashes at step 200 of 240. What happens next?"
Most candidates say "we retry the run" and stop there, which is the wrong half of the answer. What the interviewer is actually testing:
Retrying is the easy part. The cost of that retry is the real question.
200 steps is 40 minutes of tokens you already paid for, plus tool calls that already changed things.
If the run state lived in process memory, a pod eviction or a routine deploy takes all of it. You start again at step 1, pay for every step twice, and every write fires a second time.
A journaled run goes forward from step 201 instead.
3 things make that work.
1. Checkpoint the run state. Write each finished step to a durable store before the next one starts. The unit is the step boundary. LangGraph's MemorySaver is in-process, so it dies with the process.
2. Key every mutation. Mint the idempotency key before the call and store it with the step. The dedupe has to live at the provider, since your retry logic died too.
3. Replay reads, skip sends. Repeatable calls run again. Anything that spends money or reaches a human loads its recorded result. Model output counts as a side effect.
One line worth saying out loud in that interview: a checkpoint is not durable execution. Something outside the process still has to notice the death and restart the run.
What is your agent's run state sitting in right now, Postgres or the process that is about to get redeployed?
Connect/Follow for more such deep dives.
"Your agent crashes at step 200 of 240. What happens next?"
Most candidates say "we retry the run" and stop there, which is the wrong half of the answer. What the interviewer is actually testing:
Retrying is the easy part. The cost of that retry is the real question.
200 steps is 40 minutes of tokens you already paid for, plus tool calls that already changed things.
If the run state lived in process memory, a pod eviction or a routine deploy takes all of it. You start again at step 1, pay for every step twice, and every write fires a second time.
A journaled run goes forward from step 201 instead.
3 things make that work.
1. Checkpoint the run state. Write each finished step to a durable store before the next one starts. The unit is the step boundary. LangGraph's MemorySaver is in-process, so it dies with the process.
2. Key every mutation. Mint the idempotency key before the call and store it with the step. The dedupe has to live at the provider, since your retry logic died too.
3. Replay reads, skip sends. Repeatable calls run again. Anything that spends money or reaches a human loads its recorded result. Model output counts as a side effect.
One line worth saying out loud in that interview: a checkpoint is not durable execution. Something outside the process still has to notice the death and restart the run.
What is your agent's run state sitting in right now, Postgres or the process that is about to get redeployed?
Connect/Follow for more such deep dives.
Validated Content
Technically accurate throughout — this is a well-grounded, correctly reasoned explanation of a real architectural problem, not just interview-prep fluff.
- "LangGraph's MemorySaver is in-process, so it dies with the process" — confirmed exactly. LangGraph's MemorySaver (also called InMemorySaver) stores checkpoints in RAM, is intended for development, notebooks, and testing rather than production, and any process restart, pod eviction, or deployment wipes out every in-flight run using it. This is precisely and correctly described.
- "A journaled run goes forward from step 201 instead" — accurate framing of what proper checkpointing enables: LangGraph's own architecture saves state at every superstep specifically so a graph can resume from the last completed step rather than restarting from zero.
- "Checkpoint the run state at the step boundary, before the next one starts" — accurate description of how LangGraph checkpointing actually works; it writes a checkpoint after each node/step execution, not just at the end of the run.
- "Production needs a durable backend — Postgres, not in-process memory" — accurate. This matches how LangGraph's own documentation and third-party guides consistently frame the distinction: MemorySaver for dev/testing, PostgresSaver (or similar durable backends) for any multi-replica or production deployment where state needs to survive restarts, deploys, and crashes.
- "Idempotency keys, dedupe living at the provider, replay reads vs. skip sends" — this is sound general distributed-systems reasoning applied correctly to the agent context. It's not a LangGraph-specific claim, but a well-established pattern (exactly-once-effect via idempotency keys checked at the provider boundary) that's correctly applied here — separating safe-to-repeat reads from side-effecting actions (payments, sends, human-facing outputs) is standard practice in reliable systems design, and treating model output as a side effect when it triggers downstream actions is a subtle but correct point.
- "A checkpoint is not durable execution — something outside the process has to notice the death and restart the run" — this is an accurate and genuinely important nuance. Checkpointing alone only preserves state; it doesn't provide the orchestration/supervision layer (a scheduler, workflow engine, or watchdog) needed to actually detect failure and resume the run. This distinction is correct and often missed.