AI agent memory and AI agent state are not the same thing and confusing them can make recovery dangerous.

A conversation can survive a restart while the workflow still has no safe way to continue. A run can have perfect traces while the system has forgotten the input needed to reproduce it. And a failed agent can “recover” by repeating an email, database write or API action that already happened before the crash.

This guide separates four concepts that production systems need to model independently: conversation memory, workflow state, execution history and real-world side effects. It is based on failures we encountered while operating ApowerB, but it is written as a general production architecture rather than a duplicate of our original DEV.to incident report, “Amnesia and déjà vu”.

What is AI agent state management?

AI agent state management is the practice of persisting the information required to understand, continue, replay or safely terminate an agent run independently of the process currently executing it.

That state may include run ID, original input, current status, session reference, completed transitions, tool calls, approvals, external identifiers, idempotency keys, error information and retry metadata.

Memory vs state vs execution history vs business state

A chat transcript can help reconstruct intent. It is not a reliable ledger of external side effects.

Failure mode 1: amnesia after a process restart

In development, an in-memory session service is convenient. In production, the process is disposable.

During one internal ApowerB deployment audit, we observed roughly eight application restarts per day through ordinary deploys and cluster activity. When sessions lived in process memory, every restart could erase conversational context.

The fix is conceptually straightforward: move state that must survive the process into durable external storage.

In our case, short-term sessions were moved to Postgres, long-term memory to the application database with user/agent scoping, and artifacts to durable object storage with an appropriate fallback path.

Persisting memory does not make the run resumable

A database-backed conversation means the next worker can reconstruct what was said. It does not necessarily know which runtime step had completed, which tool call had crossed the network, whether an external system committed a write, or whether it is safe to perform the action again.

You therefore need a run record separate from conversation memory.

Failure mode 2: the run disappears but the side effect survives

Consider a customer-service agent:

1. read the request;

2. draft a response;

3. send the email;

4. update the CRM.

If the process dies after the email is sent but before the CRM update completes, blindly restarting from the beginning may send the email twice.

This is the production version of “déjà vu”: the software has lost confidence about where the run stopped, while the outside world remembers what already happened.

Why “resume from step 3” is harder for LLM agents

Traditional deterministic workflows often have stable program counters. Agentic execution may not.

A model can choose a different path after replay because context, provider behavior, model version or intermediate outputs differ. “Step 3” is not always a stable coordinate in a probabilistic plan.

Checkpointing should therefore be defined around validated state transitions and external effects, not simply around numbered model turns.

The safe-replay decision

Before replaying a failed run, ask three questions:

1. Do we have the original input?

2. Do we know which side-effecting tools were invoked?

3. Can we prove that repeating them is harmless or deduplicated?

If the answer to the third question is “unknown,” automatic replay should usually stop.

Tool classification: reads, reversible writes and irreversible writes

Idempotency keys need stable intent

Hashing generated tool arguments is not always enough. On replay, the model may phrase or structure equivalent arguments differently.

Where possible, derive idempotency from stable business identity: provider message ID, business operation ID, order/payment identifier, content checksum or an explicit request ID generated before reasoning begins.

The idempotency key should describe the business action, not the exact text the model happened to generate.

Unknown outcomes are the dangerous middle state

A network timeout does not tell you whether the remote service committed the action.

The safe pattern is:

1. send a stable operation ID where the API supports it;

2. persist that ID before dispatch;

3. if the response is ambiguous, query the system of record;

4. retry only when external state confirms the operation did not complete.

Checkpoints should follow committed transitions

A useful checkpoint says more than “the model finished a turn.” It captures a validated transition:

{
  "run_id": "run_123",
  "status": "waiting_for_approval",
  "completed_actions": ["lookup_customer"],
  "pending_action": "send_email",
  "external_refs": {"customer_id": "C-481"},
  "version": 7
}

{
  "run_id": "run_123",
  "status": "waiting_for_approval",
  "completed_actions": ["lookup_customer"],
  "pending_action": "send_email",
  "external_refs": {"customer_id": "C-481"},
  "version": 7
}

The exact schema varies by runtime. The principle is stable: checkpoints should be small enough to reason about and authoritative enough to resume from.

Concurrency and state versioning

Durability alone does not prevent two workers from updating the same run.

Use optimistic versioning, row locks, leases or queue ownership so only one executor can advance a run transition at a time. Otherwise two “recovery” workers can create the duplicate side effect you were trying to prevent.

Memory retention becomes a data-governance problem

Once memory moves from RAM to durable storage, it becomes retained user data.

• define tenant and user isolation;

• set retention periods;

• define deletion behavior;

• encrypt sensitive state;

• audit access;

• control what enters long-term memory;

• allow correction where required.

In our production work, we also had to bound database growth because session/event histories can expand indefinitely if retention is never designed.

Observability is not state management

Traces tell you what happened. State management lets the system continue correctly afterward.

You can have excellent traces and still be unable to retry because the original run input was never stored. Conversely, you can persist run input but have no explanation for why the run failed.

Production agents need both. See AI observability.

A practical run-state schema

• run: id, owner, input, status, timestamps, parent/workflow reference;

• tool invocation: tool, business key, arguments, status, external reference;

• transition: from state, to state, reason, version;

• approval: requested action, approver, decision, time;

• error: class, retryability, raw trace reference;

• checkpoint: recoverable state snapshot/version.

How ApowerB approached the problem

The ApowerB production work that inspired this article followed a conservative rule: persist sessions and run inputs, record tool activity, and avoid automatic replay when a previous tool call makes the outcome uncertain.

That rule intentionally refuses some replays that might have been harmless. It trades convenience for safety until tool-level idempotency semantics are strong enough to make a better decision.

The original incident narrative includes the code-oriented details and discussion that led to further hardening: read the DEV.to field note.

Inspect how the runtime persists and recovers agent work

ApowerB is open source, so its session, run and tool-execution paths can be reviewed directly.

State management checklist

How this fits into the broader production architecture

State management is one layer of a production agent runtime. It depends on infrastructure that can persist data and a deployment model that expects processes to disappear.

For the full stack, read Production-ready AI agents: architecture, runtime, orchestration and deployment. If you operate the runtime on Kubernetes, see How to deploy AI agents on Kubernetes.

The practical takeaway

The safest recovery question is not always “where did the agent stop?”

What do we know has already happened, and what can we prove is safe to do again?

That shift changes state management from a convenience feature into a production control. It also explains why durable memory alone is not enough: memory preserves context; safe recovery preserves consequences.