Code Agent Dissection (10): When the Agent Crashes, How Does It Recover? Where Is Conversation History Stored?

A deep dive into MyCodeAgent's persistence and recovery mechanism: TranscriptStore's append-only JSONL event stream, five event types, ResumeLoader rebuilding runtime state from events, UncertainAction's tool interruption handling, and SessionMemory for cross-run feed-forward memory. Understand why agents can't rely on memory for history, and the complete chain for deterministic crash recovery.

·9 min read·AI Engineering

Memory Is Unreliable

When an agent runs, all conversation history lives in HistoryManager's in-memory list. Once the process crashes — network timeout, OOM, Ctrl+C — all that history is gone. After restart, the user starts from scratch; dozens of steps of exploration results vanish entirely.

The tool call interruption problem is even trickier: if the agent crashed while the Edit tool was modifying a file, the file might be half-changed or unchanged, with nothing in the process logs. After restart, the agent can't see the result of this Edit call and doesn't know whether the file was changed.

MyCodeAgent solves both problems with Transcript: continuously writing all key facts to an append-only JSONL file, then rebuilding runtime state from the file after a crash.


Core Design: Event Stream Rather Than Snapshot

There are two common persistence approaches:

  • Snapshot: periodically serialize and save the entire state (like a database backup)
  • Event stream: append a record after each operation completes (like a database WAL)

Transcript uses event streams. Reason: snapshots require atomicity guarantees (can't write half), and after restart you need to choose which snapshot to restore from. Event streams only append, are naturally atomic (each line is independent), and replaying all events after restart restores state.

memory/transcripts/transcript-{session_id}.jsonl
 
{"event_id":"evt-abc","timestamp":"...","session_id":"s-123","run_id":"run-1","step":0,"event_type":"message","payload":{"role":"user","content":"Help me refactor the auth module"}}
{"event_id":"evt-def","timestamp":"...","session_id":"s-123","run_id":"run-1","step":1,"event_type":"state_transition","payload":{"reason":"model_returned_tool_calls",...}}
{"event_id":"evt-ghi","timestamp":"...","session_id":"s-123","run_id":"run-1","step":1,"event_type":"tool_lifecycle","payload":{"tool_name":"Read","tool_call_id":"call-xyz","status":"requested"}}
{"event_id":"evt-jkl","timestamp":"...","session_id":"s-123","run_id":"run-1","step":1,"event_type":"tool_lifecycle","payload":{"tool_name":"Read","tool_call_id":"call-xyz","status":"completed",
    "result":"..."}}
...
{"event_id":"evt-zzz","timestamp":"...","session_id":"s-123","run-1","step":28,"event_type":"terminal","payload":{"reason":"completed"}}

Five Event Types

Each event record captures one specific fact in the loop:

Event TypeRecordsWritten When
messageuser/assistant/tool message contentAfter each history_manager.append_*() call
state_transitionloop state transition reason + detailsAfter each _transition() call
tool_lifecycletool's requested/started/completed/failedAt each tool execution phase
checkpointcontext compression checkpoint (summary + split point)After compression triggers
terminalloop termination reasonWhen loop ends

Write path:

RuntimeRunner calls self._emit(event_type, payload, step=step)

RuntimeEventSink.emit(RuntimeEvent)
    ↓  [CompositeRuntimeEventSink]
    ├─ TraceRuntimeEventSink → trace_logger (JSONL trace file, for debugging)
    └─ TranscriptRuntimeEventSink → TranscriptRecorder → TranscriptStore.append_event()

Transcript and Trace are two independent sinks subscribing to the same event stream. Trace is detailed debug logs; Transcript is fact logs for recovery. They overlap in content but serve different purposes.


TranscriptStore: Append-Only Writing

# runtime/transcript.py
class TranscriptStore:
    def append_event(self, event: TranscriptEvent) -> TranscriptEvent:
        line = json.dumps(event.to_dict(), ensure_ascii=False)
        with self._lock:                      # file lock to prevent concurrent write corruption
            self._repair_trailing_record()    # repair incomplete trailing line (crash remnant)
            with self.path.open("a") as f:
                f.write(line)
                f.write("\n")
                f.flush()                     # flush to disk immediately, don't rely on OS buffer

_repair_trailing_record() checks the file's trailing content before each write: if there's an incomplete line (no newline, or JSON parse failure), it means the last write was interrupted midway — truncate that line. This guarantees every line in the file is complete, valid JSON.


How Events Flow into Transcript

Writing doesn't call TranscriptStore directly; there are two layers of abstraction in between:

loop's self._emit("message", {...}, step=step)

RuntimeRunner._emit_runtime_event(run_id, step, event_type, payload)

host.runtime_event_sink.emit(RuntimeEvent)
    ↓  CompositeRuntimeEventSink simultaneously forwards to two sinks
    ├─ TraceRuntimeEventSink.emit()      → extensions/tracing/logger.py (debug trace)
    └─ TranscriptRuntimeEventSink.emit() → TranscriptRecorder.record_*()

                                           TranscriptStore.append_*()

                                           JSONL file appends one line

TranscriptRecorder is a Facade for TranscriptStore: it encapsulates high-level operations — 'write message', 'write state_transition', 'write tool_lifecycle' — into record_message(), record_state_transition(), record_tool_lifecycle(), hiding the underlying JSON serialization details. It also holds an on_recorded callback that calls SessionMemoryManager.ingest_event() after each write to incrementally update Session Memory.


Recovery: Rebuilding Runtime State from the Event Stream

After user restart, calling agent.resume_transcript() or using the CLI --resume flag leads to ResumeLoader.load_session():

# runtime/transcript.py — ResumeLoader._load_events() (simplified)
def _load_events(self, events, *, run_id):
    history_messages = []    # rebuild history message list
    checkpoint = None        # last compression checkpoint
    terminal = None          # terminal event
    tool_events = {}         # complete lifecycle of each tool call
 
    for event in events:
        if event.event_type is MESSAGE:
            history_messages.append({role, content, metadata})
 
        elif event.event_type is CHECKPOINT:
            checkpoint = event.payload   # keep the last checkpoint
 
        elif event.event_type is TERMINAL:
            terminal = event.payload
 
        elif event.event_type is TOOL_LIFECYCLE:
            # aggregate all states for the same tool_call_id
            tool_events[(run_id, tool_call_id)]["statuses"].append(status)

Then analyze tool_events and classify each tool call:

completed   → already done, no replay needed
failed      → already failed, no replay needed
requested but not started → not executed, pending (can be replanned)
started but no completed/failed → uncertain (interrupted, state unknown)

UncertainAction is the most subtle concept in recovery: the tool started executing, but the agent crashed before the result was written back. The result could be success, failure, or any intermediate state.

uncertain_actions.append(UncertainAction(
    tool_name=tool_name,
    tool_call_id=tool_call_id,
    replay_allowed=tool_name not in {"Edit", "Bash", "Task"},
    # Read/Grep/Glob: idempotent, can be replayed
    # Edit/Bash/Task: have side effects, can't be blindly replayed; need user judgment
))

During CLI recovery, uncertain actions are printed so the user knows 'these tools may or may not have executed — please verify yourself.'


apply_to_host After Reconstruction Is Complete

ResumeState.apply_to_host(host) injects rebuilt state into the running agent:

# runtime/transcript.py — ResumeState.apply_to_host()
def apply_to_host(self, host):
    # 1. reset context engine, clear compression checkpoints
    host.context_engine.reset()
 
    # 2. write rebuilt history messages into HistoryManager
    host.history_manager.load_messages(self.history_messages)
 
    # 3. if there's a compression checkpoint, reactivate it
    #    (ProjectionBuilder will fold old history when it reads this)
    if self.checkpoint:
        host.context_engine.compact_store.set_active(CompactCheckpoint(...))
 
    # 4. restore Read tool's optimistic lock cache (mtime snapshots to avoid Edit conflict false positives)
    if read_cache := self.runtime_state.get("read_cache"):
        host.tool_registry.import_read_cache(read_cache)

After recovery, the agent behaves as if it never crashed — history complete, compression state restored, optimistic lock cache valid.


Session Memory: Feed-Forward Memory Across Runs

Transcript stores the complete fact stream; Session Memory is a bounded summary derived from the fact stream.

SessionMemoryDeriver scans all transcript events and extracts high-level information:

@dataclass(frozen=True)
class SessionMemory:
    current_goal: SessionMemoryItem | None  # user's most recent goal (latest user message)
    completed_work: tuple[...]              # what was completed (final-type assistant messages)
    key_decisions: tuple[...]               # key decisions (compression checkpoints, important state transitions)
    failed_attempts: tuple[...]             # what failed (model_recovery_failed, etc.)
    todo_items: tuple[...]                  # incomplete TodoWrite items
    verification_status: tuple[...]         # verification status (completion gate info)

Session Memory is injected as a system message in build_model_view(), placed after the system prompt and before history messages:

[system prompt]
[Session Memory]  ← "Previously you completed X, failed at Y, current goal is Z"
[history messages]

Why Session Memory instead of reading transcript directly?

Transcript might have thousands of event lines; putting all of them in the model view would exceed the token budget. Session Memory is a bounded high-level summary, controlled to within a few hundred lines, letting the model have cross-run context without reading the full event stream.

Session Memory is maintained incrementally: TranscriptRecorder calls SessionMemoryManager.ingest_event() with every event written; SessionMemoryDeriver.update() incrementally appends new events without needing a full rebuild each time.


Actual File Structure

memory/
├── transcripts/
│   ├── transcript-session-abc123.jsonl   ← main agent session
│   └── transcript-subagent-child-xyz.jsonl ← sub-agent session (independent per Task call)
└── traces/
    ├── session-abc123.jsonl              ← debug trace (detailed)
    └── session-abc123.html               ← visual report (optional)

Transcript and Trace files sit side by side but serve different purposes:

  • Transcript: source of truth for crash recovery; only stores key facts
  • Trace: for debug analysis; records all details (token usage, per-step timing, raw model output)

Design Highlights

Event stream rather than snapshot: Each event is persisted on write; doesn't wait for agent completion; recovery starts from the last complete event after a crash.

Uncertain actions are explicitly marked: Tool interruptions aren't silent failures — they're explicitly marked as uncertain, leaving it to the user to decide whether to replay, rather than having the agent guess automatically.

Recovery is deterministic: Reading from the same transcript produces the same history_messages and loop_state every time; doesn't depend on any random state.

Session Memory maintained incrementally: Not rebuilt in full on recovery; updated incrementally after each event write, spreading reconstruction cost across each write operation.


Summary

ComponentResponsibility
TranscriptStoreAppend-only JSONL writing with file lock and trailing repair
TranscriptRecorderFacade layer, encapsulates high-level write operations, callbacks to update SessionMemory
TranscriptRuntimeEventSinkRoutes RuntimeEvents to TranscriptRecorder
ResumeLoaderRebuilds history/checkpoint/tool_states/uncertain_actions from event stream
ResumeState.apply_to_hostInjects rebuilt state into agent, completing recovery
SessionMemoryDeriverDerives bounded working memory from event stream for feed-forward injection into model view
UncertainActionExplicitly marks interrupted tool calls, protecting users from blindly replaying side effects

About the Source Code

All analysis in this series is based on the open-source project MyCodeAgent.

The source code includes inline comments aligned with the series' explanation order at key locations — you can follow along while reading, or clone it, run it, modify it, and extend it to build your own agent.

git clone https://github.com/chendongqi/MyCodeAgent
cd MyCodeAgent
cp .env.example .env   # fill in your LLM API key
uv sync
uv run python main.py

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage