From Ephemeral Loops to Crash-Resilient Runtimes: The Architecture Philosophy of Pi AgentHarness v2

Source Architecture Design: Pi AgentHarness v2 Design Document

Core Module: packages/agent/src/harness · Implementation: earendil-works/pi

Design Goal: Provide a production-grade Agent execution engine featuring complete crash recovery (durability), parallel multi-lane execution, deterministic step-level testing, and strict protection for LLM KV caching.

In most open-source demos and introductory tutorials, the core runtime of an AI agent is commonly reduced to a simple while loop: receive user input, construct context, prompt the language model, parse tool calls, execute them, append the results back to an in-memory array, and continue. For single-turn prototypes or quick CLI scripts, this approach is clean and compelling. But the moment an agent runtime is deployed into real-world production environments—grappling with long-running tasks, sudden process terminations, multi-threaded conversations, flaky network connections, and compounding token costs—this naive loop quickly collapses under its architectural fragility.

When a host process loses power in the middle of writing a file via a tool call, how does the system know where it left off upon reboot? Does it lose the user’s mid-flight steering corrections, or does it blindly re-execute non-idempotent external tools? If multiple Slack threads or background subagents need to operate concurrently over the same session history, how does the state machine prevent dirty reads and writes while enforcing a strict single-writer boundary? Furthermore, when a user injects feedback while the model is actively generating, does the runtime forcibly jam the new message into the ongoing turn—thereby invalidating the provider's KV cache—or does it enforce strict monotonic append-only discipline?

The Durable AgentHarness v2 design, engineered for the Pi project's core agent runtime, offers one of the most rigorous, database-inspired blueprints for autonomous agent infrastructure to date. Rather than piling on leaky framework abstractions, it returns to the first principles of distributed systems and journaled storage, establishing a minimal set of axioms and state machines that comprehensively eliminate ghost states and concurrency races.

flowchart TD
    App[Application / UI] -->|prompt, steer, abort| Harness[AgentHarness Runtime]
    Harness -->|snapshots + ordered events| App
    Harness -->|interception hooks| Ext[Extension System]
    Harness --> Lanes[Parallel Lanes: main, thread-1, ...<br/>one operation per lane, parallel across lanes]
    Lanes --> Loop[Step Primitives<br/>provider request / tool batch]
    Loop --> Provider[LLM Provider / Deferred Handle]
    Loop --> Tools[External Tools]
    Harness --> Session[Session Core State<br/>shared tree · lane operation logs · global facts]
    Session --> Storage[(Storage: SQLite / JSONL / Memory)]

---

Write Intent Before Effect: Crash Resilience Without Distributed Transactions

In traditional backend architectures, multi-step atomicity relies on two-phase commits (2PC) or database transactions. However, in the world of autonomous agents, tool invocations involve irreversible physical side effects (such as making external HTTP calls, writing local files, or executing shell commands), while streaming LLM generation constitutes asynchronous, stateful network I/O that cannot be wrapped in a database rollback. Attempting to force heavy distributed transactions onto an agent runtime is a dead end.

AgentHarness v2 establishes a razor-sharp Durability Rule:

Before initiating any external effect, write a durable intent record specifying what is about to happen and the pre-allocated entry IDs it will produce; after the effect completes, append the actual result as a tree entry using exactly those provisioned IDs.

    prompt("fix the bug")
H   before_run                          (Hook intercepts; can inject messages or override system prompt)
R   operation_started                   (Intent record: logs operation kind and pre-allocated entry IDs)
E   user message                        (User message entry appended with the provisioned ID)
R   step_attempt                        (Intent record: assistant step, attempt 1)
E   assistant message [tool call]       (Model returns tool call, committed as entry)
H   before_tool                         (Hook validates tool parameters)
R   tool_started                        (Intent record: logs tool name, effective args, replay safety, and resultEntryId)
E   tool result                         (Tool settles, committed as result entry using provisioned ID)
R   operation_finished                  (Outcome record: completed)

Under this protocol, the storage layer requires zero multi-record atomicity; every record in the lane's operation log and every entry appended to the tree is durable independently. If the process crashes at any arbitrary point between any two lines, the state left on disk can only be one of two possibilities: either the intent was never committed (the operation was never accepted, and the caller is cleanly rejected), or the intent was recorded on disk while its resulting entry remains unfulfilled.

This turns crash recovery into a pure reduction function. When a new process boots and restores a session, it does not need to guess at transient in-memory state; it merely performs a deterministic fold over the lane’s operation log against the actual conversation tree:

• If a tool_started record exists but no corresponding tool_result entry is present with that provisioned ID, the recovery engine immediately recognizes that the crash occurred while the tool was executing. The engine then inspects the tool's declared replay safety: if the tool declared replay: "safe", it safely re-executes the tool with the persisted effective arguments; if declared replay: "never", it synthesizes an "interrupted" result entry, refusing to risk double-executing non-idempotent mutations.
• If a step_attempt record exists without its assistant message entry, the system identifies an unfulfilled model attempt, automatically scheduling the next attempt while preserving the durable attempt counter across restarts to prevent infinite reboot loops.

Crucially, billing and token consumption records (usage records) are committed immediately when each provider request settles, prior to any classification, discard, or retry logic. Even if a model response is discarded due to context overflow, or if the system crashes before committing an assistant entry, the token spend ledger remains intact. The system completely decouples the durability of cost accounting from the durability of conversational results.

---

Trees and Lanes: A Concurrency Model Inspired by Git Worktrees

Most agent frameworks model conversation histories as flat arrays. The moment branching, rollback, subagents, or multi-threaded interactions are introduced, flat lists rapidly break down. While some systems adopt tree structures to handle branches, they immediately run into a fundamental coordination problem: who owns the tree's active focus? If a background subagent needs to branch off and perform exploratory research while the user continues chatting on the main trunk, how are these concurrent workflows isolated?

AgentHarness v2 introduces a clean separation between the Shared Passive Tree and Active Independent Lanes.

Conversation Tree (Append-Only, Passive Data)     Active Lanes
a ── b ── c ── d                                  main            → d   (Lane Op Log: op_started...)
      └── e ── f                                  slack:171943…   → f   (Lane Op Log: op_started...)

Global Facts: name = "Refactor auth", label(b) = "checkpoint-1"

In this architecture:

1. The conversation tree is pure, passive, and append-only. Composed of immutable entries linked by parentId pointers, it strictly grows and never modifies or deletes existing nodes. The tree belongs to no single lane and contains zero execution or orchestration pointers.
2. Lanes are active execution positions. A lane consists of a named pointer to an entry (its leaf) paired with an isolated, serialized operation log. Every session includes a default main lane, and applications can spin up arbitrary named lanes mapped to external identities (such as a Slack thread ID, an email chain, or a subagent task).

This model is conceptually identical to Git: the conversation tree acts as Git's underlying commit graph, while lanes operate as independent branches checked out inside dedicated worktrees. Spawning a new lane duplicates zero history; multiple lanes can anchor on the same historical node simultaneously. As each lane runs prompts and executes tool batches, it records private step intents in its own operation log, appends child entries to the tree, and advances its leaf pointer independently.

Lanes run concurrently inside a single harness instance with zero lock contention and zero inter-lane coordination. Physical writes from all lanes are interleaved seamlessly into storage via a monotonic sequence number. If one lane faults or suspends on an interrupted operation, all other lanes continue executing unhindered.

---

KV Cache Discipline: Checkpoint Serialization and Single-Lane Mutation

In production agent operations, provider prompt caching (KV caching) is the primary determinant of both latency and inference expenditure. For modern LLMs, any alteration to the context prefix breaks subsequent KV caches from that modification point forward, forcing the provider to recompute the entire prompt at full latency and token cost.

A classic race condition occurs mid-turn: suppose an agent is executing a 30-second refactoring tool batch, and the user submits a steering command ("Also add unit tests"). If the runtime immediately forces this user message into the active conversation tail, the context order presented to the model upon tool completion becomes [user prompt, user steering, tool results], completely destroying the cache prefix established in the preceding turn.

AgentHarness v2 enforces a strict Append-Only Context invariant:

Across consecutive provider requests on a lane, context must grow exclusively at the tail. No insertion before the previous request's tail is permitted.

[User Prompt]
      ↓
[Turn Start] ── Run Assistant Step ──→ (User submits Steer command) ──→ Committed to queue_enqueued (Not in tree)
      ↓                                                                           │
[Execute Tool Batch] ── Finish and append tool_result Entry                       │
      ↓                                                                           │
[Arrive at Checkpoint] ←──────────────────────────────────────────────────────────┘
      ├─ 1. Apply queued Deferred Writes
      ├─ 2. Consume queued Steering / Follow-Up Messages
      └─ 3. Evaluate context pressure; trigger Compaction if needed
      ↓
[Next Turn] ── Model request sent with perfect append-only tail; KV Cache fully preserved

To harmonize instant user input acceptance with strict KV cache protection, the harness introduces Queued Intent and Checkpoint Serialization:

• While a model step or tool execution is in flight, user steering inputs (steer), follow-up tasks (followUp), and configuration mutations are durably committed to the lane's operation log as queue_enqueued or write_deferred records, resolving immediately to the caller.
• These inputs are not immediately inserted into the conversation tree as message entries, ensuring they do not collide with or invalidate the ongoing turn.
• Only when an entire turn concludes and reaches a Checkpoint does the runtime drain and materialize pending deferred writes and steering messages, cleanly appending them to the leaf of the tree.

This is powered by the Lane Mutation Line—a localized Promise FIFO queue per lane. All state-dependent decisions (evaluating whether a run can finish, consuming queues, processing cancellations) execute inside microsecond in-memory validation jobs paired with a single durable write. Heavy I/O, network streams, and tool executions run strictly outside the mutation line. By serializing lightweight state transitions and placing heavy I/O outside the line, the architecture mathematically eliminates check-then-act race conditions.

---

The Deterministic Effects Boundary: From Fragile Black Box to Step-by-Step State Machine

Testing autonomous agents has traditionally been notoriously difficult. Due to non-deterministic LLM generations, uncontrollable tool network calls, and concurrency interleavings, most automated tests either rely on coarse end-to-end black-box checks or brittle timeouts and flaky mocks. Engineers could never reliably simulate edge cases such as "the process crashing at millisecond 200 of the third tool execution while the user simultaneously clicked cancel."

AgentHarness v2 resolves this at the architectural foundation via The Effects Boundary and Dual Drive Modes.

flowchart LR
    subgraph Procedure[Agent Procedure (Pure Orchestration Logic)]
        Direction[State Progression / Intent Generation]
    end

    Direction -->|all writes, provider calls, tools, hooks| FX[Effects Gateway: fx]

    subgraph DualDrive[Dual Drive Modes]
        Auto[drive: 'automatic'<br/>Production: direct passthrough to I/O]
        Manual[drive: 'manual'<br/>Testing: GatedEffects Action Interceptor]
    end

    FX --> DualDrive
    Manual --> ActionQueue[Action Queue: peekAction / executeAction]

Inside the harness, procedural loops are strictly constrained to interact with external realities exclusively through an injected Effects handle (fx), completely barring direct access to databases, LLM SDKs, or raw tool implementations. Every side effect—writing records, advancing pointers, streaming models, running tools, firing hooks, or sleeping—is funneled through explicit methods on Effects:

1. In production, the system runs with drive: "automatic", where fx acts as a zero-overhead passthrough to underlying I/O.
2. In automated test suites, the system switches to drive: "manual", wrapping fx inside a GatedEffects gate. Every pending effect automatically parks before execution, exposing a typed descriptor (ActionInfo).

Test suites can call peekAction() to inspect the next queued action, use executeAction() to step a single effect, or invoke harness.close() between arbitrary actions to simulate abrupt power failures. Reopening the backend and calling resume() allows deterministic verification of recovery behavior across every conceivable crash site (X1 through X5).

Production and test environments run the exact same underlying state machine logic; the drive mode merely governs the pacing of effect release. This turns the agent runtime from an unpredictable black box into a discrete, steppable, and provably verifiable state machine.

---

From Ephemeral Loops to Database-Grade Agent Infrastructure

Looking at the evolution of computer science, every computational architecture matures by shifting from ad-hoc in-memory manipulation to rigorous data abstractions and explicit durability boundaries:

• Operating systems introduced virtual memory, journaling file systems, and process control blocks, replacing bare-metal scripts that crashed entire machines on a single fault;
• Relational databases established Write-Ahead Logging (WAL) and ACID transactions, liberating applications from manual state reconciliation;
• The infrastructure supporting modern AI Agents is undergoing the exact same paradigm shift.

Early agent frameworks built around naive while loops resemble bare-metal memory scripts, entangling orchestration state, conversation history, concurrency locks, and external I/O inside volatile process memory. The architecture of Pi Durable AgentHarness v2 signals the end of that fragile era.

Through intent-first fold reduction, decoupled tree and lane concurrency, strict KV cache discipline, and gated effects boundaries, the system not only constructs an airtight defense against real-world network fluctuations, sudden crashes, and concurrent workloads, but also sets a mature engineering standard for autonomous agent infrastructure.

When the probabilistic creativity of large language models is anchored inside a deterministic, crash-resilient runtime, autonomous AI applications can finally step beyond toy demonstrations and shoulder the demands of mission-critical production systems.