Test Agent Recovery with Deterministic Fault Injection

Agent Harness Fault Injection tests whether an agent workflow preserves state, budgets, safety boundaries, and evidence when a dependency fails, using a deterministic, non-production fault schedule rather than random chaos testing. It runs strictly in a disposable sandbox with synthetic inputs and stubbed tools, network disabled by default, and every injected fault is an in-memory or fixture-controlled event - it never deletes real data, revokes real credentials, or mutates a live service to create a failure. Before injecting anything, it writes a recovery contract (what state must survive, what side effects must not repeat) and models the workflow as explicit states (created -> running -> checkpointed -> waiting_for_tool -> failed/recovering -> resumed -> completed), then selects the smallest fault matrix that covers the new recovery logic - sandbox denial, MCP/tool timeout, worker restart, missing or stale checkpoint, parallel branch failure, memory loss, and retry/deadline exhaustion - each with a required observation and expected containment. A worker-restart test must reject a future-version or corrupted checkpoint instead of guessing at progress, and if exactly-once delivery cannot be proven after resumption the verdict is downgraded rather than assumed clean. Parallel-branch tests represent each branch as its own child attempt under one predeclared join policy - all_required (any required branch failure stops the join), best_effort (continue with an explicit degraded marker), or compensate (run a bounded compensating action, then stop or resume) - and a successful sibling branch is never allowed to erase a failed branch from the final ledger. Faults are scheduled by event ordinal rather than wall-clock randomness, for example a tool timeout on the second search call, and the same schedule is run twice to confirm the normalized timeline matches before trying a different one.

Use it when a multi-step agent, state machine, or multi-agent workflow has a new recovery path, when sandbox execution, an MCP/tool call, a worker, a checkpoint store, or memory can time out or disappear, or before claiming retry/resume/deadline/isolation/partial-failure behavior is production-ready. Never use it against a production target, real user data, live credentials, or an unbounded external service - convert those to a local simulator or an authorized staging harness first. A missing test scope, fixture, or recovery contract makes the verdict inconclusive by definition, not a pass, and the skill itself does not select production SLOs, repair a broken workflow, or grant permission to test a system outside the declared sandbox.

Input is a frozen workflow revision, model/prompt config, tool schemas, seed, input fixture, timeout, retry budget, and expected terminal states, plus the fault schedule itself. Output is a small fault matrix, an event timeline, and one of four verdicts: recovered (the declared invariant held and the workflow completed within budget), contained_failure (the fault was isolated and the workflow stopped safely as designed - reported separately from completed work, since containment is not the same as autonomous success), unrecoverable (recovery violated an invariant, repeated a side effect, crossed a boundary, or exceeded budget), or inconclusive (the fixture, checkpoint, contract, or evidence was insufficient to judge). The verdict is gated on invariants and remaining budget, never on output text alone, since a green final answer can still hide a deadline violation or a duplicated effect - a read timeout on a tool call is never treated as proof that the underlying write did not happen, so a side effect is only retried when the tool's contract says the idempotency key is safe to replay. Retry does not clear the original fault: fault id, original error class, attempt count, and retry lineage stay in the evidence ledger.

None beyond the agent harness itself and its stubbed tools/checkpoint store; secrets and personal data are redacted before a timeline is stored or attached to a pull request. It pairs with @agent-evaluation-reporting for outcome reporting, @cross-platform-contract-propagation-audit for tracing recovery fields across consumers, and @multi-agent-patterns for choosing a topology before testing its failure behavior.

Engineers building or reviewing a multi-step agent workflow's retry, resume, or failure-isolation logic who need reproducible, replayable failure evidence instead of a one-off chaos test or an assumption that recovery "probably works".