Test Agent Recovery with Deterministic Fault Injection

Runs a deterministic, non-production fault-injection schedule against a multi-step agent workflow to test whether its recovery logic actually preserves state, budgets, safety boundaries, and evidence when a dependency fails - rather than trusting a claim that retry, resume, or partial-failure handling is production-ready. It works from a written recovery contract (the invariant that must survive a fault, e.g. "resume from the latest durable checkpoint... commit each externally visible effect at most once"), an explicit state model of the workflow, and a fault matrix that selects the smallest set of faults covering the new recovery path - sandbox denial, an MCP/tool timeout, a worker restart, a missing or stale checkpoint, a parallel-branch failure, memory loss, or retry/deadline exhaustion - each with its required observation and expected containment. Faults are injected on a deterministic, JSON-defined schedule keyed to event ordinals rather than wall-clock randomness, so the same schedule can be replayed and compared before ever moving to a randomized run.

The output is one machine-readable evidence record (run id, monotonic sequence, state before/after, actor, event, fault id, attempt, checkpoint sequence, retry/deadline remaining, and a redacted evidence reference) plus a concise human summary, and a verdict of recovered, contained_failure, unrecoverable, or inconclusive - explicitly treating a merely-contained failure as distinct from actual completed work, and treating any missing evidence, fixture, or recovery contract as inconclusive rather than a pass.

Use it when a multi-step agent, state machine, loop, or multi-agent workflow has a new recovery path; when a sandboxed tool call, MCP/tool call, worker, checkpoint store, or memory can time out or disappear; before claiming retry, resume, deadline, isolation, or partial-failure behavior is production-ready; or when a regression needs reproducible failure evidence instead of a random chaos run.

Do not use it against a production target, real user data, live credentials, or an unbounded external service - convert those cases to a local simulator or an authorized staging harness first. It also doesn't select production SLOs, repair broken workflows, or grant permission to test systems outside the declared sandbox, and a local stub can't prove the behavior of a deployed model, MCP server, scheduler, filesystem, or network.

Inputs: a frozen workflow revision, model/prompt configuration, tool schemas, seed, input fixture, timeout, retry budget, and expected terminal states; a written recovery contract naming the invariant that must survive; and a deterministic fault schedule (a JSON list of faults tied to event ordinals, e.g. a tool timeout on call 2 or a worker restart after checkpoint 2). Tests run in a disposable sandbox with synthetic inputs, stubbed tools, and network disabled unless a local test server is explicitly needed - every injected failure is in-memory or fixture-controlled, never a mutation of real data, credentials, or a live service.

Output is the fault matrix result: a machine-readable JSON record with the run id, per-invariant pass/fail, the faults injected and whether each was handled, and an event timeline; a human summary stating the frozen contract, injected schedule, verdict, failed invariants, and budget consumption; and one of the four terminal verdicts above.

Composes with @agent-evaluation-reporting to report autonomous, assisted, failed, timed-out, and invalid outcomes; @cross-platform-contract-propagation-audit to trace recovery fields and status contracts across consumers; and @multi-agent-patterns to choose a multi-agent topology before testing its failure behavior.

Teams building multi-step or multi-agent workflows who need reproducible, evidence-backed proof that a new recovery path actually holds its invariants under sandbox denials, tool timeouts, worker restarts, or partial fan-out failures - rather than a chaos-testing run whose pass/fail is unrepeatable.