Optimize Agent Performance Systematically

This skill drives systematic improvement of an existing agent through a four-phase workflow. Phase 1 establishes baseline metrics via the context-manager tool (analyze-agent-performance --days 30): task completion rate, response accuracy, tool-usage efficiency, latency, token consumption, and user-satisfaction signals like corrections and retries - then classifies failures by root cause (instruction misunderstanding, output-format errors, context loss, tool misuse, constraint violations, edge-case handling).

Phase 2 applies prompt-engineering improvements: chain-of-thought enhancement with explicit reasoning steps and self-verification checkpoints, curated few-shot examples (including both good and bad examples with explanations), role-definition refinement (core purpose, expertise domains, constraints, success criteria), Constitutional AI-style self-critique-and-revise loops, and output-format tuning.

Phase 3 runs a rigorous A/B test comparing the original and improved agent versions - minimum 100 tasks per variant, 95% confidence level, effect-size (Cohen's d) calculation - scored across task-level, quality, and performance metrics, with a blind human-evaluation protocol alongside automated scoring.

Phase 4 governs version control and rollout: semantic versioning (agent-name-vMAJOR.MINOR.PATCH), a staged rollout (5% alpha, 20% beta, canary ramp to 100%, then a 7-day monitoring window), and explicit rollback triggers (success rate drops over 10%, critical errors up over 5%, cost per task up over 20%, or safety violations detected) with a documented rollback process.

Use this when improving an existing agent's performance or reliability, analyzing failure modes or tool-usage patterns, running structured A/B tests, or designing an iterative optimization cadence (weekly monitoring, monthly pattern analysis, quarterly major updates).

Not for building a brand-new agent from scratch, and not usable without existing metrics, user feedback, or test cases to establish a baseline against - the whole workflow depends on having something measurable to compare against.

Inputs: historical performance data and user interaction logs for the existing agent, representative test scenarios (golden path, regression, edge case, stress, adversarial, cross-domain), and defined success criteria (target: success rate up 15%+, corrections down 25%+, no safety regressions, latency within 10% of baseline, cost up no more than 5%).

Outputs: a quantitative baseline report, a revised prompt/workflow with documented changes, A/B test results with statistical significance, a versioned and staged rollout, and a post-deployment review after 30 days feeding the next optimization cycle.

context-manager (performance data collection), prompt-engineer agent (prompt optimization techniques), parallel-test-runner (A/B testing framework), Git-based prompt version storage.

Teams operating a production agent that needs systematic, safety-conscious performance improvement - with baseline measurement, A/B validation, and rollback safeguards - rather than ad hoc prompt tweaking.