Design Chaos Engineering Experiments

Acts as a chaos engineering expert that designs, implements, and analyzes controlled failure experiments to improve system resilience, built on five core principles: form hypotheses around steady-state behavior using metrics and KPIs, vary real-world failure events that reflect production scenarios, run experiments in production where failures actually matter, automate experiments as part of the continuous delivery pipeline, and minimize blast radius by starting small and scaling up gradually. Its experiment-design framework has two stages. Hypothesis formation produces a structured definition: a steady-state hypothesis, named metrics with thresholds (for example response-time p95 under 500ms, availability above 99%, error rate under 1%), explicit failure conditions (a target service, a failure type such as network partition, and a duration), and a blast-radius scope (percentage of traffic and target environment). Progressive rollout then expands that blast radius through defined stages - 1%, 5%, 10%, 25%, 50% of traffic - monitoring steady state for five minutes at each stage and pausing for a two-minute recovery window before the next.

Use it to plan and run chaos experiments at the Kubernetes level (pod-delete experiments via Litmus ChaosEngine), at the network level (latency and jitter injection via Chaos Mesh's NetworkChaos, schedulable via cron), and at the infrastructure level (EC2 instance termination via Chaos Toolkit's AWS provider), and to wire up the observability and safety net around them: Prometheus instrumentation for experiment outcomes and recovery time, and a circuit breaker that aborts a run automatically.

Input is the target system plus a stated steady-state hypothesis. Output is a structured experiment definition - a YAML hypothesis with steady-state metrics, failure conditions, and blast radius - plus the supporting code to execute it: a progressive-rollout experiment runner, Kubernetes and network-chaos manifests, an AWS Chaos Toolkit experiment definition, Prometheus instrumentation, and a circuit-breaker safeguard.

Built around Chaos Monkey, Litmus, Gremlin, and Chaos Toolkit as the reference tooling, with concrete implementations shown for Litmus ChaosEngine (Kubernetes pod deletion, configurable duration/interval/force/percentage-affected), Chaos Mesh's NetworkChaos (delay, correlation, jitter, cron scheduling), Chaos Toolkit's AWS provider (chaosaws.ec2.actions.terminate_instances, filtered by environment tag), and Prometheus client metrics - a Counter for total experiments by status, a Gauge for whether steady state was maintained, and a Histogram for recovery time.

For engineers and SRE/platform teams who need to validate distributed-system resilience through controlled, production-safe failure injection rather than ad hoc breakage. The best-practice guidance built in: start with low-impact experiments like added delay before moving to service failures, always baseline steady-state behavior first, document every experiment and its results, automate rollback so any chaos injection can be reversed quickly, get stakeholder sign-off before running in production, schedule recurring game days to keep the team ready, and measure the business impact of what the experiments reveal. As the source puts it, chaos engineering is not random breakage but the systematic discovery of weaknesses under controlled conditions.

apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: pod-delete-chaos
  namespace: default
spec:
  appinfo:
    appns: 'default'
    applabel: 'app=nginx'
    appkind: 'deployment'
  chaosServiceAccount: litmus-admin
  experiments:
  - name: pod-delete
    spec:
      components:
        env:
        - name: TOTAL_CHAOS_DURATION
          value: '60'
        - name: CHAOS_INTERVAL
          value: '10'
        - name: FORCE
          value: 'false'
        - name: PODS_AFFECTED_PERC
          value: '50'