Design and Execute Stress Test Scenarios

This skill designs comprehensive stress test scenarios for applications, systems, and infrastructure, aiming to reveal breaking points, performance bottlenecks, and failure modes under extreme conditions. It defines seven test categories - volume stress (maximum data volumes), load stress (beyond normal user capacity), memory stress (allocation limits and garbage collection), CPU stress (computationally intensive operations), I/O stress (disk/network/database throughput limits), concurrency stress (thread safety and race conditions), and resource depletion (behavior when resources are exhausted) - and a five-step design framework: establish a baseline of normal operating parameters, identify the breaking point (maximum sustainable load), test recovery behavior after stress is removed, analyze cascading failure impact on dependent systems, and monitor CPU/memory/I/O/network metrics throughout.

stress_scenario:
  name: "Progressive API Load Test"
  duration: 30m
  stages:
    - users: 100, duration: 5m    # Warm-up
    - users: 500, duration: 10m   # Normal load
    - users: 2000, duration: 10m  # Stress load
    - users: 5000, duration: 3m   # Peak stress
    - users: 100, duration: 2m    # Recovery
  
  success_criteria:
    response_time_p95: < 2000ms
    error_rate: < 5%
    system_recovery: < 60s

It gives concrete scenario templates per stress type: a JMeter-style progressive load plan (as above, warm-up through peak stress and recovery, with p95 response time, error rate, and recovery-time success criteria); a Python memory-exhaustion test that progressively allocates chunks (e.g. 100MB at a time) while monitoring psutil memory/CPU usage and halting at a critical threshold (95% memory used); a SQL Server script that exhausts the connection pool by holding long-running cross-join queries open for a fixed duration up to a max connection count; a Kubernetes Job manifest using the polinux/stress image to apply CPU, I/O, and memory worker load with defined resource requests/limits and a timeout; and a chaos-engineering bash script using tc qdisc to simulate network latency, packet loss, and bandwidth limitation while polling application response time, with cleanup to remove the simulated conditions afterward.

For monitoring it defines a metrics dashboard structure covering system metrics (CPU utilization, memory usage, disk I/O ops/sec, network throughput) and application metrics (p95 response time, error rate, active connections, queue depth, GC frequency), each with critical thresholds (e.g. CPU at 90%, memory at 85%, response time at 5000ms, error rate at 10%).

Use it when designing a stress test for a specific failure mode - load capacity, memory exhaustion, database connection limits, CPU/IO saturation, or network degradation - and when you need concrete thresholds and monitoring metrics to define success/failure criteria, not just an ad hoc load spike.

Best practices to follow: use dedicated test environments that mirror production, implement circuit breakers to prevent cascading failures, monitor downstream dependencies during the test, and document baseline metrics beforehand. Apply stress gradually - start around 2x normal load and increase incrementally, allow stabilization between increases, isolate one component at a time, and include realistic user behavior with think times. Afterward, verify recovery to baseline, check data integrity, confirm no memory leaks or resource cleanup issues remain, and document exact test conditions, observed breaking points and failure modes, and runbooks for the failure scenarios found, on a regular schedule aligned with releases.

Input is the system or component to stress test and the failure mode of interest (volume, load, memory, CPU, I/O, concurrency, or resource depletion). Output is a runnable stress test scenario in the appropriate tool/language - a JMeter-style load plan, a Python memory-stress script, a SQL connection-exhaustion script, a Kubernetes stress Job, or a tc-based network chaos script - paired with a monitoring dashboard definition and defined success/failure thresholds.

SREs, performance engineers, and infrastructure teams who need to design and run realistic stress tests to find system breaking points before production incidents do, across application, database, and infrastructure layers.