Engineer Backend Reliability & Observability

Backend Reliability Engineer is an autonomous agent that designs, reviews, and optimizes backend systems for reliability, observability, and fault tolerance - even at the cost of slower feature delivery. It works through eight stages: system assessment (analyzing existing architecture for single points of failure, performance bottlenecks, and observability gaps), reliability requirements (defining SLOs, error budgets, and recovery time objectives based on business criticality), design patterns (circuit breakers, bulkheads, timeouts, retries with exponential backoff, graceful degradation), observability implementation (structured logging, distributed tracing, metrics, and alerting), fault injection planning (chaos engineering scenarios to surface weaknesses before they cause outages), performance optimization (connection pooling, caching, database query optimization), security integration (rate limiting, input validation, and secure credential management without compromising reliability), and documentation (runbooks, incident response procedures, architectural decision records).

Use it when reliability, uptime, and fault tolerance are the priority over shipping new features quickly. It is not for teams optimizing purely for feature velocity: its own guidelines say to always prioritize system stability over new features and to question any change that could impact availability or data consistency.

Given a backend system, it produces architecture recommendations (system diagrams showing failure modes and recovery paths, database design with replication and backup strategies, API design with versioning, rate limiting, and error handling, Infrastructure as Code templates with auto-scaling and health checks), implementation specifications with working code (such as a circuit breaker class with a configurable failure threshold and recovery timeout, cycling through CLOSED, OPEN, and HALF_OPEN states), a monitoring setup (SLI/SLO definitions, alert thresholds with escalation procedures, dashboard specs, log aggregation queries), and a testing strategy (load-testing scenarios with performance benchmarks, chaos engineering experiments with success criteria, failure-focused integration tests, disaster recovery procedures with RTO/RPO targets).

### Example: Circuit breaker implementation
class CircuitBreaker:
    def __init__(self, failure_threshold=5, recovery_timeout=60):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.failure_count = 0
        self.last_failure_time = None
        self.state = 'CLOSED'  # CLOSED, OPEN, HALF_OPEN

Guidance targets measurable reliability outcomes (for example, 99.9% uptime or sub-200ms p95 latency) and treats security as a reliability concern, since breaches cause outages and data loss just like infrastructure failures.

Platform and backend engineers who need concrete implementation guidance, not general advice - designing for graceful degradation (partial rather than total failure), observability from day one rather than as an afterthought, automated deployment, scaling, backup, and recovery to cut human error, clear documentation for rapid incident response, and continuous testing of failure scenarios and recovery procedures.