Implement graceful shutdown for zero-downtime deploys

This skill implements graceful shutdown for servers, workers, and long-running processes. It ensures in-flight requests complete, background jobs finish or checkpoint, database connections close cleanly, and the process exits with a proper status code. It is essential for zero-downtime deployments in container orchestrators (Kubernetes, ECS, Docker Compose) and bare-metal process managers (systemd, PM2). The examples are mostly TypeScript for Node.js, with one Python example.

The five steps:

1. Register signal handlers early. Trap SIGTERM (orchestrator shutdown) and SIGINT (Ctrl+C) at process startup and set a flag so the application knows it is shutting down. A guard prevents a double shutdown when both signals arrive.
2. Stop accepting new work. For HTTP servers, close the server; for queue workers, stop polling for new jobs. Mark the health check as not-ready so load balancers stop routing.
3. Drain in-flight work with a deadline. Wait for active requests and background tasks to finish, flush buffered data such as logs, metrics, and queues, close external resource handles, and enforce a hard deadline so the process never hangs. The drain timeout must be less than the orchestrator's termination grace period, and the force-exit timer should be unreferenced so it does not keep the event loop alive.
4. Implement readiness and liveness probes. Liveness proves the process is alive and readiness controls whether traffic is routed. While the listener is available during a drain, keep liveness healthy and return 503 only from readiness. After the listener closes, new probes cannot connect, so do not promise that HTTP liveness stays reachable for the whole termination window.
5. Track active connections with a counter of in-flight requests, and use a once guard over both the finish and close events so client aborts correctly decrement it.

Use it when building an HTTP server that must not drop active connections during deploys, writing a background worker that processes queue jobs, deploying to Kubernetes, Docker, or any environment that sends SIGTERM before killing, or when the user mentions graceful shutdown, draining connections, handling SIGTERM, zero downtime, or not killing active requests. Also use it when implementing health check endpoints for orchestrators.

Know its limits. It does not replace environment-specific validation or expert review. WebSocket and SSE connections need application-level close frames before severing, since closing the server alone will not end them gracefully. In clustered setups each worker must handle signals independently. Some cloud platforms, such as Heroku and Railway, send SIGTERM with very short grace periods of 10 to 30 seconds, so adjust drain timeouts.

The input is your server or worker code. The output is shutdown logic with signal handling, readiness and liveness endpoints, request tracking, and a deadline. Three examples are given: an Express server that registers health routes before the drain-rejection middleware, rejects new requests with 503 and a Connection close header during drain, and force-exits after 25 seconds; a Python FastAPI app under Uvicorn, where Uvicorn owns SIGTERM handling and draining and the lifespan hook only releases application resources after the drain; and a background worker that checkpoints its current job on SIGTERM before exiting. For Uvicorn, run it with a deadline shorter than the orchestrator's kill timeout:

uvicorn app:app --timeout-graceful-shutdown 25

If readiness must turn 503 before SIGTERM arrives, coordinate a separately secured and tested pre-stop drain signal plus a propagation delay, because the lifespan hook runs too late for that.

Kubernetes is covered with a deployment configuration setting the termination grace period to 30 seconds, an HTTP liveness probe on the healthz path, and a readiness probe on the readyz path. Set the grace period to at least the drain timeout plus a 5 second buffer; if the longest request takes 60 seconds, use 70 for the grace period and 65 for the drain timeout. The skill also mentions Docker, ECS, systemd, and PM2.

Best practices: always set a drain timeout shorter than the orchestrator's kill timeout (25 seconds against Kubernetes' 30 second default), return a Connection close header during draining, reject new requests with 503 during shutdown, flush async buffers before exit, use exit code 0 for clean shutdown and 1 for timeout or error, and test shutdown by simulating SIGTERM in integration tests.

Common pitfalls: keep-alive connections that keep the server open after close resolves, which should be tracked and destroyed when idle; a load balancer still routing after SIGTERM because readiness returns 200, fixed by flipping readiness to 503 immediately on the signal; and a deadlocked drain, which needs a hard force-exit timeout as the final backstop.

It is for backend and platform engineers who deploy services and workers to orchestrated environments and need deploys that do not drop requests or lose jobs.