Graceful shutdown and what SIGTERM starts

A deploy goes out. Zero-downtime rollout, so the new instance comes up, passes its health check, starts taking traffic. Then the old instance has to go away. At that exact moment it is halfway through a card charge: the payment provider has been called, the response hasn't come back, the order row hasn't been written.

If that process just dies, you have a charge with no order. The customer sees a failed checkout, retries, and pays twice. Nothing about zero-downtime deployment protects you here — the whole mechanism is about the new server being ready. What the old server does on its way out is your code's problem.

The operating system doesn't just pull the plug

Every application runs inside a process, and every process has a lifecycle: it starts, it runs, it terminates. Graceful shutdown lives entirely in that last phase.

When the OS wants a process gone, it doesn't cut power. It opens a conversation, and the vocabulary of that conversation is signals — Unix's interprocess communication mechanism. Your application registers handlers for the signals it cares about; the OS sends one; the handler runs.

Three signals matter.

SIGTERM is the polite one. A tap on the shoulder: finish up and leave. Your process gets to decide what "finish up" means and how long it takes. This is what Kubernetes sends. It's what systemd sends, what PM2 sends, what Docker sends on docker stop. Anything that manages processes for a living uses SIGTERM.

SIGINT is the same request with a human behind it — Ctrl+C in a terminal. Development environments, mostly, because it requires a key press.

Handle both identically. It does not matter whether a deploy pipeline or a developer's left hand initiated the shutdown; the intent is the same, so the response should be the same.

ctx, stop := signal.NotifyContext(context.Background(),
    os.Interrupt, syscall.SIGTERM)
defer stop()

<-ctx.Done()
shutdown()

SIGKILL is the third, and it is not part of the conversation. It cannot be caught, cannot be handled, cannot be ignored. The process stops mid-instruction. No cleanup, no final log line, no rollback. This is the plug coming out of the wall, and it is what you get if you ignore SIGTERM long enough — Kubernetes sends SIGTERM, waits out terminationGracePeriodSeconds, then sends SIGKILL. Your grace period is a loan, not a gift.

Everything you'll ever write about shutdown is an attempt to finish before SIGKILL arrives.

Step one: stop taking new work

A restaurant closing at 11pm doesn't kill the lights on people mid-meal. Someone at the door stops seating new parties first. Then the room is told they have twenty minutes.

That's connection draining, and the order is the entire point. Stop accepting new connections before you start waiting on existing ones, otherwise the set you're waiting on keeps growing and you never converge.

The mechanics change per protocol, the shape does not:

• HTTP server — stop accepting new connections, let in-flight requests run to completion, then close the listener.
• Database — stop admitting new queries, let open transactions commit or roll back, then drop connections.
• WebSockets — send a close frame so clients know to reconnect elsewhere, then tear down the socket. A socket that vanishes without warning looks like a network fault to the client, and it will retry against an instance that's already gone.

Most frameworks hand you this as one call. Go's http.Server has Shutdown(ctx); Node's has server.close(). The library stops the listener and blocks until the handlers return.

Step two: pick a timeout, and mean it

You cannot wait forever. A request stuck on a hung upstream will happily hold the door open until SIGKILL settles it for you, which is precisely the outcome you were trying to avoid.

So every real shutdown path is bounded. Thirty seconds is the usual default, and for an ordinary HTTP backend it's generous — once you've stopped accepting new connections, a p99 of 300ms drains in well under a second.

The number is a genuine tradeoff, not a constant to copy:

• Too short and you interrupt legitimate work. A report endpoint that takes 45 seconds gets guillotined at 30.
• Too long and every deploy crawls. Rolling twenty pods at 120 seconds each is a slow rollout and a slow rollback, which matters most on the day you need the rollback.

Derive it from your actual request duration distribution — the long tail, not the median — and make sure the platform's grace period is larger than your application's timeout. If your code waits 30 seconds and Kubernetes only allows 20, your timeout is decorative.

Draining also has to be coordinated with whatever routes traffic to you. The load balancer needs to have marked the instance unhealthy and stopped sending it new requests before, or at least as, you begin draining. Otherwise you drain cleanly and then take a fresh request into a server that has already closed its database pool. In Kubernetes this is the reason for preStop sleeps: give endpoint deregistration time to propagate before the process starts refusing work.

Step three: release what you acquired, backwards

Once requests are done, hand back what the process is holding: file handles, network connections, database pools, Redis clients, temporary files. The OS caps how many file descriptors and sockets a process can hold, and leaked handles are how you end up with a slow memory climb across restarts.

Database connections deserve specific attention. Open transactions must be explicitly committed or rolled back before the pool closes. Leave them dangling and the server holds locks until its own timeout expires — which shows up as deadlocks and mysterious contention on the new instance, minutes after the old one is gone.

Release in reverse order of acquisition. If you opened Redis, then Postgres, then the job server, you close the job server, then Postgres, then Redis. The job server depends on Redis; kill Redis first and the shutdown path of the thing depending on it fails halfway through. It's the same discipline as defer or a stack unwind, and for the same reason.

The full sequence, then:

SIGTERM received
  -> stop accepting new connections
  -> wait for in-flight requests (bounded, ~30s)
  -> close job workers
  -> close database pool
  -> close cache connections
  -> exit 0

Run it locally and Ctrl+C your own server. If it exits instantly with no log line between the interrupt and the prompt, nothing is draining — you just got lucky that there was nothing in flight. In production there always is.