Chapter 15: Graceful Shutdown

15.1 What “Graceful” Actually Promises

The word does a lot of unexamined work. Before designing a shutdown it is worth being precise about what you are promising, because “graceful” is not one guarantee — it is four, they are independent, and most implementations deliver two of them.

15.1.1 The Four Promises

No new work is accepted. After the signal, a request that has not yet been admitted is refused cleanly — refused in a way the caller can retry elsewhere, not dropped on the floor.

Accepted work finishes. A request already inside a handler, a job already pulled from a queue, a transaction already begun runs to completion or is deliberately abandoned in a recoverable state. This is the promise that “graceful” usually means and the one most often broken by accident.

Resources are released in an order that is safe. The database connection closes after the last query that needs it, not before. The lease is surrendered rather than left to expire. The buffer is flushed rather than dropped.

The outcome is reported. If something was abandoned, the process says so — in its exit code, its logs, and its metrics — instead of exiting zero and leaving the operator to infer it.

The four are independent. A server that calls Shutdown and waits keeps the first two and can still break the last two. A process that flushes its telemetry perfectly can still have refused a hundred requests it should have served. Naming them separately is what lets you say which one a given bug violates.

15.1.2 What an Abrupt Exit Loses

Chapter 2 said the runtime does not wait for your goroutines and does not run their deferred functions when the process ends. That is exactly true, and shutdown is where the bill arrives:

// Illustrative snippet — not a complete program
func main() {
    defer flushTelemetry() // never runs
    lease := acquireLease()
    defer lease.Release()  // never runs
    serve()
}

The first three are the shell’s 128 + signal number. SIGQUIT is the odd one: the Go runtime intercepts it, prints a full goroutine dump, and exits 2 rather than 131 — which §15.7.5 turns into the diagnostic for a shutdown that has hung.

The same is true of os.Exit, which skips deferred functions by specification, and of a panic in any goroutine that nothing recovers (§14.6). Three different ways to reach the same place.

What is lost is not abstract:

That last row is the one that generates support tickets. The client knows the request failed; it does not know whether the work failed. If the operation was not idempotent, neither does anyone else.

15.1.3 Shutdown Is Startup in Reverse

Every dependency you built up on the way in has to come down in the opposite order.

startup                     shutdown
───────                     ────────
1  load config              5  close config sources
2  dial the database        4  close the pool
3  start workers            3  stop workers, drain
4  start listeners          2  stop accepting
5  announce ready           1  fail readiness

A component may not close while anything above
it still needs it. Reverse the numbering and the
ordering question answers itself.

That is the whole of the ordering rule, and it explains most shutdown bugs by inspection: closing the database pool while workers are still draining is step 4 before step 3.

The rule has one important exception, and it is the first step. Readiness comes down before listeners, not after — which is startup order reversed, but for a reason that has nothing to do with dependencies and everything to do with the fact that something outside your process is still sending you traffic. §15.3 measures what happens when you get that one backwards.

15.1.4 The Clock You Do Not Own

Shutdown has a deadline, it is not yours, and it is enforced with a signal you cannot catch.

Under Kubernetes the container runtime sends SIGTERM, waits terminationGracePeriodSeconds (30 seconds by default), then sends SIGKILL. Under systemd the unit’s TimeoutStopSec plays the same role. The numbers differ; the shape does not. Something started a timer when it signalled you, and it will not negotiate.

This is why “shutdown timeouts” is not one number but a division problem. The propagation wait, the drain and the resource release all come out of the same budget, and §15.7 is about carving it up and about what to do when the carving turns out to have been optimistic.

15.1.5 Common Mistakes

Summary: What “Graceful” Actually Promises

Graceful is four independent promises: refuse new work cleanly, finish what was accepted, release resources in a safe order, and report what happened. Most implementations keep the first two and quietly break the second two, and naming them separately is what makes a given bug describable.

An abrupt exit keeps none of them. SIGTERM with no handler is exit 143 with no deferred function run at all — the same place os.Exit and an unrecovered panic reach by different routes. What is lost is buffered telemetry, held leases, torn writes and accepted requests whose callers cannot tell a failure from a duplicate.

The ordering rule is that shutdown is startup reversed: a component may not close while anything above it still needs it. The single exception is readiness, which comes down first, because something outside the process is still routing traffic to you.

And the deadline is not yours. The platform sends SIGTERM, starts a timer, and ends the argument with SIGKILL. Your design problem is not choosing a timeout but dividing a budget someone else set.

Self-Check Questions: What “Graceful” Actually Promises

Which of the four promises does this break?

// Illustrative snippet — not a complete program
func main() {
    defer db.Close()
    srv := &http.Server{Handler: h}
    go func() { <-sigCh; srv.Shutdown(context.Background()) }()
    srv.ListenAndServe()
}

15.2 The Signal, and the Deadline It Starts

Shutdown begins with a signal, and the signal starts a clock. This section is about catching the first correctly and about where the second comes from — they are one topic, because a signal you handle badly costs you budget you did not have to spare.

15.2.1 The Channel Form, and the Bug vet Catches

The form Chapters 2 and 4 used, and the one still in most codebases:

// Illustrative snippet — not a complete program
c := make(chan os.Signal, 1) // buffered -- this matters
signal.Notify(c, syscall.SIGINT, syscall.SIGTERM)
<-c

The buffer is not decoration. signal.Notify's documentation is unusually blunt about why:

A signal delivered while nobody is receiving is discarded, silently. With an unbuffered channel there is a window between Notify and the receive in which your one and only SIGTERM can vanish, and the process then sits there until SIGKILL arrives.

Chapters 13 and 14 opened with code that compiles, vets clean and passes -race. This is the sharper variant: a bug the toolchain already knows about, in code that ships anyway because nobody ran vet in CI.

15.2.2 signal.NotifyContext

Since Go 1.16 the whole dance is one call:

// Illustrative snippet — not a complete program
ctx, stop := signal.NotifyContext(context.Background(),
    syscall.SIGINT, syscall.SIGTERM)
defer stop()

<-ctx.Done() // the signal arrived

No channel, no buffer decision, no vet finding. And the result is a context.Context, which means the shutdown signal composes with everything Chapters 13 and 14 built: pass it to Shutdown, derive a budget from it with WithTimeout, hand it to an errgroup.

That last point is the one to hold on to. The signal is not an event you handle at one place; it is the root of a cancellation tree that reaches every component. §15.6 is that tree.

15.2.3 What You Cannot Catch

SIGKILL and SIGSTOP cannot be caught, blocked or ignored. This is a kernel guarantee, not a Go limitation, and it is the reason the budget in §15.1.4 is a hard bound rather than a suggestion.

It is also why “handle SIGKILL for cleaner shutdown” is not a task you can accept. When SIGKILL arrives the process stops executing instructions. There is no handler, no defer, no final flush. Everything you wanted to happen had to have happened already.

15.2.4 Naming the Signal

ctx.Done() tells you shutdown started. ctx.Err() tells you almost nothing:

NotifyContext is built on context.WithCancelCause (§13.3.4; since Go 1.26) and passes a cause naming the signal. SIGINT gives interrupt signal received, SIGHUP gives hangup signal received.

This matters more than it first appears. SIGINT is usually a developer at a terminal who wants their prompt back; SIGTERM is usually an orchestrator that has already started counting. They deserve different budgets, and context.Cause lets one handler tell them apart without a second channel:

// Illustrative snippet — not a complete program
switch cause := context.Cause(ctx); {
case errors.Is(cause, context.Canceled):
    // nobody signalled us -- we initiated this ourselves
    budget = 30 * time.Second
default:
    slog.Info("shutdown signalled", "cause", cause)
    budget = platformBudget()
}

15.2.5 defer stop() Is the Bug

Here is the idiom nearly everyone writes:

// Illustrative snippet — not a complete program
ctx, stop := signal.NotifyContext(
    context.Background(), syscall.SIGTERM)
defer stop() // ✗ leaves the handler installed for the whole shutdown
<-ctx.Done()
drain()      // if this stalls, the operator is stuck

NotifyContext's documentation says what this costs, and it is easy to read past:

While the handler is installed, every subsequent signal is absorbed. The second Ctrl-C does nothing. The second SIGTERM from an impatient operator does nothing. If drain() has stalled on a connection that will never close, the only way out is SIGKILL.

So the rule is: call stop() as the first thing you do after ctx.Done() fires, not at the end of main.

// Illustrative snippet — not a complete program
ctx, stop := signal.NotifyContext(context.Background(),
    syscall.SIGINT, syscall.SIGTERM)
defer stop() // still needed for the early-return paths

<-ctx.Done()
stop() // hand the signal back: a second one now kills us

slog.Info("shutdown started", "cause", context.Cause(ctx))
drain()

Keeping defer stop() as well is correct and not redundant: it covers the paths where main returns without a signal ever arriving, and calling stop twice is safe.

15.2.6 What the Escape Hatch Costs

Handing the signal back is not free, and this is the half that usually goes unsaid.

That is the trade. You gave the operator a way to escape a stalled drain, and in exchange you accepted that they can cut you off mid-cleanup. It follows that anything that must happen cannot live in a defer placed after stop(). Put it on the shutdown path, early, in the order §15.6 establishes — release the lease before you drain the last connection, not after.

Both halves of §15.1.2's lesson are now in play. defer does not run when the platform kills you, and it does not run when the operator does either.

15.2.7 Where the Budget Comes From

The signal starts a timer in the process that sent it.

When it expires, SIGKILL. The numbers vary by more than an order of magnitude across those three rows, which is the point: there is no default you can safely assume.

Derive your budget from the platform’s, with headroom, and put every phase inside it:

// Illustrative snippet — not a complete program
<-ctx.Done()
stop()

// One budget, slightly inside the platform's, for the whole sequence.
shutdownCtx, cancel := context.WithTimeout(
    context.WithoutCancel(ctx), 25*time.Second)
defer cancel()

Two details in that snippet earn their place. The headroom (25 against a 30-second grace period) leaves room to log the outcome before SIGKILL lands. And context.WithoutCancel is required, not stylistic: ctx is already cancelled — that is why we are here — so deriving a timeout from it directly produces a context that is dead on arrival. §15.5.6 returns to this.

15.2.8 A Signal Is Not the Only Trigger

Everything so far has assumed shutdown begins with SIGTERM. In a real service it is one of at least four triggers, and two of them can arrive together:

• A signal — the platform, or an operator.
• A component failing fatally — the database pool cannot reconnect, the consumer’s credentials expired. §15.6's group turns that into a cancelled context.
• The process deciding to leave — a node draining itself, a config reload that needs a restart, a leader that lost its lease.
• A second signal, which after stop() is not a trigger at all but the process ending (§15.2.5).

The first three have to funnel into one function, and that function must be safe to call more than once from more than one goroutine. A shutdown that runs twice concurrently closes a channel twice and closes a pool twice — turning a controlled exit into the panic §14.6 says takes the whole process down.

sync.Once is the entire answer, and §12.1 already built it:

// Illustrative snippet — not a complete program
type App struct {
    once   sync.Once
    result error
}

// Stop runs the shutdown sequence exactly once. Concurrent and
// later callers block until it finishes and get the same error.
func (a *App) Stop() error {
    a.once.Do(func() { a.result = a.shutdown() })
    return a.result
}

The reflex here is to add a done channel so that late callers can wait for the first one. It is unnecessary, and the reason is a sentence in sync.Once's own documentation — written as a warning about re-entrancy, but containing the guarantee we want: “Because no call to Do returns until the one call to f returns, if f causes Do to be called, it will deadlock.”

15.2.9 Common Mistakes

Summary: The Signal, and the Deadline It Starts

signal.NotifyContext is the entry point. It replaces the channel form, avoids the unbuffered-channel bug that go vet already catches, and — because it produces a context — makes the signal the root of a cancellation tree rather than an event handled in one place.

That context carries more than a bare cancellation. ctx.Err() is always context canceled, but context.Cause(ctx) names the signal, which lets one path distinguish a developer’s Ctrl-C from an orchestrator’s SIGTERM, and both from a shutdown the process started itself.

defer stop() is the bug. While the handler is installed every later signal is absorbed, so an operator watching a stalled drain has no escape but SIGKILL. Call stop() as the first thing after ctx.Done() — and accept the price, which is that a second signal then kills you mid-cleanup, so nothing that must happen may sit in a defer after that point.

The budget comes from the platform and varies from 10 seconds to 90 across common runtimes. Derive yours from a detached context with headroom, and treat it as a total to divide rather than a timeout to set.

Self-Check Questions: The Signal, and the Deadline It Starts

Why does go vet reject this, and what actually goes wrong?

// Illustrative snippet — not a complete program
c := make(chan os.Signal)
signal.Notify(c, syscall.SIGTERM)
<-c

What is wrong with this budget?

// Illustrative snippet — not a complete program
<-ctx.Done()
shutdownCtx, cancel := context.WithTimeout(ctx, 25*time.Second)
defer cancel()
srv.Shutdown(shutdownCtx)

// Illustrative snippet — not a complete program
shutdownCtx, cancel := context.WithTimeout(
    context.WithoutCancel(ctx), 25*time.Second)

15.3 The Sequence

There are five steps and their order is the whole design. Four of them follow from §15.1.3's reversal rule. The first does not, and getting it wrong is the most common production shutdown bug there is.

15.3.1 The Five Steps

1  fail readiness          tell the LB to stop routing
      │
      │  <-- propagation window: it has not noticed yet
      │
2  stop accepting          close listeners, stop pulling jobs
      │
3  drain                   let accepted work finish
      │
4  release                 close pools, surrender leases,
      │                    in reverse dependency order
5  report                  exit code, log, metric

Steps 2-5 are startup reversed. Step 1 is not: it
exists because something outside this process is
still sending you traffic and learns otherwise
only by polling.

Steps 2 through 5 need no justification beyond §15.1.3 — you stop taking new work before you finish old work, you release a resource after the last thing that needs it, and you say what happened. Step 1 is the one worth measuring.

The sequence is the same whichever of §15.2.8's four triggers started it. That is the point of funnelling them through one sync.Once: there is one shutdown, and it runs these five steps once.

15.3.2 Deregistration Is Concurrent With the Signal, Not Before It

The instinct is that shutdown starts when the signal arrives, and that by then traffic has already stopped. Neither half is true.

In an orchestrated environment two things happen at once. The control plane marks the pod terminating and begins updating endpoints, and the kubelet sends SIGTERM. These are concurrent, and the second usually wins the race: your process learns it is going away before the load balancer does.

control plane          your process           load balancer
     │                       │                      │
     ├── mark terminating ───┤                      │
     ├── SIGTERM ───────────►│                      │
     │                       │◄──── still routing ──┤
     ├── endpoints update ──────────────────────────┤
     │                       │                      │
     │                       │   now it has stopped │

Between SIGTERM and the endpoints update, requests
are still arriving. Close your listeners in that
window and every one of them is refused.

The propagation delay is real and is not small: an endpoints update has to reach every proxy, each of which is watching on its own schedule. Hundreds of milliseconds is normal, and under load it is worse.

15.3.3 Measured: What the Window Costs

The experiment: four clients dial the server in a loop, pausing 3 ms between requests, with keep-alives disabled so every request is a fresh connection — which is what a load balancer’s pool does. After 80 ms of steady state the shutdown begins, and the modelled balancer goes on routing for a further 150 ms before it notices. The only variable is whether the server keeps serving across that 150 ms window before closing its listeners.

Measured, ten runs:

The refusals are not a rounding error; they are roughly twice the number of requests successfully served. The absolute counts scale with the client rate and the length of the window — it is the ratio, and the zero, that survive a change of machine. And every one is a request the load balancer believed it had routed to a healthy backend — so the client sees a connection error, not a retryable 503, and whether it retries at all depends on a policy you do not control.

The fix is the one that looks like a hack:

// Illustrative snippet — not a complete program
<-ctx.Done()
stop()

health.SetNotReady()            // step 1
sleepCtx(hardCtx, propDelay)    // <-- required, not a hack
srv.Shutdown(shutdownCtx)       // step 2 and 3

The wait itself is a sleep the hard deadline can cut short:

// Illustrative snippet — not a complete program
func sleepCtx(ctx context.Context, d time.Duration) {
    t := time.NewTimer(d)
    defer t.Stop()
    select {
    case <-t.C:
    case <-ctx.Done():
    }
}

That wait is doing real work. It is not waiting for anything inside the process; it is waiting for a fact to propagate through a system you can only observe indirectly. There is no channel that tells you the load balancer noticed, which is why the shape is a sleep rather than a receive.

It is a bounded sleep, and that part is not decoration. §15.7.1 divides the grace period into phases and §15.7.8 makes it a rule that every phase carries its own deadline; a bare time.Sleep(propDelay) is the one construct in this chapter that cannot be cut short, so five seconds of it out of a thirty-second budget is five seconds the hard deadline can never reclaim. sleepCtx is the same wait with an escape, and every later listing uses it — §15.3.5 in the full sequence, and the select form in §15.4.2 that retires keep-alives at the same time.

15.3.4 Choosing the Wait

The wait should cover the worst case of “how long until every proxy has stopped routing to me”, which is roughly:

With a 2-second readiness period, a failure threshold of 1 and a proxy that reconciles every second, five seconds is a defensible number and ten is not paranoid. That is a large fraction of a 30-second budget, which is exactly why §15.7 treats the total as something to divide deliberately.

Two ways to spend less of it. If your platform supports a preStop hook, the wait can happen there, before SIGTERM is ever sent — the grace period clock starts after preStop completes, so the wait comes out of a different budget. And if you can observe your own readiness endpoint being polled, you can wait for the actual last poll rather than a worst-case constant. Both are optimisations on the same principle: the window exists, and something has to cover it.

15.3.5 The Rest of the Sequence

With step 1 established, the remainder is §15.1.3 applied literally:

// Illustrative snippet — not a complete program
func shutdown(ctx context.Context, app *App) error {
    // 1. stop being routed to
    app.health.SetNotReady()
    sleepCtx(ctx, app.cfg.PropagationDelay)

    // 2 + 3. stop accepting, then let accepted work finish
    if err := app.server.Shutdown(ctx); err != nil {
        return fmt.Errorf("http drain: %w", err)
    }
    if err := app.workers.Drain(ctx); err != nil {
        return fmt.Errorf("worker drain: %w", err)
    }

    // 4. release, innermost last
    app.lease.Release(ctx)
    app.db.Close()
    app.telemetry.Flush(ctx)

    return nil
}

Steps 2 and 3 are one call for the HTTP server, because Shutdown does both — it closes listeners and then waits. §15.4 is about what that call does and does not include, starting with the half-step this listing leaves out: keep-alives should be retired during the propagation wait, not at Shutdown time (§15.4.2). The worker drain is separate because a worker pool’s intake is a channel, not a listener, and §15.5 covers the distinction. And the release order is the reverse of the order those things were created, with one deliberate exception: the lease goes first, because §15.2.6 established that everything after stop() can be cut off, and a held lease is the most expensive thing to lose.

15.3.6 Common Mistakes

Summary: The Sequence

Five steps: fail readiness, wait out propagation, stop accepting, drain, release and report. Four of them are §15.1.3's reversal rule applied mechanically. The first is not, and it is the one that gets skipped.

It gets skipped because the instinct is that traffic has already stopped by the time the signal arrives. It has not. Deregistration and SIGTERM are concurrent, and the signal usually arrives first, so there is a window in which the load balancer is still routing to a process that has decided to leave.

The fix is a sleep, it is required rather than a workaround, and it should be commented with the readiness period and proxy sync interval it is derived from so that the next reviewer does not delete it.

Self-Check Questions: The Sequence

15.4 Stopping the HTTP Server

Server.Shutdown(ctx) is the most-used shutdown API in Go and the most misunderstood. It has four holes. Every one is documented, every one is measurable, and every one produces a shutdown that looks correct in code review and loses data in production.

15.4.1 ErrServerClosed Returns Immediately

The first hole is the classic bug, and the standard library’s documentation calls it out by name:

// Illustrative snippet — not a complete program
// ✗ BROKEN: main exits while Shutdown is still draining
go func() { <-ctx.Done(); srv.Shutdown(shutdownCtx) }()
if err := srv.ListenAndServe(); err != nil {
    log.Fatal(err) // fires on a *successful* shutdown
}

Two bugs in three lines. log.Fatal calls os.Exit(1), killing every in-flight request and skipping every deferred function (§15.1.2) — and it fires on the success path, because ErrServerClosed is exactly what a correct shutdown returns. The correct shape inverts which value you wait on:

// Illustrative snippet — not a complete program
errCh := make(chan error, 1)
go func() {
    err := srv.ListenAndServe()
    if !errors.Is(err, http.ErrServerClosed) {
        errCh <- err // a real listener failure
    }
}()

select {
case err := <-errCh:
    return err
case <-ctx.Done():
}
stop()
return srv.Shutdown(shutdownCtx) // wait for THIS one

ErrServerClosed is filtered out because it is not a failure; anything else on that channel is. And the value the function returns is Shutdown's, which is the one that means draining finished.

15.4.2 Retiring Keep-Alive Connections

There is a phase between “stop advertising” and “close listeners” that Shutdown cannot help with, because by the time you call it the decision has already been made.

HTTP keep-alive means a client that finished a request holds the TCP connection open and reuses it. That is what makes a connection pool work, and it is also why a client that has already connected to you keeps sending requests down that connection long after the load balancer stopped choosing you for new ones. Failing readiness does nothing about it: endpoint removal governs which address gets dialled next, not connections already established.

Server.SetKeepAlivesEnabled(false) is the lever. It makes the server mark responses so each client retires its pooled connection after the response it is currently receiving.

Measured, one client with a warm pooled connection, second request after the call:

resp.Close is net/http's normalised view of the close signal, which is why the raw Connection header reads empty on the client side. The effect is that the second client will not reuse the connection, so its next request goes through a fresh dial — which the load balancer sends elsewhere, because you already failed readiness.

It belongs in phase 1, alongside the propagation wait, for exactly that reason:

// Illustrative snippet — not a complete program
app.ready.Store(false)             // stop advertising
app.srv.SetKeepAlivesEnabled(false) // retire pooled connections

select {                            // let both take effect
case <-time.After(propagationDelay):
case <-hardCtx.Done():
}

Two seconds of propagation wait with keep-alives still enabled is two seconds of a warm client pipelining more work into a process that is leaving. With them disabled, the same two seconds drains the pool.

15.4.3 Shutdown Does Not Cancel In-Flight Requests

The second hole is the one that surprises people who have read the first.

The ctx you pass to Shutdown bounds how long you are willing to wait. It does not bound how long the handler runs, and it is not delivered to the handler at all.

Measured, a 600 ms handler against a 200 ms shutdown budget:

Shutdown gave up. The handler was never told anything happened — its request context’s Err() was still nil when it finished, 400 ms later. If main returns on that context deadline exceeded, the process exits with a handler mid-write.

SHUTDOWN'S CLOCK               THE HANDLER'S CLOCK
  the ctx you pass                 r.Context()
    │                                 │
    │ 200 ms budget                   │ still running
    ▼                                 │
  gives up, returns                   │
  context deadline exceeded           │
                                      ▼
                                finishes at 600 ms
                                Err() was nil throughout

Nothing in net/http joins these two. BaseContext
is the only wire you can run between them.

This is worth sitting with, because it inverts the usual mental model. Passing a context to a function in Go normally means “stop when this is done”. Here it means “stop waiting when this is done”, and the work continues unattended.

15.4.4 BaseContext, and the Rule That Makes It Safe

The fix is Server.BaseContext, a hook that supplies the context every request context descends from:

// Illustrative snippet — not a complete program
base, cancelBase := context.WithCancelCause(context.Background())

srv := &http.Server{
    Handler:     h,
    BaseContext: func(net.Listener) context.Context { return base },
}

Now r.Context() in every handler is a child of base, so cancelling base reaches all of them. Measured, the same 600 ms handler:

That is Chapter 13's context tree doing exactly what it was built for, one level higher than usual: the root is not a request, it is the process.

And here is the rule, which matters more than the mechanism. Cancel base alongside Shutdown and you have not built a graceful shutdown — you have built an abrupt one with extra steps. Every in-flight handler aborts mid-response.

Measured, base context cancelled at the same moment as Shutdown, with a handler that writes part of its response and then stops when cancelled:

The first two rows are the ones to be frightened of. A handler that returns cleanly when its context is cancelled produces a well-formed HTTP response — net/http computes a Content-Length for what was actually written, or terminates the chunk stream correctly — so the client reads a 200 OK to completion with no error of any kind and stores a truncated result. Nothing on the client side can tell.

handler returns on ctx.Done()
    │
    ├─ net/http sees an ordinary return
    ├─ frames what was written: Content-Length,
    │  or a correctly terminated chunk stream
    └─ sends a complete, well-formed message
            │
            ▼
      client: 200 OK, short body, no error
              nothing on that side can tell

handler panics with http.ErrAbortHandler
    │
    └─ connection broken mid-message
            │
            ▼
      client: unexpected EOF -- at least it is true

Handling cancellation correctly is what makes
the truncation invisible.

That is the sting: the polite thing a handler can do on cancellation is exactly what erases the evidence. http.ErrAbortHandler is the sanctioned way to say “this response is not finished” — panicking with it aborts the connection without logging a stack trace, and the client gets an error it can act on rather than a lie it cannot detect.

Either way it is §14.1's thesis — the failure mode is silence, not a crash — arriving at the process boundary. And it establishes the ordering:

Shutdown(ctx) ─────► budget expires ─────► cancelBase(cause)
      │                                          │
 handlers that                          handlers that did not
 finish in time                         are told to stop, and
 are undisturbed                        can return 503

Fire the second alongside the first and you cut
short every handler that was going to finish.

§15.7.2 puts this on a full escalation ladder. The point here is that BaseContext is a footgun without it: the natural reading of “cancel the context to shut down” is precisely the thing that truncates responses.

15.4.5 Hijacked Connections Are Never Waited For

The third hole is documented and absolute:

RegisterOnShutdown exists for this, and its own documentation limits what it does:

// Illustrative snippet — not a complete program
srv.RegisterOnShutdown(func() {
    close(wsQuit) // notify -- does not wait
})

The callbacks are launched in their own goroutines and nothing waits for them. So the pattern is: use the hook to notify, and track the connections yourself with a WaitGroup you can actually wait on.

// Illustrative snippet — not a complete program
var wsConns sync.WaitGroup

// in the handler, before hijacking
wsConns.Add(1)
defer wsConns.Done()

// in the shutdown path, after Shutdown returns
close(wsQuit)
waitCtx, cancel := context.WithTimeout(shutdownCtx, 5*time.Second)
defer cancel()
waitFor(waitCtx, &wsConns) // now you have actually waited

If your service has long-lived connections — WebSockets, SSE, gRPC streams — this is not an edge case. Shutdown returning nil means nothing about them.

15.4.6 The Chained Cause

Three chapters converge here, and the payoff is one line in a handler.

Wire the base context with WithCancelCause rather than plain WithCancel, and pass the shutdown cause through:

// Illustrative snippet — not a complete program
// WithoutCancel: keep the values middleware put above the
// server, but do not inherit the signal's cancellation.
base, cancelBase := context.WithCancelCause(
    context.WithoutCancel(sigCtx))
srv := &http.Server{Handler: h,
    BaseContext: func(net.Listener) context.Context { return base }}

// later, when the budget expires:
cancelBase(fmt.Errorf("drain budget expired: %w",
    context.Cause(sigCtx)))

Now a handler can find out not just that it is being cut short, but why:

// Illustrative snippet — not a complete program
select {
case <-r.Context().Done():
    slog.Warn("request abandoned",
        "cause", context.Cause(r.Context()),
        "trace", r.Context().Value(traceKey))
    return
case result := <-work:
    respond(w, result)
}

Measured, the full chain from a real signal:

The signal named itself in §15.2.4, NotifyContext put that name in a cause, and the shutdown path copied that cause into the base context’s own — it does not travel there by itself. That distinction is the whole of the wiring: base is deliberately not cancellable from sigCtx, or every handler would die the instant the signal landed, so the only way the reason reaches a request is that you carried it. Chapter 13's WithCancelCause, Chapter 14's error attribution and this chapter’s shutdown, in one line of handler code.

15.4.7 A Timed-Out Shutdown Closes Nothing

The fourth hole follows from §15.4.3 but is worth stating separately, because it changes what your code has to do next.

When the context expires, Shutdown returns ctx.Err() and stops waiting. It does not close the remaining connections and it does not stop the handlers. Everything is exactly as it was, minus your attention.

Server.Close() is the hard stop: it closes all listeners and all connections immediately, active ones included. So the complete pattern is two-phase:

// Illustrative snippet — not a complete program
if err := srv.Shutdown(shutdownCtx); err != nil {
    slog.Warn("graceful drain did not finish", "err", err)
    cancelBase(errShutdownExpired) // step 2 of the ladder
    if cerr := srv.Close(); cerr != nil {   // step 3
        slog.Error("force close failed", "err", cerr)
    }
}

Without the Close, a Shutdown that timed out leaves you with open connections and a process that will be SIGKILLed holding them.

15.4.8 Shutdown Polls

One mechanical detail, because it has a cost that shows up in §15.7's budget arithmetic. Shutdown does not wait on a condition variable; it polls for quiescence with a backoff, and the standard library is candid about it:

// Illustrative snippet — not a complete program
// Ideally we could find a solution that doesn't involve polling,
// but which also doesn't have a high runtime cost (and doesn't
// involve any contentious mutexes), but that is left as an
// exercise for the reader.
const shutdownPollIntervalMax = 500 * time.Millisecond

The interval starts at 1 ms, doubles each round, and clamps at 500 ms with jitter. The consequence — that a long drain adds dead time after the last handler has already finished — is measured in §15.7.3.

15.4.9 Common Mistakes

Summary: Stopping the HTTP Server

Server.Shutdown has four holes, all documented and all measurable.

Serve returns ErrServerClosed at the instant Shutdown begins, hundreds of milliseconds before draining finishes — so code that waits on ListenAndServe and calls log.Fatal exits mid-drain, on the success path.

Shutdown's context bounds how long you wait, not how long handlers run.

Hijacked connections are outside all of it; Shutdown returned nil in ~0 ms with one still live, and RegisterOnShutdown notifies without waiting.

And a timed-out Shutdown closes nothing, so Close() is the required next rung.

The reward for wiring the base context with WithCancelCause is that a handler can log which signal is cutting it short — Chapter 13's cause, Chapter 14's attribution and this chapter’s shutdown meeting in one line.

Self-Check Questions: Stopping the HTTP Server

What is wrong with this, and which of §15.1.1's four promises does it break?

// Illustrative snippet — not a complete program
go func() { <-sig; srv.Shutdown(context.Background()) }()
if err := srv.ListenAndServe(); err != nil {
    log.Fatal(err)
}

15.5 Draining Workers and Queues

An HTTP server has one intake and Shutdown closes it for you. A worker pool has an intake you built, a backlog you chose to hold, and work already in progress — three different things, and “drain the pool” means a different thing for each.

15.5.1 Three Things Called Draining

producers ──► [ jobs channel ] ──► workers ──► done
                the BACKLOG          IN FLIGHT

1  close the intake     stop accepting new jobs
2  drain the backlog    process what is queued
3  finish in flight     let running jobs complete

Always do 1 and 3. Whether you do 2 is a policy
decision, and for a durable queue the answer is
often no.

Step 1 and step 3 are non-negotiable: you must stop taking new work, and you must let work already started finish or be abandoned deliberately. Step 2 is where designs differ, and conflating it with step 3 is the mistake this section exists to prevent.

15.5.2 Closing the Intake

For a channel-fed pool the intake is the channel, and closing it is what tells the workers to stop:

// Illustrative snippet — not a complete program
type Pool struct {
    jobs chan Job
    wg   sync.WaitGroup
}

func (p *Pool) Start(ctx context.Context, n int) {
    for range n {
        p.wg.Go(func() {
            for job := range p.jobs { // ends when jobs is closed
                p.process(ctx, job)
            }
        })
    }
}

func (p *Pool) Drain(ctx context.Context) error {
    close(p.jobs) // step 1: no more work in
    return waitFor(ctx, &p.wg) // steps 2 and 3: finish what is there
}

for range p.jobs is doing double duty, and it is worth being explicit about why. A closed channel with buffered values still yields those values before the range ends — so this loop drains the backlog and then exits. One construct, both steps, which is convenient when draining the backlog is what you want and misleading when it is not.

The waitFor helper is the one piece the standard library does not give you: WaitGroup.Wait has no context-aware form.

// Illustrative snippet — not a complete program
func waitFor(ctx context.Context, wg *sync.WaitGroup) error {
    done := make(chan struct{})
    // close never blocks, so this goroutine cannot leak on a timeout
    go func() { wg.Wait(); close(done) }()
    select {
    case <-done:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    }
}

This is §14.2.4's closer goroutine with a select on the end. The goroutine outlives a timeout — it is parked in wg.Wait() until the workers actually finish — but it holds only a stack and a channel, and it cannot block anything because close never blocks.

15.5.3 In-Flight Is Not Queued

The distinction that matters most, and the one the phrase “drain in-flight operations” hides.

A job in flight has been pulled from the queue and is executing. If you drop it, its work is half-done — a partial write, a charged card with no receipt, a lease taken and not released. Nobody else will pick it up, because as far as the queue is concerned it was delivered.

A job in the backlog has not started. If you drop it, nothing has happened yet. Whether it is lost depends entirely on where the backlog lives.

That third row is the important one, and it is why the answer to “should I drain the backlog?” is genuinely it depends.

15.5.4 The Add-After-Close Race

Closing the intake is what stops the workers, and it is also the most dangerous line in a shutdown, because close cannot be guarded by a select:

// Illustrative snippet — not a complete program
// ✗ BROKEN: Submit can be mid-send when Close fires
func (p *Pool) Submit(j Job) { p.jobs <- j }
func (p *Pool) Close()       { close(p.jobs) }

An atomic flag does not fix it. if closed.Load() { return } followed by p.jobs <- j leaves a window between the check and the send, and the window is exactly where the panic lives — it makes the crash rarer, which mostly means it now happens in production rather than in tests.

The fix is mutual exclusion that spans the send:

// Illustrative snippet — not a complete program
type Pool struct {
    jobs   chan Job
    wg     sync.WaitGroup // as in §15.5.2
    mu     sync.RWMutex
    closed bool
}

func (p *Pool) Submit(j Job) error {
    p.mu.RLock()
    defer p.mu.RUnlock()
    if p.closed {
        return ErrShuttingDown
    }
    p.jobs <- j // safe: Close cannot proceed while we hold RLock
    return nil
}

func (p *Pool) Close() {
    p.mu.Lock()
    defer p.mu.Unlock()
    p.closed = true
    close(p.jobs)
}

The RWMutex is doing real work. Submit holds the read lock across the send, so Close cannot take the write lock — and therefore cannot call close — while any send is in progress. Many Submit calls still proceed concurrently, which is why it is an RWMutex and not a Mutex; the close is simply ordered strictly after all of them.

One consequence: Submit can block holding a read lock, stalling Close behind a full channel. Give it an escape if that matters:

// Illustrative snippet — not a complete program
select {
case p.jobs <- j:
    return nil
case <-ctx.Done():
    return ctx.Err()
}

15.5.5 When Abandoning the Backlog Is Correct

With at-least-once delivery and an ack-after-processing discipline, an unstarted job that you never ack goes back to the queue and is redelivered — to a replica that is not shutting down and has a full budget rather than the seconds you have left.

Draining that backlog is therefore not just unnecessary, it is worse: you spend your scarce shutdown budget doing work another instance would have done with time to spare, and you risk being SIGKILLed halfway through a job that would otherwise have been redelivered cleanly.

// Illustrative snippet — not a complete program
func (p *Pool) Drain(ctx context.Context) error {
    close(p.jobs)         // 1. stop pulling new work

    // 2. abandon the backlog -- unacked jobs will be redelivered
    drained := 0
    for range p.jobs {
        drained++
    }
    if drained > 0 {
        slog.Warn("abandoned queued jobs for redelivery",
            "count", drained)
        abandonedJobs.Add(int64(drained))
    }

    // 3. always wait for what is already running
    return waitFor(ctx, &p.wg)
}

Note what is not skipped: step 3. In-flight work is finished regardless, because that is the work with no other copy.

Note also that abandoning is reported. §15.1.1's fourth promise applies here specifically — a shutdown that silently drops forty queued jobs and one that drops none look identical from the outside unless you count them.

15.5.6 Cleanup That Must Outlive the Cancellation

Shutdown produces a situation Chapter 13 anticipated: the context is cancelled, and you still have work to do that needs a context.

A worker that has been told to stop still has to record what it did — flush a metric, write a checkpoint, release a lease. Every one of those is an outbound call that wants a context, and the only context in scope is cancelled.

// Illustrative snippet — not a complete program
// ✗ BROKEN: the checkpoint write is cancelled before it starts
func (w *Worker) run(ctx context.Context) {
    <-ctx.Done()
    w.store.Checkpoint(ctx, w.progress) // fails immediately
}

context.WithoutCancel (Go 1.21) is the answer, and §13.3.5 promised this chapter would use it:

// Illustrative snippet — not a complete program
// ✓ detached from the cancellation, with a bound of its own
func (w *Worker) run(ctx context.Context) {
    <-ctx.Done()

    cleanupCtx, cancel := context.WithTimeout(
        context.WithoutCancel(ctx), 3*time.Second)
    defer cancel()

    if err := w.store.Checkpoint(cleanupCtx, w.progress); err != nil {
        slog.Error("checkpoint failed", "err", err)
    }
}

Measured, the three states that matter:

The middle row is the fix and the bottom row is why it is only half a fix. Detaching without re-bounding trades cleanup that never runs for cleanup that never ends.

Two things carry over from the parent that you want: its values, so a trace ID or request ID still propagates into the cleanup call. And nothing else — which is the point.

15.5.7 Pipelines Run in Reverse

A Chapter 7 pipeline shuts down the way §14.5.3 unwinds one, with the trigger moved from an error to a signal.

generator ──► stage A ──► stage B ──► consumer
    │            │           │           │
 close its    range ends   range ends  range ends
 output       defer close  defer close
    │            │           │           │
    └────────────┴───────────┴───────────┘
            closure cascades downstream

Stop the generator. Everything after it drains
in order and stops on its own. Cancel the context
instead and you abandon whatever was in flight.

Stopping the generator is the graceful form: it closes its output, each stage’s for range ends, each stage’s defer close(out) fires, and the values already in the pipeline flow through to the consumer. Nothing is dropped.

Cancelling the context is the forceful form, and it is the escalation — the same ladder as §15.4.4. It frees stages blocked on a send, which is what you need when a downstream stage has stopped reading, but everything in flight is abandoned.

The rule from §14.5 still holds and matters more here: every send needs a select on ctx.Done(), or a stage blocked on a send during shutdown never returns and your waitFor burns the whole budget waiting for it.

15.5.8 Common Mistakes

Summary: Draining Workers and Queues

“Drain” names three separate things: closing the intake, processing the backlog, and finishing work already in flight. The first and third are obligatory. The second is a policy decision, and conflating it with the third is the mistake.

In-flight work has no other copy — abandon it and its effects are half-applied. A backlog entry may or may not have another copy, and with at-least-once delivery and ack-after-processing it does: an unacked job goes back to the queue and is redelivered to an instance with a full budget instead of your remaining seconds. In that case draining the backlog is actively worse than abandoning it, and abandoning it must be counted and reported.

Cleanup at shutdown needs a context, and the one in scope is cancelled. context.WithoutCancel detaches from the cancellation while keeping values, and it must be paired with a fresh WithTimeout — detaching removes the deadline too, so an unbounded detached context blocks until SIGKILL.

Pipelines shut down by stopping the generator and letting closure cascade. Cancelling the context is the escalation, not the mechanism.

Self-Check Questions: Draining Workers and Queues

Why does this fail, and what does the fix have to include beyond WithoutCancel?

// Illustrative snippet — not a complete program
<-ctx.Done()
w.store.Checkpoint(ctx, w.progress)

// Illustrative snippet — not a complete program
cleanupCtx, cancel := context.WithTimeout(
    context.WithoutCancel(ctx), 3*time.Second)
defer cancel()

15.6 Coordinating Components

A real service is a handful of long-lived things — an HTTP server, a worker pool, a metrics exporter, a queue consumer — that start together, run together, and have to stop in an order determined by what depends on what. Chapter 14's errgroup is the natural tool, and it behaves differently here than it did there in one specific way that costs an entire shutdown budget.

15.6.1 Start Order Determines Stop Order

The dependency graph is not a matter of taste; it is fixed by construction, and §15.1.3's rule reads it directly.

started 1st   config                    closed 4th
started 2nd    └─ database pool         closed 3rd
started 3rd        └─ worker pool       closed 2nd
started 4th            └─ HTTP server   closed 1st

Each is nested inside the one above. Close from
the inside out: the HTTP server stops first
because everything else outlives a request, and
config closes last because everything read it.

The one deliberate departure, from §15.2.6: a distributed lease is released early rather than in its dependency position, because everything after stop() can be cut off by a second signal and a lease held to its TTL is the most expensive thing to be holding when that happens. Order by dependency, then promote whatever is most costly to lose.

That promotion is a trade, not a free win, and it is worth naming: releasing a partition lease while workers still hold in-flight work for that partition lets a second owner start before the first has finished. Where the work is not idempotent, promote the release only as far as after the worker drain and before the pool close — still early enough to matter, late enough to keep the exclusion the lease was for.

15.6.2 The errgroup Return Value Means Something Else Here

Chapter 14 established the return-value decision: return err for fail-fast, return nil for collect-all (§14.4.5). For a batch of finite tasks that is the whole story.

For a set of long-running components it is not, and the difference is easy to miss because the code looks identical.

// Illustrative snippet — not a complete program
g, ctx := errgroup.WithContext(ctx)

g.Go(func() error { return runServer(ctx) })   // runs for hours
g.Go(func() error { return runWorkers(ctx) })  // runs for hours
g.Go(func() error { return runExporter(ctx) }) // runs for hours

return g.Wait()

The intent is: if any component stops, stop them all. The derived context should cancel and everyone should unwind.

That is not what return nil does. Measured, two components where the first exits at 80 ms and the second watches ctx.Done():

errgroup only cancels the derived context when a closure returns a non-nil error (§14.4.8 — if err := f(); err != nil). A clean nil return means “I finished successfully”, and for a batch task that is exactly right. For a component that was supposed to run until told otherwise, it means “I am done, and everyone else keeps running forever.”

The exporter exits cleanly because its endpoint was closed. Nothing cancels. The server and the workers run on, g.Wait() blocks, and your shutdown budget expires waiting for components that were never told to stop.

15.6.3 Sentinel Errors for Component Exit

The fix is that in a lifecycle group, stopping is an event worth reporting, so every component returns a non-nil value when it stops:

// Illustrative snippet — not a complete program
var errComponentStopped = errors.New("component stopped")

g.Go(func() error {
    if err := runExporter(ctx); err != nil {
        return fmt.Errorf("exporter: %w", err)
    }
    return fmt.Errorf("exporter: %w", errComponentStopped)
})

Now an exporter that exits for any reason cancels the derived context, every sibling sees ctx.Done(), and g.Wait() returns promptly with something that names what happened.

The caller then distinguishes the expected stop from a real failure:

// Illustrative snippet — not a complete program
err := g.Wait()
switch {
case err == nil:
    // unreachable in a lifecycle group -- see below
case errors.Is(err, errComponentStopped):
    slog.Info("component stopped, shutting down", "err", err)
case errors.Is(err, context.Canceled):
    slog.Info("shutdown signalled")
default:
    slog.Error("component failed", "err", err)
    exitCode = 1
}

That first case is worth a comment in real code, because it documents an invariant: in a lifecycle group Wait returning nil means every component returned nil, which means every component stopped without telling anyone, which is the bug this section is about.

15.6.4 The ErrServerClosed Trap

The component most likely to break this rule is the one from §15.4, because the natural way to write it is exactly wrong:

// Illustrative snippet — not a complete program
// ✗ BROKEN: maps the stop signal to "nothing happened"
g.Go(func() error {
    err := srv.ListenAndServe()
    if errors.Is(err, http.ErrServerClosed) {
        return nil // <-- siblings now run forever
    }
    return err
})

§15.4.1 established that ErrServerClosed is not a failure, so filtering it out looks right — and in the select-based shape of §15.4.1 it is right, because the signal context is what drives shutdown there.

Inside a lifecycle group it is wrong. The server has stopped; that is the event the group exists to propagate. Map it to a sentinel instead:

// Illustrative snippet — not a complete program
// ✓ the server stopping is news, and the group should hear it
g.Go(func() error {
    err := srv.ListenAndServe()
    if errors.Is(err, http.ErrServerClosed) {
        return fmt.Errorf("http server: %w", errComponentStopped)
    }
    return fmt.Errorf("http server: %w", err)
})

15.6.5 The Full Shape

Components run in a group; the group’s context is the shutdown signal; teardown is ordered and happens after Wait returns.

// Illustrative snippet — not a complete program
func Run(ctx context.Context, app *App) error {
    ctx, stop := signal.NotifyContext(ctx,
        syscall.SIGINT, syscall.SIGTERM)
    defer stop() // §15.2.5 -- for the paths with no signal at all

    // §15.2.5 again, and this is the load-bearing half: hand the
    // signal back the moment it lands, not when the drain ends.
    // The drain is exactly when an operator reaches for Ctrl-C.
    go func() { <-ctx.Done(); stop() }()

    g, gctx := errgroup.WithContext(ctx)

    g.Go(func() error { return app.runServer(gctx) })
    g.Go(func() error { return app.runWorkers(gctx) })
    g.Go(func() error { return app.runExporter(gctx) })

    // Each component watches gctx and returns when it is done;
    // whichever stops first cancels the rest.
    runErr := g.Wait()

    // Teardown gets a budget of its own, detached from the
    // cancellation that got us here (§15.2.7).
    tctx, cancel := context.WithTimeout(
        context.WithoutCancel(ctx), app.cfg.ShutdownBudget)
    defer cancel()

    return errors.Join(runErr, app.teardown(tctx))
}

errors.Join (§14.2.3) is doing real work in that last line: a component may have failed and teardown may have timed out, and those are two different facts an operator needs. Returning only one of them is the same silent-loss problem §15.1.1's fourth promise names.

The teardown itself is a plain ordered sequence, not a group — because ordering is the entire requirement:

// Illustrative snippet — not a complete program
func (a *App) teardown(ctx context.Context) error {
    var errs []error
    // Innermost first, plus the lease promoted to the front.
    if err := a.lease.Release(ctx); err != nil {
        errs = append(errs, fmt.Errorf("lease: %w", err))
    }
    if err := a.workers.Drain(ctx); err != nil {
        errs = append(errs, fmt.Errorf("workers: %w", err))
    }
    if err := a.db.Close(); err != nil {
        errs = append(errs, fmt.Errorf("db: %w", err))
    }
    if err := a.telemetry.Flush(ctx); err != nil {
        errs = append(errs, fmt.Errorf("telemetry: %w", err))
    }
    return errors.Join(errs...)
}

Each step runs even if an earlier one failed, and every failure is collected. A database that will not close is not a reason to skip flushing telemetry — especially since that telemetry is how anyone will find out about the database.

15.6.6 When a WaitGroup Beats a Group

errgroup is the right tool when components can fail and one failure should stop the rest. It is the wrong tool when they cannot fail and you only need to know they are finished.

Waiting for hijacked WebSocket connections (§15.4.5) is that case: there is no error to collect, no cancellation to propagate, just a count reaching zero. A WaitGroup and the waitFor helper from §15.5.2 say that directly, and reaching for a group would add an error channel nobody reads.

The distinction is the same one §14.5.5 draws when it runs both at once: errgroup owns errors and cancellation across the stages; the WaitGroup tracks exactly the goroutines that send on one channel. errgroup manages lifecycles with failure; a WaitGroup counts.

15.6.7 Common Mistakes

Summary: Coordinating Components

Components stop in the reverse of the order they started, because that is what the dependency graph says — with one deliberate exception, the lease, promoted to the front because everything after stop() can be cut off.

errgroup is the right coordinator, and it means something different here than in Chapter 14.

Every component therefore returns a sentinel when it stops — including the HTTP server, whose ErrServerClosed is exactly the value you would naturally filter to nil.

Teardown happens after Wait, as an explicit ordered sequence rather than inside the group, on a budget detached from the cancellation that triggered it. Every step runs even if an earlier one failed, and the errors are joined, because the telemetry flush is how anyone learns about the database close that failed.

Self-Check Questions: Coordinating Components

Why is this wrong inside a lifecycle group when §15.4.1 said ErrServerClosed is not a failure?

// Illustrative snippet — not a complete program
g.Go(func() error {
    err := srv.ListenAndServe()
    if errors.Is(err, http.ErrServerClosed) {
        return nil
    }
    return err
})

// Illustrative snippet — not a complete program
return fmt.Errorf("http server: %w", errComponentStopped)

15.7 When the Budget Expires

Everything so far assumed the budget was enough. Sometimes it is not: a connection will not close, a queue will not drain, a downstream call hangs. This section is about dividing the budget deliberately, escalating when a phase overruns, and making sure that what you abandoned is visible afterwards — §14.7's degradation lens applied to exit.

15.7.1 Dividing the Budget

The platform gives you one number (§15.2.7). Four phases have to fit inside it, and if you do not divide it explicitly the first phase takes all of it.

0s        5s                      25s      28s   30s
┌─────────┬───────────────────────┬─────────┬─────┐
│ propa-  │       drain           │ release │ head│ SIGKILL
│ gation  │                       │         │room │
│  §15.3  │    §15.4, §15.5       │  §15.6  │     │

Every phase gets its own deadline carved from the
total. Headroom at the end is for reporting -- a
shutdown that is killed before it can log why it
failed has broken the fourth promise.

In code that is one budget and a series of sub-deadlines:

// Illustrative snippet — not a complete program
const grace = 30 * time.Second // must match the platform's setting

total, cancel := context.WithTimeout(
    context.WithoutCancel(sigCtx), grace-2*time.Second) // headroom
defer cancel()

phase := func(d time.Duration) (context.Context, context.CancelFunc) {
    return context.WithTimeout(total, d)
}

Each phase then takes phase(5*time.Second) or whatever its share is, and each one is bounded twice — by its own share and by the total. A phase that finishes early gives its unused time back to the phases after it, because they are bounded by total as well.

15.7.2 The Escalation Ladder

When a phase overruns, you escalate rather than give up. Each rung is more forceful and loses more.

1  Shutdown(ctx)        wait politely      loses nothing
      │ budget expires
2  cancelBase(cause)    tell handlers      loses in-flight
      │ still not gone     to stop           responses
3  srv.Close()          drop connections   loses the
      │ still not gone                       connections
4  report + exit(1)     say what was lost  loses nothing
                                            more

Never skip a rung, and never fire rung 2 alongside
rung 1 -- §15.4.4 measured what that costs.

In code:

// Illustrative snippet — not a complete program
err := srv.Shutdown(drainCtx)
if err == nil {
    return nil // rung 1 was enough
}

slog.Warn("drain did not finish in budget", "err", err)
drainTimeouts.Inc()

// Rung 2: now, and only now, tell handlers to stop.
cancelBase(fmt.Errorf("shutdown budget expired: %w", err))

// Give them a moment to notice and unwind cleanly.
reap, cancel := context.WithTimeout(total, 2*time.Second)
defer cancel()
if err := srv.Shutdown(reap); err == nil {
    return nil
}

// Rung 3: drop what is left.
return srv.Close()

The second Shutdown after cancelling the base context is the rung most implementations skip, and it is nearly free. Handlers that check their context return within milliseconds of being told to, so this usually succeeds — and the difference between a handler returning early and a connection being dropped underneath it is the difference between a client seeing a 503 it can retry and a client seeing a reset it cannot interpret.

15.7.3 The Cost of the Poll

§15.4.8 noted that Shutdown polls with a backoff. The consequence belongs here, in the budget, because it is time you pay for nothing.

The interval starts at 1 ms and doubles to a 500 ms cap, and the backoff restarts on each Shutdown call. So the longer a drain runs, the coarser the polling gets — and the longer the gap between the last handler actually finishing and Shutdown noticing.

Measured, the lag between the handler returning and Shutdown returning, over eight runs each. Shutdown is called the moment the handler starts, which is what fixes how far the backoff has advanced by the time the handler ends — call it later and every row shrinks:

The short rows are tight and reproducible; the long ones swing by a factor of two because more doublings mean more accumulated jitter, and a run outside these bounds on a busier machine would not be surprising. Treat the trend as the finding and the individual numbers as approximate — the shape is what matters: a slow request makes your shutdown slower even after that request has finished.

At the top of the table this is a rounding error. For a service whose p99 request is several seconds, up to a fifth of a second of pure dead time per Shutdown call is worth knowing about when you are dividing a budget into phases — and it is an argument for not calling Shutdown more times than the ladder requires.

15.7.4 What to Abandon

When the budget will not stretch, the question is not whether to lose something but what. §15.5.3's table decides most of it:

The last two rows are why §15.6.5 puts the lease first and the telemetry flush last but unconditional. Everything else can be sacrificed to them.

15.7.5 When Shutdown Hangs

Every rung of the ladder assumes something is making progress. Sometimes nothing is, and the process sits there until the grace period expires and SIGKILL decides. A hung shutdown is a §10.2 deadlock with worse timing: it happens on deploys, under load, and rarely on a machine you can attach a debugger to.

Three causes account for most of them, and this chapter has already named all three:

A stage blocked on a send with nobody receiving. §14.5's rule at shutdown: a pipeline stage whose consumer has gone parks on out <- v forever unless the send is guarded by select on ctx.Done(). Shutdown returns; wg.Wait() never does.

A wg.Wait() for a goroutine that took an early exit. A worker that returned before its Done was registered, or a path that skips it, leaves the counter permanently above zero — §2.3's rule with a shutdown-shaped symptom.

A component waiting on one already released. §15.6's ordering, reversed: the consumer is draining and the pool it needs is closed, so the drain cannot finish.

The diagnostic is §10.4's, and it works on a running process with no instrumentation:

Two goroutines and the whole answer between them: main is in wg.Wait, and the thing it waits for is asleep in stuckDrain. In a real dump there will be hundreds, most of them runtime and GC workers you can skip. The ones that matter carry a blocking state in the brackets — sync.WaitGroup.Wait, sync.Mutex.Lock, chan send, chan receive, select, semacquire — and, for a module declaring go 1.27 or later, any runtime/pprof labels the goroutine carries, printed after the bracket as {request: "…"}, which is what makes Chapter 19’s per-request labels useful in a dump — which is the same reading §10.4 teaches, applied to a process that is trying to leave.

The structural fix is §15.7.1's: every phase bounded by a context derived from one deadline. A phase that cannot exceed its slice cannot hang the shutdown — it fails, gets logged, and the next phase starts. A hang means some phase has no bound, and finding which one is the same question as reading the dump.

15.7.6 Making the Loss Visible

A shutdown that abandoned work and exited zero is indistinguishable from one that abandoned nothing. §14.7.6's three signals apply unchanged, one for each audience:

// Illustrative snippet — not a complete program
type ShutdownReport struct {
    Graceful       bool
    Duration       time.Duration
    RequestsInFlight int // abandoned mid-request
    JobsAbandoned    int // backlog left for redelivery
    Errors         []error
}

func report(r ShutdownReport) int {
    slog.Info("shutdown complete",
        "graceful", r.Graceful,
        "duration", r.Duration,
        "requests_abandoned", r.RequestsInFlight,
        "jobs_abandoned", r.JobsAbandoned,
        "errors", errors.Join(r.Errors...))

    shutdownDuration.Observe(r.Duration.Seconds())
    requestsAbandoned.Add(float64(r.RequestsInFlight))

    if !r.Graceful {
        ungracefulShutdowns.Inc()
        return 1 // the exit code is a signal too
    }
    return 0
}

Three audiences, three signals: the exit code for the orchestrator and anything scripting a rollout, the structured log for whoever investigates, the metric for the dashboard that notices a trend nobody was investigating.

The exit code is the one most often left at zero. An orchestrator that sees a clean exit has no reason to slow a rollout down — so a deploy that abandons a few requests per pod looks perfectly healthy right up until it is doing it a thousand times a minute.

15.7.7 Proving It Works

Shutdown code runs rarely, is timing-dependent, and fails silently — which is the exact profile of code that is never tested. Two techniques make it testable, and Chapter 16 covers the general versions.

Count goroutines around the operation. The leak that shutdown produces is a goroutine that outlives the process’s intent, and counting before and after is enough to catch it deterministically. That is what this chapter’s exercise does in its fourth gate, and what Chapter 14's exercise did.

Use virtual time for deadline behaviour. testing/synctest went stable in Go 1.25 and makes a test with a 30-second budget run instantly and deterministically.

There is a limitation worth knowing before you build a test suite on it.

So synctest is the right tool for a shutdown sequencer expressed in channels and contexts, and the wrong tool for anything holding a socket. Since Go 1.27 that boundary has moved for HTTP specifically: httptest.NewTestServer(t, h) serves the handler over an in-memory network rather than a real socket, so a client–server round trip can run inside a bubble on virtual time; Chapter 16 (§16.4.8) covers it. This chapter’s exercise keeps a real net.Listener, so it still uses real short timeouts and goroutine counting. Chapter 16 takes the technique further.

15.7.8 Common Mistakes

Summary: When the Budget Expires

The platform gives one number and four phases must fit inside it. Divide it explicitly with a sub-deadline per phase, all bounded by the total so unused time flows forward, and leave headroom at the end — a shutdown killed before it can log why it failed has broken the fourth promise.

When a phase overruns, escalate rather than give up: wait politely, then tell handlers to stop, then drop the connections, then report. The rung most often skipped is the second Shutdown after cancelling the base context, and it is nearly free — handlers that check their context return in milliseconds, which is the difference between a client seeing a retryable 503 and an uninterpretable reset.

Shutdown polls, and the poll costs real time.

What to abandon is decided by whether the work has another copy. Unacked backlog goes first because it will be redelivered; a held lease and unflushed telemetry go last because losing them is more expensive than anything they were competing with.

And all of it has to be reported — exit code, log, metric — because a shutdown that quietly abandoned work is indistinguishable from one that did not.

Self-Check Questions: When the Budget Expires