Chapter 17: Rate Limiting and Flow Control

17.1 A Limiter Is a Queue

17.1.1 The Four You Already Own

Four mechanisms, one shape: a fixed number of permits, handed out and returned. Chapter 5 built it from a channel, Chapter 14 wrapped it in a method, and both were right for the problem in front of them.

What none of them can express is a bound on events per unit time.

17.1.2 Rate and Concurrency Are Different Units

A semaphore of ten says “ten at once” and is completely silent about whether that is ten per second or ten thousand. If the work is fast enough, a concurrency limit permits an unbounded rate.

The reverse fails just as badly. A rate limit of 100/s says nothing about how many requests are in flight, because that depends on how long each one takes — and how long each one takes belongs to your dependency, not to you.

Neither limit implies the other, and using one where the other was needed is the whole bug class this chapter is about. It produces a system that is provably compliant with a limit nobody cared about.

17.1.3 Little’s Law Decides the Third Number

The two units are related, and the relationship is one line:

Run it the other way and the failure is sharper. Bound the rate at 2,000/s, let latency go from 50 ms to 500 ms, and L climbs from 100 to 1,000. You bounded arrivals and the in-flight count grew tenfold, because W was never yours. That is the first listing in the cold open: perfect compliance, climbing memory.

Chapters 5 and 7 both asked how big a buffer should be and answered with heuristics. This is the arithmetic those heuristics were approximating.

17.1.4 The Four Dispositions

Once the work does not fit, there are exactly four things to do with the excess. They are independent choices, not points on a scale.

Wait. Hold the caller until a permit is free. Costs latency, preserves the work, and is the only disposition that requires the caller to have slack.

Reject. Refuse immediately and say so, in a way the caller can act on — a 429, a Retry-After, an error it can route elsewhere. Costs the work, preserves latency, and is the only disposition that hands the caller a decision.

Drop. Discard silently. Distinct from rejection because nobody is told, which is occasionally correct — a metrics sample, a cache warm, a frame of telemetry — and catastrophic everywhere else.

Degrade. Serve a cheaper answer: a stale cache, a partial result, the answer without the recommendations. §14.7 built the machinery; this is where you decide to use it under load rather than under failure.

Chapter 15 named four promises so a given bug could be attributed to one. These work the same way. “The service got slow” is not a diagnosis. “We chose to wait when we should have rejected, and the callers had no slack” is.

In code they are four different returns from one place, which is the point — the decision is made once, at admission, not scattered through the handler:

// Illustrative snippet — not a complete program
// One admission point, four dispositions. Which one you pick is a
// product decision; making it explicit is an engineering one.
func (s *Server) admit(
    ctx context.Context, r *Request,
) (*Reply, error) {
    switch {
    case s.lim.Allow():
        return s.serve(ctx, r)          // there was room

    case r.CanWait && ctx.Err() == nil:
        if err := s.lim.Wait(ctx); err != nil {
            return nil, err             // WAIT: caller had slack
        }
        return s.serve(ctx, r)

    case r.Sheddable:
        s.dropped.Add(1)                // DROP: nobody is told
        return nil, nil

    case s.cache.Has(r.Key):
        return s.cache.Stale(r.Key), nil // DEGRADE: cheaper answer

    default:
        return nil, ErrTooManyRequests   // REJECT: caller decides
    }
}

Notice that three of the four are one line. The expensive one is Wait, and it is expensive precisely because it is the only branch that keeps the caller.

17.1.5 Measured: When a Limiter Is Worth Nothing

Here is the measurement that should decide whether you install any of this, and it is the one most writing on the subject omits.

The model: a downstream with fixed concurrency and a fixed per-request cost, behind an unbounded intake queue. Both arms are given the same courtesy — a worker checks the request’s context before spending time on it and drops it for free if the caller has already gone — because withholding that from the unlimited arm would rig the comparison. The limited arm puts a semaphore in front, where waiting is free and cancellable and no downstream capacity is committed until a caller is through it.

The only thing that varies is whether the caller has slack: whether its total budget exceeds the deadline it puts on a single attempt.

Block [1] is six identical rows. When the caller has no slack, the limiter is inert. A client population whose total budget equals its per-attempt deadline is already self-limiting: it waits 250 ms, leaves, and leaving costs the server nothing beyond the work already in flight. Admission control changes nothing at all, because there was never anything to gain by making those callers wait — they had no waiting left to give.

Block [2] is where the limiter earns its place, and the mechanism is in the waste column. Without it, a fifth to a third of the downstream’s capacity goes to requests whose caller had already gone. With it, that number is zero, because nobody is admitted unless there is a slot free right now.

The 1-try rows are the headline: around 36 served becomes exactly 300, which is roughly 8×. But read the wall column before celebrating. The limiter did not make anything faster. It made the system finish: 600 ms rather than 80 ms, which is exactly 300 ÷ 500 per second — Little’s Law again, and the best possible time for that much work through that much capacity. What it converted was a fast failure into a slow success.

The 10-try rows are the more honest comparison, because retrying is what a real client does. Ten attempts without a limiter eventually push 259–273 of 300 through — but they burn 34–40% of the downstream doing it, take 800 ms rather than 600 ms, and still lose thirty or more callers. Retries are a way of rediscovering admission control badly, which §17.6 measures properly.

That gives the chapter its spine, and it is the sentence to hold for everything that follows:

One more thing is visible in that table, and it is not about throughput at all. Every limit=none row is a range; every limit=10 row is a single number, identical on all five runs. The unlimited arm’s outcome depends on which of three hundred goroutines reaches a free slot first, so it moves. The limited arm’s outcome is decided at a serialised admission point and by Little’s Law — 300 requests ÷ 500 per second is 600 ms, every time.

That is worth naming, because it generalises past the measurement: admission control makes a system reproducible as well as efficient. The unlimited arm is unpredictable for exactly the reason it is wasteful — nothing decides the order in which work is admitted, so nothing decides which callers win. A service without a bound does not merely perform worse; it performs differently on every run, which is what makes overload incidents so hard to reproduce afterwards.

17.1.6 Where the Queue Goes When You Do Not Build One

The corollary matters as much as the theorem. Not installing a limiter does not mean there is no queue. It means the queue is somewhere you did not choose:

• The kernel’s accept backlog, sized by somaxconn and net.ListenConfig
• Socket receive buffers, sized by the operating system
• The load balancer’s own queue, sized by someone on another team
• The client’s retry loop, sized by whoever wrote the client

Every one of those has a capacity, a drain rate and a latency consequence you did not pick, and none of them checks a deadline or exports a metric. The choice is never queue or no queue. It is a queue you can see and bound, or one you cannot.

17.1.7 Common Mistakes

Summary: A Limiter Is a Queue

You already own four limiters, and all four bound concurrency. None bounds a rate, and the two are different units: a concurrency limit permits an unbounded rate when the work is fast enough, and a rate limit leaves the in-flight count free to grow as your dependency slows.

L = λW is the only bridge, and it runs in the direction that happens to you rather than the one you configure.

Whether a limiter helps at all is measurable and conditional. With no caller slack it was inert across six rows; with slack it was worth roughly 8×, and the price was named in the wall column — 80 ms became 600 ms, which is Little’s Law’s best possible time for that work through that capacity. Ten retries without a limiter eventually pushed 259–273 of 300 through while burning a third of the downstream, which is admission control rediscovered badly.

The four dispositions — wait, reject, drop, degrade — are independent choices, and naming them separately is what makes a bug describable.

And not building a queue does not remove it. It relocates it to the kernel, the load balancer, or the client’s retry loop, none of which checks a deadline or exports a metric.

Self-Check Questions: A Limiter Is a Queue

17.2 Token Buckets, and Why Not a Ticker

Chapter 11 asked you to find the bugs in a rate limiter, and you found them. This section begins by pointing out that the answer it gave you is still wrong.

17.2.1 The Debt Chapter 11 Left

The self-check in §11's atomic anti-patterns presents a fixed-window counter with three defects — a window-reset race, a check-then-act race, and an invariant spanning two values — and offers this as the fix:

// Illustrative snippet — not a complete program
func (r *RateLimiter) Allow() bool {
    r.mu.Lock()
    defer r.mu.Unlock()

    now := time.Now().Unix()
    if now > r.window {
        r.window = now
        r.requests = 0
    }
    if r.requests < r.limit {
        r.requests++
        return true
    }
    return false
}

Every claim Chapter 11 made about that code is true. The races are gone, -race is clean, the invariant holds. As an answer to “where are the data races?” it is complete.

As a rate limiter it is broken, in exactly the way a rate limiter must not be.

A fixed window resets on a wall-clock boundary. Nothing stops a caller spending its full allowance at t = 0.99 and its next full allowance at t = 1.01. Across those twenty milliseconds the downstream sees two hundred requests — twice the configured rate, delivered in a fiftieth of the configured interval, which is precisely the burst the limiter was installed to prevent.

That distinction is why this chapter exists as something other than a tour of libraries. Chapters 8 through 12 taught you to make concurrent code agree. Agreement is not correctness, and a limiter is the cleanest case in the book where you can have one without the other.

17.2.2 The Token Bucket

The fixed window’s problem is that it forgets everything at the boundary. A token bucket never forgets; it accrues.

refill: r tokens per second, continuously
capacity: b tokens (the burst)

    ┌ tokens ┐
    │ ● ● ●  │      b = 5,  r = 2/s
    │ ● ●    │
    └────────┘
        │
        │  take 1 per request; empty means wait or refuse
        ▼

Idle for 3s at r=2/s: +6 tokens, capped at b. The cap is
the whole design — history is forgiven only up to b, and
never any further than that.

Two parameters answering different questions. Rate is the sustained throughput you permit. Burst is how much unused allowance you remember — not slack, not a safety margin, but a bounded quantity of forgiven history.

The fixed window is this same idea with an unbounded burst at every boundary, which is why 2× is its floor rather than its ceiling.

17.2.3 Measured: A Ticker Degrades to the Receiver’s Rate

time.Ticker looks purpose-built for this. It delivers a value at a fixed interval; a limiter delivers a permit at a fixed interval.

// Illustrative snippet — not a complete program
tk := time.NewTicker(10 * time.Millisecond) // "100 per second"
defer tk.Stop()
for req := range requests {
    <-tk.C
    handle(req)
}

The channel is unbuffered — it has been since the Go 1.23 timer rewrite, and since Go 1.27 there is no go.mod or GODEBUG setting that brings the old one-element buffer back — and the documentation is explicit: the ticker “will adjust the time interval or drop ticks to make up for slow receivers.” If nobody is receiving when a tick fires, the tick is discarded rather than queued.

Fifty were due; ten to fifteen arrived across twenty-four runs. Derived: ten is the floor and it is not a coincidence — 500 ms of runtime divided by a 50 ms body is ten iterations, so the loop ran at the rate of its own body and the ticker contributed nothing but a name. (This is a range rather than an exact number for the second reason §17.2.8 gives: when a tick and the run deadline come ready in the same instant, select picks between them at random.) You configured a limit that was never reached, and the number in your config file has no relationship to the behaviour of the program.

That is dangerous rather than merely wrong, and the reason is that it never pages anyone. It surfaces weeks later as a throughput ceiling nobody can explain.

17.2.4 Three Reasons Not to Ship It

It cannot express a burst. One permit exists at any moment and nothing accrues. A client idle for a minute is treated exactly like one that has been hammering you, which is the opposite of what a burst parameter is for.

It enforces the receiver’s rate, silently. §17.2.3.

Its leak is now a ghost, which is its own hazard. time.Tick returns a channel you cannot stop, and before Go 1.23 an unreachable ticker was never collected, so time.Tick in a request path was an unbounded leak. That is no longer true.

17.2.5 What rate.Limiter Does Instead

The natural assumption is that golang.org/x/time/rate is a well-tested version of the ticker loop — a goroutine, a timer, a channel. It is none of those. It has no goroutine, and no timer exists until you call Wait.

The mechanism is arithmetic over time.Now():

// Illustrative snippet — not a complete program
// from x/time/rate, condensed
func (lim *Limiter) advance(t time.Time) (newTokens float64) {
    elapsed := t.Sub(lim.last)
    delta := lim.limit.tokensFromDuration(elapsed)
    tokens := lim.tokens + delta
    if burst := float64(lim.burst); tokens > burst {
        tokens = burst // the cap, and the whole burst design
    }
    return tokens
}

A call computes how many tokens accrued since the last call, caps at the burst, subtracts what you asked for, records the time. Tokens are float64, so a rate of 2.5/s is exactly representable rather than rounded to a tick interval. The bucket refills lazily, at the moment somebody looks at it.

Two consequences worth carrying. An idle limiter costs nothing — no goroutine to leak, no ticker to stop — which is why you can hold thousands of them, one per tenant, and why §17.7.2 is about what that costs instead. And everything happens under one mutex, which makes a shared limiter shared mutable state in the precise Chapter 9 sense; §17.3.6 measures the bill.

17.2.6 Choosing the Burst

Burst is the parameter people set carelessly, usually by making it equal to the rate because the numbers look like they belong together. They are different units — burst is in requests, rate is in requests per second — and equality means nothing.

The question burst answers is: how much simultaneous arrival can the thing I am protecting absorb? That is a concurrency question, not a rate question (§17.1.2), and its answer comes from the downstream.

• Burst 1 is a strict pacer. Requests come out evenly spaced and idleness earns no credit. Use it when the downstream’s problem is simultaneity.
• Burst = the downstream’s safe concurrency is the usual right answer, because that is literally what it permits to arrive at once.
• Burst = rate forgives a full second of history, which is fine for batch clients and wrong for a downstream sized by rate rather than by concurrency.

17.2.7 The Algorithms You Will Hear Named

Token bucket and leaky bucket are usually described as opposites and are better understood as the same accrual with different outputs: a token bucket lets you spend what has accrued, a leaky bucket paces output regardless. rate.Limiter with burst 1 is close enough to a leaky bucket to be the practical way to get strict pacing.

17.2.8 Why This Chapter’s Figures Are Exact

Everything here is time-dependent, which normally means every test of it is slow, flaky, or both. §15.7.7 measured the limit of testing/synctest: a real net/http server inside a bubble never advances, because a socket is not a state the runtime can see every goroutine blocked on (Go 1.27’s httptest.NewTestServer moves HTTP onto an in-memory network for exactly this reason — §16.4.8 — but the point stands for anything holding a real socket).

A limiter has no socket. It is pure time and pure channels — exactly what the bubble was built for.

Not “about a second”. Exactly a second, in zero real time, on every run and on any machine.

This is the precise inverse of Chapter 15's finding, and it is why the load figures in this chapter are stated as exact numbers rather than ranges: they are computed on a virtual clock, so the program that produced the value matrix in §17.1.5 prints those same numbers wherever it runs. Chapter 16 covers the machinery; what matters here is the licence it grants.

17.2.9 Common Mistakes

Summary: Token Buckets, and Why Not a Ticker

Chapter 11's rate limiter is race-free and still wrong. A fixed window forgets its counter at a wall-clock boundary, so a caller spends a full allowance either side of it — measured at 200 requests in 100 ms against a configured 100/s. The mutex fixed a synchronization bug; the algorithm was never a synchronization bug.

A token bucket accrues instead of forgetting. Rate is sustained throughput, burst is bounded forgiven history, and burst is a question about what the downstream can absorb at once — a concurrency answer, not a rate one.

time.Ticker cannot express a burst and drops ticks for slow receivers by design: 10 to 13 arrived where 50 were due, so the limiter silently enforced its own loop body’s rate. Since Go 1.23 an unstopped ticker is collectable, which makes defer tk.Stop() a statement of intent rather than a leak fix.

rate.Limiter runs no goroutine and no timer — lazy float arithmetic under one mutex, free when idle and contended when shared.

And because a limiter is pure time and pure channels, it is fully deterministic under synctest: 11 events at 10/s took exactly one second of virtual time. That is the inverse of §15.7.7, and it is why this chapter’s load figures are exact wherever a clock or a serialised admission point decides the outcome — and honest ranges wherever scheduling gets a vote.

Self-Check Questions: Token Buckets, and Why Not a Ticker

17.3 Allow, Reserve, Wait: Choosing How to Fail

rate.Limiter has three ways to ask for a permit. They are usually presented as convenience variants of one operation. They are three different answers to the question this chapter keeps returning to — what happens to the one who cannot be served now? — and choosing between them is the most consequential decision in this section.

17.3.1 Three Answers to One Question

                        permit available?
                         │              │
                        yes             no
                         │              │
Allow()     ────────────►true         false      → reject
Reserve()   ────────────►ok, d=0      ok, d>0    → you decide
Wait(ctx)   ────────────►nil          blocks     → wait

Allow is the rejection disposition. It asks, gets an answer, returns. Nothing waits. If you are shedding load this is the call, and it is the only one of the three that cannot make a caller slow.

The three are the same handler with one line changed, which is the clearest way to see that the choice is about the caller and not about the limiter:

// Illustrative snippet — not a complete program
// REJECT. Never blocks, never keeps the caller.
if !lim.Allow() {
    http.Error(w, "rate limited", http.StatusTooManyRequests)
    return
}
serve(w, r)

// WAIT. Blocks until a permit or the caller's deadline, whichever
// comes first. Bounded only by ctx -- see §17.3.2.
if err := lim.Wait(r.Context()); err != nil {
    return // caller gave up, or a permit would arrive too late
}
serve(w, r)

// DECIDE. Quote a price, then choose. The only one that can put a
// number in the refusal.
res := lim.Reserve()
if d := res.Delay(); d > budget {
    res.Cancel()
    w.Header().Set("Retry-After", strconv.Itoa(int(d.Seconds())+1))
    http.Error(w, "rate limited", http.StatusTooManyRequests)
    return
}
time.Sleep(res.Delay())
serve(w, r)

Wait is the waiting disposition. It blocks until a permit is free or the context is done. It is the shortest to write, which is why it is the one nearly everyone reaches for first.

Reserve is neither. It takes a permit, tells you how long until you may use it, and hands the decision back:

// Illustrative snippet — not a complete program
r := lim.Reserve()
if !r.OK() {
    return errTooBig // n exceeds burst; never satisfiable
}
if d := r.Delay(); d > budget {
    r.Cancel() // give the tokens back — §17.3.4
    return errWouldBeTooSlow
}
time.Sleep(r.Delay())

Quote me a price and I will decide is the right shape whenever the caller has a deadline it can reason about, and it is the least used of the three.

It also has a use the other two cannot serve. r.Delay() is the one number in this chapter that answers “how long until this would work?” — which is exactly the number a Retry-After header carries (§17.6.2). Refusing with Retry-After: <the delay you just cancelled> turns a bare rejection into a scheduling instruction, and it costs one line on a path you are already taking.

17.3.2 Wait Is an Unbounded Queue

§14.4.4 measured what a parked goroutine costs: 2,081 bytes of stack and 606 bytes of heap, 2.7 KB apiece, and ten thousand of them holding 25.6 MiB where a limited group needed 27 KiB. That measurement is cited, not repeated. It is the same goroutines and the same bill.

What differs is who decides how many there are.

In Chapter 14 the count came from a slice you chose to submit. Ten thousand items meant ten thousand goroutines because you wrote the loop. The number was yours, knowable before the loop started, and SetLimit capped it.

limiter.Wait(ctx) in a request handler has no such bound. The count is set by arrivals, and arrivals are set by the internet.

At an offered load below the limit this is indistinguishable from correct. Above it, every excess request becomes a parked goroutine holding 2.7 KB plus its request context plus whatever the handler allocated before the call — and the limiter reports, accurately, the whole time, that it is holding the configured rate.

Wait converts a rate problem into a memory problem, and memory problems page later and louder. The rate problem was visible at the limiter; the memory problem surfaces as an OOM kill with a heap profile full of runtime.gopark.

17.3.3 Measured: The Two Ways Wait Fails Without Waiting

Wait has two paths that return an error immediately rather than blocking, and they are easy to confuse.

The deadline short-circuit. Given a context with a deadline, WaitN computes how long the permit will take and compares it against the time remaining. If the permit would arrive too late it fails now rather than sleeping until the deadline and failing then:

That is the right behaviour, and it is worth reading as a design principle rather than a detail: if the work cannot finish in time, refusing now is strictly better than refusing later. Refusing now returns the caller’s remaining budget so it can try a fallback, a different replica, or a degraded path while it still has time. It is §17.5.2's deadline-as-shed-signal, implemented inside a library.

It also means Wait under load does not necessarily park. Callers with tight deadlines get fast errors; only callers with slack accumulate — §17.1.5's condition, appearing again as a mechanism.

The dead limiter.

Nobody writes 0 deliberately. It arrives as a config value that failed to parse, a struct field nobody set, or arithmetic — burst := rps / 10 with rps of 5. The limiter then rejects one hundred percent of traffic and reports it only through a return value on the shed path, which is the path least likely to have an alert on it. The service is fully available, entirely idle, and refusing everything.

17.3.4 Reserve: The Promise You Have Already Paid For

Reserve hands you the decision and attaches an obligation that is easy to miss: the tokens are consumed when you reserve them, not when you act. Decide not to proceed and they are gone unless you give them back.

The limiter is ten tokens in debt for ten requests that were never sent, and the next caller waits eleven times longer than one arriving at an idle limiter. Scale that to a path where reservations are abandoned routinely — a cache hit, a validation failure, a disconnected client — and the limiter enforces a rate materially below the one you configured, with no error anywhere and no metric that would show it.

Two caveats keep this honest. Cancel restores tokens only “as much as possible”: if another reservation has been issued behind yours, yours cannot be cleanly withdrawn without moving somebody else’s slot. Cancel promptly and restoration is full. And there is no ReserveContext — for a reservation bounded by a deadline you compare r.Delay() against your own budget yourself.

17.3.5 Measured: Wait Cleans Up After Itself

That hedge in CancelAt invites an obvious worry. Wait cancels its reservation when the context is done — so does a workload with many cancelled waits leave holes in the schedule and achieve less than the configured rate?

Three hundred cancelled reservations cost nothing measurable. The mechanism explains it: CancelAt declines to restore only the tokens reserved after yours, and in a workload where each cancellation resolves before the next reservation is made, there are none. The hedge protects a case requiring reservations to overlap and outlive each other, which a Wait-driven workload does not produce.

Worth noting how the test had to be built, because the obvious version answers a different question. Cancelling with an already-expired context does not exercise the path at all: Wait short-circuits on §17.3.3's deadline check and never makes a reservation, so nothing is ever cancelled. The reservation has to be made and then cancelled, from elsewhere, while the caller is parked.

So the rule is not “cancellation is dangerous”. It is narrower and more useful:

While the schedule holds, so does the ordering.

17.3.6 Measured: The Limiter Is One Mutex

Every Allow, Wait and Reserve takes lim.mu. A shared limiter is shared mutable state in the exact Chapter 9 sense.

16.5× under contention. At high fan-in the limiter becomes the bottleneck it was installed to prevent — a serialization point on the hot path of every request, doing arithmetic that costs 10.9 ns and mutex handoff that costs the rest.

Sharding by key helps and keeps helping: 137 ns at eight keys down to 128 at sixty-four. It never inverts. But look where it stops. From 64 keys to 512 the figure barely moves, and it never comes close to the unshared 10.9 — because past a certain point you are no longer measuring the limiters at all. The floor is the map’s own lock, taken by every request regardless of key, and adding shards cannot break a floor that does not shard.

Which is why the last row matters more than the sharding rows. Taking a write lock to perform a read is the actual cost, and a keyed limiter map is read-mostly by construction: the entry is created once and read forever after. Swapping sync.Mutex for sync.RWMutex takes 128.5 ns to 49.6 — a bigger win than every sharding decision combined, from a one-line change. §17.7.2 is where that map’s other problem lives.

17.3.7 rate.Sometimes

One more type ships in the same package, and it answers a question people keep solving badly with a limiter: not how often may this happen, but how often should I bother mentioning it. Log sampling, first-N-then-periodic error reporting, and “tell me when this starts and when it stops but not ten thousand times in between” are all this shape, and a token bucket is the wrong tool for every one of them.

17.3.8 Choosing

17.3.9 Common Mistakes

Summary: Allow, Reserve, Wait: Choosing How to Fail

Three methods, three dispositions: Allow rejects, Wait waits, Reserve quotes a price and hands the decision back. Reserve is the least used and often the right one when the caller has a deadline it can reason about.

Wait is an unbounded queue whenever the caller population is unbounded. Chapter 14 measured a parked goroutine at 2.7 KB; the difference here is that arrivals decide the count. It converts a rate problem into a memory problem.

Wait fails without waiting in two ways: the deadline short-circuit, which returns the caller’s budget so a fallback can use it, and the dead limiter — NewLimiter(r, 0) refuses everything forever, reported only on the shed path.

Reserve consumes on reservation. Measured, ten abandoned reservations put the limiter ten tokens in debt and made the next caller wait 110 ms instead of 10 ms.

But Wait cleans up after itself, measured: three hundred cancelled waits cost nothing. That asymmetry — Wait cancels for you, Reserve makes you do it — is the reason to prefer Wait unless you need the delay value. Wait is also strictly FIFO, on every run.

And a shared limiter is one mutex: 180.3 ns/op against 10.9 unshared, 16.5× for the privilege of being shared — and a keyed map that takes a write lock to do a read gives most of that back for one line (§17.3.6).

Self-Check Questions: Allow, Reserve, Wait: Choosing How to Fail

17.4 The Semaphores You Already Have, and the One You Don’t

§17.1.1 counted four limiters already in your hands. This section is about when to reach past them, and the honest answer is: less often than the existence of a library suggests.

17.4.1 What the Channel Semaphore Cannot Say

Chapter 5 built this and it remains correct:

// Illustrative snippet — not a complete program
sem := make(chan struct{}, 10)

sem <- struct{}{}        // acquire
defer func() { <-sem }() // release
doWork()

Ten permits, held by whoever is inside. It allocates once, costs a channel operation per acquire, and has no dependency. For most concurrency limiting it is the right answer, and reaching for a library instead is a step backwards.

It cannot say three things.

It cannot be cancelled. sem <- struct{}{} blocks until a permit frees, and nothing else. Under a context you write the select yourself — and then you must remember that the release belongs only on the path where the acquire succeeded:

// Illustrative snippet — not a complete program
select {
case sem <- struct{}{}:
    defer func() { <-sem }() // only after a successful acquire
case <-ctx.Done():
    return ctx.Err()         // no release — nothing was taken
}

Releasing a permit you never held quietly raises your limit by one for the life of the process, which is a bug that looks like a capacity-planning mystery.

It cannot weight. Every permit is identical, so a request that will use eight units of something and one that will use one are charged the same. When the protected resource is memory or bandwidth rather than a slot, that is the wrong model.

It does not promise fairness. The language specification guarantees nothing about the order in which blocked senders are woken. In practice the gc runtime queues them and wakes them FIFO — measured, twelve goroutines blocked on a full channel in a known arrival order woke as [0 1 2 … 11] on every trial. So the honest objection is not that callers get starved; it is that you would be depending on an implementation detail the spec does not owe you, on a toolchain that is free to change it.

§14.4.4 covers the fourth inventory item and its measurements stand: errgroup.SetLimit blocks the caller rather than the goroutine, so the goroutine is never created until there is room — 27 KiB against 25.6 MiB for ten thousand items. Not re-derived here. If you are limiting a group of tasks you launch yourself, SetLimit is still the answer and this section is not about your problem.

17.4.2 What Weight Buys

golang.org/x/sync/semaphore addresses the first two gaps. Acquire(ctx, n) takes a context and a weight, TryAcquire(n) is the non-blocking form, Release(n) returns the weight.

// Illustrative snippet — not a complete program
sem := semaphore.NewWeighted(maxBytes)

if err := sem.Acquire(ctx, req.Size()); err != nil {
    return err // cancelled or deadline exceeded
}
defer sem.Release(req.Size())

That is a genuine improvement when the protected thing is divisible — a memory budget, a bandwidth allowance, a pool where some operations cost more than others. It is not an improvement when you are counting slots, and using it there buys the next four behaviours for nothing.

17.4.3 Measured: TryAcquire Is Starved by a Single Waiter

TryAcquire is what you reach for to shed load: ask, get an immediate yes or no, refuse if no. It is the Allow of the semaphore world.

It returns false whenever any waiter is queued, regardless of how much capacity is free.

Six units are available. The request needs one. It is refused.

This is deliberate — it is §17.4.4's FIFO guarantee doing its job — but the consequence for load shedding is specific and severe: one blocked Acquire disables your entire fast path. Every subsequent TryAcquire returns false, so every request is shed, while the resource sits two-thirds idle. The service reports itself overloaded and its utilisation graph disagrees — and you will not find it from the outside, because Weighted exposes no accessor for its current utilisation or its waiter count. The starvation is undiagnosable unless you counted acquisitions and releases yourself, which is why §17.7.5 asks for queue depth as one of the four numbers rather than as a nicety.

17.4.4 Measured: Strict FIFO, and What It Costs

The behaviour above is not a bug, and the source says why. semaphore.Weighted maintains a FIFO waiter list, and notifyWaiters stops at the first waiter it cannot satisfy rather than stepping past it. The comment gives the reason: skipping would let a waiter with a large n be starved indefinitely by a stream of small ones.

So the package makes a choice, and it is the opposite of Chapter 5's channel:

held: 2                                        free: 6
  │▓▓│▓▓│··│··│··│··│··│··│
   1  2  3  4  5  6  7  8

waiter queue (strict FIFO):
  1st   wants 8   ── cannot be satisfied ─┐
  2nd   wants 1   ── could be satisfied  ─┴─ BLOCKED

notifyWaiters stops at the first waiter it cannot serve.
Skipping it would starve the large acquire forever.

Neither column is correct in general. If every acquisition is the same size, FIFO costs nothing and buys nothing, and you should use the channel because it is simpler. If sizes vary and the large ones matter, Weighted is buying a documented starvation guarantee — which is the real difference, since the channel’s FIFO behaviour is a property of today’s runtime rather than a promise — and charging idle capacity for it. What you must not do is choose without knowing which trade you took, because both failure modes present as “the limiter is too small”.

17.4.5 Measured: The Same Condition, Opposite Behaviour

Two packages, the same condition — a request larger than total capacity — and opposite answers.

rate checks and refuses. semaphore blocks until the context is done, and the source is explicit:

// Illustrative snippet — not a complete program
// from x/sync/semaphore
if n > s.size {
    // Don't make other Acquire calls block on one that's
    // doomed to fail.
    s.mu.Unlock()
    <-done
    return ctx.Err()
}

The reasoning is sound: a doomed acquire must not sit at the head of the FIFO queue blocking everyone behind it, so it is parked outside the queue entirely. But read the last two lines again. Under context.Background(), done is nil, <-done blocks forever, and the goroutine is gone permanently.

That is Chapter 2's goroutine leak, inside a library you had no reason to audit, on a path requiring no mistake beyond a weight that is too large — which is what happens when the weight is a request’s byte count and somebody uploads a file bigger than your budget.

17.4.6 Release Panics

Release panics with "semaphore: released more than held" on over-release. Unlike the channel semaphore, which would simply block or silently inflate the limit, this one tells you loudly. That is the better design, and one more reason to pair every acquire with exactly one deferred release on the success path only.

17.4.7 Sizing the Limit

Chapter 14 gave the starting heuristic and it stands: runtime.NumCPU() for CPU-bound work, roughly double for I/O-bound, then measure. What §17.1.3 adds is a way to derive rather than guess when you know two of the three quantities.

That arithmetic converts a limit you cannot observe into two numbers you can. You rarely know the right concurrency for somebody else’s service. You frequently know its published rate limit and its latency.

The number that actually binds is almost never yours: the connection pool, the file-descriptor ceiling, the downstream’s own limit, or total memory divided by per-item footprint. And when the dependency slows, a limit derived from yesterday’s latency is wrong in the direction that hurts — §17.7.4's argument for measuring rather than configuring.

17.4.8 Choosing a Concurrency Limiter

17.4.9 Common Mistakes

Summary: The Semaphores You Already Have, and the One You Don’t

Chapter 5's six-line channel semaphore is still right for counting slots. It cannot be cancelled without a hand-written select, cannot weight, and is not fair — and only the last two are reasons to reach for a library.

semaphore.Weighted buys weighted acquisition and strict FIFO, and the FIFO is the source of two of its four surprises. TryAcquire returns false whenever any waiter is queued, so one blocked Acquire disables load shedding entirely while capacity sits free — measured, six of eight units available and a one-unit request refused. Head-of-line blocking is deliberate and documented: the price of a starvation guarantee Chapter 5's channel does not offer.

An Acquire for more than total capacity does not error. It parks until the context is done, which under context.Background() is a permanent goroutine leak — where rate.WaitN in the identical situation returns immediately. Two packages, one condition, opposite answers.

Release panics on over-release, which is the better failure. And the limit itself is derivable rather than guessable: L = λW from a dependency’s published rate and its sustained latency.

Self-Check Questions: The Semaphores You Already Have, and the One You Don’t

17.5 Backpressure: Whose Queue Is It

§17.1.6 established that the queue exists whether or not you build one. This section is about what its size actually buys.

17.5.1 Measured: Capacity Is a Latency Budget

Chapters 5 and 7 both said to bound the queue and both left the number to judgement. Here is what the number does.

cap     served ─────────────►    useful
 10     1610  ████████████████   1610  ████████████████
 50     1650  ████████████████   1650  ████████████████
100     1700  █████████████████   496  ████
500     2000  ████████████████████ 496 ████

The consumer got busier and the service got worse. The
cliff is at cap = deadline × drain rate = 100.

Read the two middle columns against each other, because they move in opposite directions.

Throughput rises and goodput collapses. Raising capacity from 10 to 500 increased finished work by 24% — 1,610 to 2,000 — and cut useful work by 69%, from 1,610 to 496. The consumer got busier and the service got worse. Every additional unit of work it completed was for a caller who had already gone.

Zero rejections is the alarming row, not the good one. At capacity 500 nothing is refused, which is what a naive dashboard calls success. What actually happened is that the shedding decision moved from your code, where it was explicit and countable, to the caller’s timeout, where it is neither.

The wait is depth ÷ drain rate, exactly. 25 ms, 125 ms, 250 ms, 1.0 s. That is W = L/λ from §17.1.3, and it is not an approximation — it is what a queue draining at a fixed rate does.

Two details keep that table honest. The bottom row never filled: offered 500/s against a drain of 400/s is a net fill of 100/s, so 500 slots would need five seconds and the run is four. It reached depth 400, which is why rejected is 0 — not because 500 was enough, but because the window ended first. And the identical useful figure in the last two rows is not a transcription error: both runs cross the 250 ms deadline at the same instant, because both fill at the same 100/s and pass depth 100 at t = 1 s. After that moment every completion in either run is for a caller who has gone, so both harvest exactly the same useful window.

Which makes the crossover derivable rather than empirical. Derived: goodput collapses once queue wait reaches the caller’s deadline, which happens at a depth of deadline × drain rate. Here that is 0.25 × 400 = 100 — and 100 is the row where useful has already fallen off the cliff, because a queue at exactly that depth is serving requests whose callers are timing out as they are served.

So 100 is the break-even point, not the target: it is the ceiling you must stay under, and the largest capacity that still works in this run is 50. Size below deadline × drain rate, not at it. Beyond it, added capacity is not absorbing bursts; it is manufacturing work nobody will be waiting for.

17.5.2 The Deadline Is Already a Shed Signal

The cheapest rejection available costs one comparison and requires no configuration, because the caller already told you:

// Illustrative snippet — not a complete program
func (w *Worker) handle(job Job) error {
    if job.Ctx.Err() != nil {
        droppedExpired.Inc()
        return job.Ctx.Err() // never started; nobody is waiting
    }
    return w.do(job)
}

Checking the context when work is dequeued — rather than when it is enqueued — is what turns a queue from a liability into an asset under overload. Every item that expired while waiting is discarded for the cost of an atomic load, and the capacity it would have consumed goes to an item whose caller is still there.

That single check is what would have saved the 1,504 wasted completions in §17.5.1's bottom row, and it is the mechanism behind §17.1.5's waste column. It is also why that measurement gave both arms the check: it is cheap enough that withholding it would have been rigging the comparison.

It composes with §17.3.3. rate.Limiter.Wait already refuses a permit that would arrive after the caller’s deadline. The same rule at the dequeue point, at handler entry, and before each expensive stage is the cheapest load shedding a service can do — and it requires only that deadlines are actually propagated, which is Chapter 13's argument arriving with a payoff.

17.5.3 What to Drop, and From Which End

Once the queue is full, four policies are available and they are not interchangeable.

Block the producer. Backpressure in the strict sense: the caller waits, and if it is itself serving somebody the pressure propagates upstream. Correct when the producer can afford to slow down.

Reject the newest. The default in most systems and the right default when requests are equivalent. select with a default clause is exactly this.

Drop the oldest. Counter-intuitive and frequently correct: the oldest item in a full queue is the one most likely to have exceeded its caller’s deadline already. For live data — sensor readings, price ticks, position updates — it is also the least valuable by definition.

Serve newest-first. Under sustained overload, LIFO beats FIFO on the metric that matters. FIFO serves in arrival order, so once the queue wait exceeds the client timeout, everyone is late and the observed completion rate goes to zero while the server stays fully busy — which is §17.5.1's bottom row. LIFO lets recent arrivals through at full speed and starves the backlog.

The first two are one channel operation apart, which is worth seeing side by side because the difference is a single default clause:

// Illustrative snippet — not a complete program
// REJECT THE NEWEST. The default clause is the whole policy.
select {
case q <- item:
default:
    rejected.Add(1) // the caller finds out
}

// DROP THE OLDEST. Make room, then take the slot. The second send
// cannot block: this goroutine is the only producer, and it just
// freed a slot.
select {
case q <- item:
default:
    select {
    case <-q: // discard the head -- most likely already too late
        dropped.Add(1)
    default:  // drained by the consumer in between; room now
    }
    q <- item
}

Drop-the-oldest needs a single producer, or the freed slot is a race: two producers can both free one slot and then both try to take it, and one of them blocks on a queue it believed had room.

17.5.4 The Queue at Shutdown

§15.5.3 drew a line between work that is in flight and work that is queued, and §15.5.5 argued that abandoning the backlog is sometimes correct. This chapter’s waiters sit exactly on that line, and the book has not yet asked which side they belong to.

A goroutine parked in limiter.Wait(ctx) or sem.Acquire(ctx, n) has been admitted by your admission control and not started by your handler. It holds a promise but no resource. When SIGTERM arrives, is it in-flight work a graceful shutdown must finish, or backlog to abandon?

The answer comes from §15.1's four promises rather than from the mechanism. The promise is that accepted work finishes — and a request blocked at a limiter has been accepted in the only sense the client can observe: it is holding a connection and waiting for an answer. So it is in-flight from the client’s side and queued from yours, and the two genuinely disagree.

Three consequences, in the order they bite:

The waiters must be cancelled, not merely the handlers. If shutdown cancels the request tree, Wait(ctx) and Acquire(ctx, n) return immediately with ctx.Err() and the goroutines unwind. If the limiter was given a context that is not derived from the request — a common shape, since limiters are long-lived objects and reaching for a long-lived context feels consistent — nothing cancels them, and Shutdown waits for handlers that are waiting for permits a draining service will never issue.

A drain stalled at a limiter looks exactly like a deadlock. It is §10.2's shape with a deadline attached, and kill -QUIT reads it identically: every stack ending in rate.(*Limiter).WaitN or semaphore.(*Weighted).Acquire names the culprit immediately. §15.7.5 recommends the technique for precisely this.

Whether to drain them is a policy decision with a right default. Cancel them, and answer each with a 503. A client that has been waiting at your limiter through a shutdown has almost certainly exceeded its own deadline, and §15.5.5's test applies unchanged: abandon the backlog when the work has another copy — and a queued request behind a load balancer always does.

17.5.5 Where Backpressure Stops

Backpressure propagates by making the producer wait. That works while the producer is a component you control, and it stops working entirely at the edge.

You cannot make the internet wait. A browser, a mobile client, a third-party integration — none of them respond to a slow server by sending less. Most respond by sending more, because a timeout triggers a retry, which is §17.6.

So the boundary rule is: internally, block. A pipeline stage, a worker pool, a batch reader can all be slowed by a full channel, and that propagation is the whole value of an unbuffered channel — Chapter 7's argument. At the edge, reject. Return a status the client can act on, with a Retry-After if you can estimate one.

Holding an inbound request to apply backpressure to something that will not feel it converts a fast rejection into a slow one and adds a parked goroutine for the privilege. The service boundary is where the four dispositions stop being interchangeable: inside, wait is usually right; outside, reject almost always is.

17.5.6 Common Mistakes

Summary: Backpressure: Whose Queue Is It

Capacity is a latency budget, and the measurement is unambiguous. Raising a buffer from 10 to 500 slots against sustained overload raised finished work 24% and cut useful work by 69%, while rejections fell to zero — the row that looks like success is the one where the shedding decision moved from your code to the caller’s timeout. Wait is depth ÷ drain rate exactly, which makes deadline × drain rate a break-even ceiling to stay under rather than a target — here the largest capacity that still worked was 50 against a break-even of 100.

The cheapest shed signal is the caller’s own deadline, checked at dequeue rather than enqueue — one atomic load, and it is what turns a queue into an asset under overload.

Four policies decide who loses. Drop-oldest and serve-newest-first are the counter-intuitive ones that are frequently correct, because under a queue that is not draining, the fair policy is the one where nobody is served in time.

Waiters parked at a limiter sit on §15.5.3's in-flight/queued line: accepted by the client’s reckoning, queued by yours. Derive their context from the request, cancel them at shutdown, answer with a 503 — and know that a drain stalled at a limiter reads as a Chapter 10 deadlock in a goroutine dump.

And backpressure stops at the service boundary: block internally, reject at the edge, because the internet does not slow down when you do.

Self-Check Questions: Backpressure: Whose Queue Is It

17.6 Retries, Amplification, and the Breaker That Stops Them

§17.1.5 showed ten retries pushing 259–273 of 300 requests through while burning a third of the downstream — admission control rediscovered badly. This section measures what retries actually do, and then builds the mechanism that refuses before it waits.

17.6.1 Measured: What Retries Actually Amplify

The usual claim is that retries cause collapse. The measurement says something narrower and more useful.

no admission control            admission control
────────────────────            ─────────────────
1 try    peak 219               1 try    peak 10
3 tries  peak 653               3 tries  peak 10
           ▲                               ▲
           3x the concurrency,             retries absorbed
           +1 request served               at the permit

Retries multiply offered load. The bound decides whether
multiplied load costs anything.

Read the first two rows against each other. Three attempts instead of one raised offered load by 44% and raised served work by one request. What it actually changed was peak concurrency: from 219 to 653, three times the in-flight count, for no additional goodput at all. That is amplification with nothing to absorb it — and on a downstream whose service time degrades with concurrency, tripling the in-flight count is how a slowdown becomes an outage.

Now read the bottom two rows. With a cap of ten, retries change peak in-flight not at all: it is pinned at exactly 10, in both rows. The extra attempts wait at the permit instead of piling into the resource that degrades.

The waste column here does not go to zero, and it is worth saying why, because §17.1.5's limited arm did. These are different overload regimes. There, the offered load fitted inside the budget once admission was serialised, so every caller was eventually served and nothing was wasted. Here, 600 arrivals per second against roughly 500 per second of capacity is overload the limiter cannot resolve — callers still time out, and work still completes for callers who have left. A concurrency cap bounds concurrency; it does not create capacity, and when offered load genuinely exceeds capacity the residual waste is the price of the overload rather than of the retries. What the cap removed is the amplification: the peak, and with it the degradation that made each call slower than the last.

So the honest statement is narrower than “retries cause collapse”:

That reframing matters because it points at the fix. Telling clients to retry less is asking other people to protect you, and they will not. Bounding your own concurrency works regardless of what clients do.

17.6.2 Three Things That Make Retries Survivable

You will still retry, because a network that drops one packet should not fail a request. Three things make it safe.

A budget, not a count. “Retry three times” is a per-request rule that says nothing about aggregate load. A retry budget caps retries as a fraction of total requests — conventionally 10% — over a rolling window. Healthy, the budget is never reached and retries work normally. Under widespread failure it is exhausted immediately and retries stop, which is exactly the behaviour you want and the one a per-request count cannot express.

Full jitter, not just backoff. Exponential backoff without randomness synchronises clients. A thousand clients failing together retry together, then again at 2×, 4×, 8× — load arriving in spikes that each re-trigger the failure:

// Illustrative snippet — not a complete program
// Full jitter: sleep uniformly in [0, backoff), not backoff.
backoff := min(base<<attempt, maxBackoff)
time.Sleep(time.Duration(rand.Int63n(int64(backoff))))

A uniform draw over the whole interval, rather than a small perturbation of it, is what actually decorrelates the herd.

A way to be told no. Retry-After on a 429 or 503 is the server telling the client what it knows and the client does not: how long the condition will last. Honouring it is cheaper than any backoff algorithm because it is not a guess.

And the retry must be safe to send twice. Chapter 15 named this from the other end: a client seeing a connection reset knows the request failed and does not know whether the work failed. If the operation is not idempotent, a retry is a correctness bug rather than a performance one, and no budget fixes it.

17.6.3 The Breaker: Refusing Before You Wait

Every mechanism so far decides admission from capacity. A circuit breaker decides from history: it stops calling a dependency that has been failing, on the theory that the next call will fail too and the attempt is pure cost.

That makes it the only thing in this chapter that refuses before it waits.

         failures exceed threshold
  ┌──────────────────────────────────────►┐
  │                                       │
CLOSED                                  OPEN
  │   all calls pass              all calls fail
  ▲   failures counted            immediately
  │                                       │
  │                                       │ after cooldown
  │        probe succeeds                 ▼
  └──────────────────────────────── HALF-OPEN
                                    ONE call passes
              probe fails ─────────────►OPEN

Closed is normal operation with accounting. Open is the useful state: calls fail instantly, without a connection, a timeout, or a permit. Half-open is the recovery test.

There is no circuit breaker in the standard library and none in golang.org/x/. You assemble this one, which makes the three design decisions yours — and each is where implementations go wrong.

17.6.4 Half-Open Must Admit Exactly One

Half-open exists to answer one question: has the dependency recovered? Answering it takes exactly one request. Admitting more re-hammers a service that is, by hypothesis, fragile — and a dependency emerging from an outage with a cold cache and an empty connection pool is at its most fragile precisely then.

The naive implementation admits everything that arrives during half-open, which under load is thousands of requests, which knocks the recovering service over, which reopens the breaker. That oscillation is stable and can outlast the original fault by hours.

Admitting exactly one is a compare-and-swap:

// Illustrative snippet — not a complete program
const (
    stateClosed int32 = iota
    stateOpen
    stateProbing // exactly one caller holds this
)

// state is an atomic.Int32 (§11.1.2), not a bare int32 behind the
// legacy package-level functions.
func (b *Breaker) allow() bool {
    switch b.state.Load() {
    case stateClosed:
        return true
    case stateOpen:
        if time.Since(b.openedAt()) < b.cooldown {
            return false
        }
        // cooldown elapsed: try to become the prober
        return b.state.CompareAndSwap(stateOpen, stateProbing)
    default:
        return false // somebody else is probing
    }
}

Chapter 11 said CompareAndSwap is the tool when the operation is “change this value if it still looks like that”, and this is the cleanest example in the book: many goroutines observe the expired cooldown in the same instant, exactly one wins the swap, and the losers are refused without a lock.

Chapter 11 also named what atomics cannot express — a state change carrying a side effect — and this breaker is that shape too. The transition from probing back to closed must reset the failure counters, and resetting counters does not fit inside the swap. So the state machine is a CAS for the admission decision and a mutex for the transition with bookkeeping attached. That is not a compromise; it is §11's rule read correctly.

17.6.5 Measured: What Counts as a Failure

This is the most important paragraph in the chapter.

A breaker needs a predicate: given the outcome of a call, was that a failure? The obvious answer is the wrong one.

// Illustrative snippet — not a complete program
err := downstream.Call(ctx)
b.Record(err) // ✗ ctx.Err() is an error too

The context passed to Call belongs to your caller. If they gave up — a mobile client on a bad network, an upstream service shutting down, a user closing a tab — then Call returns context.Canceled or context.DeadlineExceeded, and that is an error. It is not a downstream error. Nothing whatsoever has been learned about the downstream’s health.

Feed it to the breaker anyway and you have built a device that converts other people’s impatience into your own outage.

A 100% availability loss against a downstream that did not return a single error. The first wave never even reached a failure — those two hundred requests were still running happily when their callers walked away. The breaker recorded two hundred “failures”, opened, and refused every subsequent request for a full cooldown.

The fix is one condition, and the interesting part is what it tests:

// Illustrative snippet — not a complete program
err := c.Call(ctx)
// Two questions, and one check cannot answer both.
// 1. Did we refuse this ourselves? Then it is not evidence
//    about them -- and ctx.Err() is still nil when our own
//    limiter short-circuits (§17.3.3), so this must come first.
if errors.Is(err, ErrShed) || errors.Is(err, ErrBreakerOpen) {
    return err
}
// 2. Was our caller already gone? Then the error is theirs.
if ctx.Err() == nil {
    b.Record(err)
}
return err

Two clauses, and the order matters. The second is the one people miss, so it gets the emphasis: the test is on ctx, not on err. The near-miss version — if !errors.Is(err, context.Canceled) — is close and wrong in a case that matters: a downstream enforcing its own internal deadline and returning DeadlineExceeded from it really has failed, and you want that counted. Testing ctx.Err() asks the right question, which is “was this our caller’s fault?” rather than “does this error look like a cancellation?”.

The first clause is easy to skip and §17.3.3 is the trap that punishes it. rate.Limiter.Wait refuses a permit that would arrive after the caller’s deadline — and it refuses before that deadline passes, so ctx.Err() is still nil at the moment the error is returned.

And you can do this to yourself twice over. §15.4 cancels the request tree during a graceful shutdown; §15.2 cancels it when a signal arrives. Every in-flight call returns context.Canceled at once. A breaker whose state outlives the process — persisted, or shared across a pool not being torn down — opens everything it owns as its last act.

This is §14.1's thesis arriving for the third time. Chapter 14 said the failure mode of concurrent error handling is silence rather than a crash. §15.4.4 found it at the process boundary as a 200 OK with a truncated body. Here it is at the dependency boundary, as a breaker opening against a healthy service — and in all three cases the system reports success at every layer that has a dashboard.

17.6.6 Trip on Rate, With a Minimum

The second decision is what opens the breaker. The obvious answer — a count of consecutive failures — is wrong in both directions at once.

Consecutive-failure counting is traffic-dependent. At five failures, an endpoint receiving one request a minute needs five minutes of total failure to trip. An endpoint receiving ten thousand requests a second trips in half a millisecond, during a blip that would have cleared, from a dependency at 99.95% success.

The fix is to trip on error rate over a rolling window, which is traffic-independent by construction. But a rate alone has a hole at the bottom: the first request after a deploy fails, the rate is 100%, and the breaker opens on a sample of one.

So a usable predicate has two clauses:

17.6.7 What a Breaker Does Not Fix

It does not reduce load on a healthy service. A closed breaker is a counter. If your dependency is slow rather than failing, the breaker never opens and never helps; that is §17.3's job.

It makes an optional dependency worse. §14.7 distinguished critical from optional. Wrapping an optional dependency in a breaker converts occasional slow responses into a guaranteed absence for the whole cooldown. If the right behaviour on failure is to degrade, degrade — a breaker adds a state machine and removes the chance that the next call might work.

It hides the fault. An open breaker means your error rate is generated locally, the dependency’s dashboards go quiet, and the incident looks like it is in your service. Emit a distinct metric and error type for breaker-refused calls, or you will spend the outage debugging the wrong system.

It cannot be your only protection. A breaker responds to failures that have already happened. Everything else in this chapter is about not causing them.

17.6.8 Common Mistakes

Summary: Retries, Amplification, and the Breaker That Stops Them

Retries do not cause collapse by themselves.

Safe retrying needs a budget rather than a count, full jitter rather than plain backoff, a Retry-After you honour, and an operation that is safe to send twice.

A circuit breaker is the only mechanism here that refuses before it waits, and you assemble it yourself. Half-open must admit exactly one probe — CompareAndSwap, Chapter 11's cleanest use — because admitting the backlog knocks a recovering dependency over and the oscillation outlasts the fault. Trip on error rate over a window with a minimum-request floor, never on consecutive failures.

And the predicate is where breakers become outages.

Self-Check Questions: Retries, Amplification, and the Breaker That Stops Them

17.7 Where the Limit Lives

Every number so far has been treated as though writing it down were the end of the job. It is not. A limit is enforced somewhere, by some number of processes, over some population of callers, and each of those choices changes what the number means.

17.7.1 The Limit You Configured Is Not the Limit

// Illustrative snippet — not a complete program
limiter := rate.NewLimiter(100, 10) // "100 requests per second"

That comment is true of one process. Deploy to eight replicas and the downstream sees 800 requests per second, because each replica holds its own bucket and none knows about the others.

NewLimiter(100, 10) per process

  8 replicas    ──►    800/s     burst   80
 40 replicas    ──►   4000/s     burst  400
                        │
                        └─ and you scaled to 40 BECAUSE
                           the system was under strain

Documented dependency limit: 1000/s

A protective limit that scales with the thing it protects
you from is not a limit.

Three honest responses.

Divide by the replica count. rate.NewLimiter(total/replicas, burst/replicas) is correct while the count is known and stable. It fails during deploys, when old and new pods overlap, and under autoscaling — which is when it matters.

Enforce at a chokepoint. If all traffic to the dependency passes through a proxy, a sidecar or a mesh, the limit belongs there, where there is exactly one of it. Usually the right answer and usually not available.

Enforce at the dependency. A server-side limit is the only inherently correct one, because the server is the thing being protected and it can count. Client-side limits are always an approximation; their value is refusing work before it costs a network round trip.

A shared bucket in Redis is the fourth option and a distributed systems problem with its own failure modes — the store’s latency on every request, its availability becoming yours, and what to do when it is unreachable. That is beyond this book. What matters here is knowing that a per-process limiter is a local approximation of a global intent, and writing the aggregate in the comment:

// Illustrative snippet — not a complete program
// 100/s per replica. At 8 replicas that is 800/s at the
// dependency, whose documented limit is 1000/s.
limiter := rate.NewLimiter(100, 10)

17.7.2 Keyed Limiters and the Map That Never Evicts

The standard production shape is one limiter per user, tenant or IP:

// Illustrative snippet — not a complete program
// ✗ BROKEN: every key ever seen is retained forever
func (l *Limiters) For(key string) *rate.Limiter {
    l.mu.Lock()
    defer l.mu.Unlock()
    lim, ok := l.m[key]
    if !ok {
        lim = rate.NewLimiter(10, 20)
        l.m[key] = lim
    }
    return lim
}

Correct, and it leaks. Every distinct key ever observed is retained for the life of the process, and the key space is chosen by your callers. For user IDs that is bounded by your user count, which may be acceptable. For IP addresses it is bounded by the internet, and a scan across a /16 creates 65,536 limiters that will never be used again and never be freed.

§12.3 said this exactly: rate-limiter state cannot live in a sync.Pool, because the collector empties it and the state must survive. That leaves eviction to you, and there are three workable answers.

Sweep by last use. Record a timestamp on each access and periodically delete entries idle longer than a few multiples of the refill interval.

Bound the map. An LRU with fixed capacity gives a hard memory ceiling. The cost is that an evicted key gets a fresh full bucket on its next request, so size it well above your active-key count.

Do not key at all. For abuse prevention specifically, per-IP limiters are often the wrong tool, because the attacker chooses the key. A global limit plus a cheap per-request cost is more robust.

The reason sweeping is safe is worth stating, because it is what makes eviction free rather than a trade:

In code the two phases are the whole discipline:

// Illustrative snippet — not a complete program
// ✓ Two phases. The scan holds a READ lock, so requests still
// proceed; the delete holds the write lock for a handful of keys
// at a time rather than for the length of the map.
func (l *Limiters) sweep(idle time.Duration) {
    cutoff := time.Now().Add(-idle)

    l.mu.RLock()
    var stale []string
    for k, e := range l.m {
        if e.last.Load() < cutoff.UnixNano() {
            stale = append(stale, k)
        }
    }
    l.mu.RUnlock() // released BEFORE any deletion

    for i := 0; i < len(stale); i += 128 {
        l.mu.Lock()
        for _, k := range stale[i:min(i+128, len(stale))] {
            // Re-check under the write lock: the key may have been
            // used again while we were not holding it, and evicting
            // an ACTIVE limiter hands its owner a full bucket.
            if e, ok := l.m[k]; ok &&
                e.last.Load() < cutoff.UnixNano() {
                delete(l.m, k)
            }
        }
        l.mu.Unlock()
    }
}

The re-check is not belt and braces. Between the two phases the key can be used again, and the callout above says what evicting a live limiter costs: a full burst, on demand, to a caller who can force it.

And §17.3.6's measurement applies here: sharding across keys spread limiter contention and concentrated a new bottleneck in the map’s own lock. The eviction strategy and the contention strategy are the same decision.

17.7.3 Client-Side, Server-Side, Per-Key

The pairing that works in practice is a server-side global limit for self-protection, a server-side per-key limit for fairness between tenants, and client-side limits as a cheap first filter that avoids a round trip. They are not alternatives; each catches something the others cannot.

17.7.4 Measured: One Adaptive Algorithm, and Both Its Failure Modes

Every limit so far has been a constant somebody picked. The alternative is deriving it from what is actually happening, and the simplest algorithm that works in production is Google’s client-side adaptive throttle:

requests and accepts are counted over a decaying window. While everything is accepted, requests ≈ accepts, so with K = 2 the numerator is negative and p is zero — the throttle is invisible. As acceptances fall, p rises and the client throttles itself before the downstream has to. K is how much rejection you tolerate before backing off; the +1 keeps a trickle flowing so the client can discover recovery.

It is twenty lines, and this chapter shows every other mechanism it recommends:

// Illustrative snippet — not a complete program
type Throttle struct {
    mu                sync.Mutex
    requests, accepts float64
    K, Decay          float64 // K=2.0, Decay=0.5 per window
}

// Allow reports whether to send. Call Record after the attempt.
func (t *Throttle) Allow() bool {
    t.mu.Lock()
    defer t.mu.Unlock()
    t.requests++
    p := (t.requests - t.K*t.accepts) / (t.requests + 1)
    return p <= 0 || rand.Float64() >= min(p, maxReject)
}

// Record files whether the backend accepted the request.
func (t *Throttle) Record(accepted bool) {
    t.mu.Lock()
    defer t.mu.Unlock()
    if accepted {
        t.accepts++
    }
}

// Tick decays both counters; call it once per window.
func (t *Throttle) Tick() {
    t.mu.Lock()
    defer t.mu.Unlock()
    t.requests *= t.Decay
    t.accepts *= t.Decay
}

maxReject is the floor the callout below insists on: clamp the rejection probability strictly below 1 so a trickle always escapes, whatever the ratio says.

The first row is the algorithm working. Outbound load halves during the outage with no coordination at all, and recovery climbs back to full over about five seconds, driven entirely by the trickle of probes the +1 guarantees.

The other two rows are the failure modes, and they fail in opposite directions.

K too aggressive collapses. K = 1.1 protects the downstream better during the outage — 30% against 48% — and then never comes back. Five seconds after the downstream is completely healthy the client is still sending single-digit percentages. Because it throttles so hard, too few probes get through to rebuild accepts, and decay erodes the successes faster than the trickle replaces them. The client has talked itself into a permanent outage, and no downstream metric shows anything wrong.

No decay is inert. With counters that never age, the throttle remembers a healthy past forever: after thousands of successful requests, four seconds of total failure barely moves the ratio, so 91% of the flood still goes out. This is the more common configuration mistake, because it looks conservative — “don’t throw away data” — and produces a throttle that does nothing when it is needed.

17.7.5 The Four Numbers

None of this is tunable without instrumentation, and the instrumentation that ships by default is the wrong half. A limiter exporting only its permitted rate tells you what it allowed and nothing about what it is holding — §17.3.2's OOM is invisible on that metric.

Rejected must be split by reason — limiter, breaker, deadline-expired, queue-full — because that breakdown turns §17.6.5's bug into a seconds-long diagnosis and its absence makes it nearly invisible.

Queue depth is the one most often missing and the one this chapter argues hardest for. It is the difference between a service at its limit and a service accumulating an unbounded backlog, and those have identical admitted-rate graphs.

semaphore.Weighted exposes no accessor for its utilisation, so if you want that number you count acquisitions and releases yourself. That is two atomic increments on a path that already takes a mutex, and it is worth it.

17.7.6 Common Mistakes

Summary: Where the Limit Lives

rate.NewLimiter(100, 10) is 100/s per process, so eight replicas enforce 800/s and autoscaling to forty makes it 4,000/s — the protection weakening exactly when the dependency needs it. Divide by the replica count, enforce at a chokepoint, or enforce at the dependency; write the aggregate in the comment either way.

Per-key limiters are the standard shape and they leak, because the key space belongs to your callers. Sweeping by last use is free rather than a trade, because an idle limiter is indistinguishable from a fresh one — but evicting an active one hands its owner a full bucket, and doing the sweep under the write lock blocks every request in the service.

Static limits go stale. The Google SRE throttle adapts in twenty lines, and both of its failure modes are measurable: K too aggressive collapsed to single-digit throughput and never recovered, and counters that never decay left 91% of the flood going out during a total outage. Give any adaptive limit a floor, and never adapt a limit that is a contract.

And none of it is tunable without four numbers: admitted, rejected split by reason, queue depth, and wait-time p99. Depth is the one usually missing and the one that separates a service at its limit from a service accumulating a backlog.

Self-Check Questions: Where the Limit Lives