Chapter 16: Testing Concurrent Code

16.1 Why Concurrent Tests Lie

A test is an argument that code is correct. For sequential code the argument is strong: you fixed the inputs, so you fixed the execution, so a pass today is a pass tomorrow. Concurrency breaks the middle step. The inputs are fixed and the execution is not, because a second author — the scheduler — contributes to every run and never tells you what it wrote.

This section is about the three distinct ways that shows up, because they need different fixes and are constantly confused with each other.

16.1.1 A Test That Passes Ninety-Nine Times in a Hundred

Start with the worst case, which is not the test that fails.

// Illustrative snippet — not a complete program
func racyCount(workers, iters int) int {
    counter := 0
    var wg sync.WaitGroup
    for i := 0; i < workers; i++ {
        wg.Go(func() {
            for j := 0; j < iters; j++ {
                counter++
            }
        })
    }
    wg.Wait()
    return counter
}

counter++ from two goroutines with no synchronisation is the first data race in Chapter 8, and every reader of this book can see it. The interesting part is what the test does about it.

That number is the problem. A test that fails half the time is a bug report. A test that fails one run in forty is noise — it goes in the flaky bucket, gets a retry wrapper, and the data race ships. The severity of the bug and the frequency of the symptom are unrelated, and human triage keys on frequency.

That contrast is the strongest argument for the race detector in this book, and it is worth stating precisely, because §16.3 will spend a section on the limits: unaided, the bug surfaced 3 times in 200; instrumented, it surfaced 10 times in 10. The detector did not make the race more likely. It made the race visible, which is a different and much stronger property.

16.1.2 A Test That Fails Somewhere Else

The second failure mode is rarer and much more expensive, because the test that fails is not the test that is broken.

// Illustrative snippet — not a complete program
// ✗ BROKEN: nothing waits for this goroutine
func TestOrphanA(t *testing.T) {
    go func() {
        time.Sleep(200 * time.Millisecond)
        t.Error("assertion from an orphaned goroutine")
    }()
}

The test body starts a goroutine and returns. The goroutine is still sleeping. Two hundred milliseconds later it calls t.Error on a *testing.T whose test finished long ago.

Read the two lines together. TestOrphanA is reported PASS. The run then dies inside TestLaterB, which is innocent, and the panic message is the only thing connecting the crash to its cause.

Your green test killed someone else’s.

And the ending depends on something with no relationship to either test.

This is the exact mechanism behind a large fraction of tests that “only fail in CI”. CI runs more tests, on slower hardware, in a different order.

16.1.3 A Test That Never Finishes

The third mode produces no output at all, which makes it the one people debug last and longest.

A test that deadlocks does not fail. It stops. The Go runtime has a deadlock detector, but §10.4 established its limit precisely: it fires only when every goroutine is asleep, so a test that blocks forever while the rest of the suite runs is invisible to it. What eventually intervenes is the test binary’s own alarm.

Two things are worth noticing. The running tests: block names the culprit and how long it has been stuck, which is more than the runtime’s detector gives you. And the price of that information is the full timeout — five seconds here, ten minutes at the default, per occurrence, on every CI run until someone fixes it.

§16.4 shows the same information arriving in 0.00 seconds, from a mechanism that does not need a timeout at all.

16.1.4 Schedules and Invariants

Three failure modes, three different fixes. What connects them is what the test was asserting on.

A test asserts on an invariant when it says something that is true of every correct execution: after Wait returns, the counter equals the number of increments. It asserts on a schedule when it says something that is only true of some executions: within 100 ms the worker will have finished, or by the time this line runs, that goroutine will have started.

Every flaky concurrent test is a test that asserts on a schedule. The sleep in the third opening listing is the obvious case, but §16.1.2's orphan is the same mistake wearing a different hat — it assumes the test outlives the goroutine, which is a claim about scheduling, not about correctness.

That gives the chapter’s thesis, and it has two halves because there are exactly two ways out:

Either stop asserting on the schedule, which means finding the invariant and asserting on that instead. Or take ownership of the schedule, which means making it an input to the test rather than a property of the machine. §16.4 and §16.6 are the two ways to do the second; §16.3 and §16.7 are what you do when neither is available.

16.1.5 The Determinism Ladder

Those options form a ladder, and knowing which rung you are on is most of the skill.

rung 4   NO CONCURRENCY IN THE ASSERTION
         Test the invariant on a sequential path. Fastest,
         strongest, and more often possible than you think.
           │
rung 3   VIRTUAL TIME  (§16.4)
         synctest: the bubble owns the clock and the
         definition of "everyone is blocked".
           │
rung 2   FORCED SCHEDULE  (§16.6)
         Barriers, seams, injected hooks. You choose the
         order; the scheduler does not get a vote.
           │
rung 1   REAL TIME
         Sleeps and timeouts. A bet on the machine. Sound
         only as an upper bound (§16.6.3).

Climb as far as the code allows. Anything holding a socket
stops you at rung 2 (§16.4.8).

Rung 4 deserves more attention than it gets. A great deal of code that runs concurrently can be tested sequentially, because the interesting logic is not the concurrent part. If a worker pool’s scheduling policy is a pure function of queue state, test that function directly with no goroutines at all, and use rung 2 or 3 only for the small amount of code that genuinely coordinates.

The most common mistake in this chapter’s subject is reaching for rung 1 when rung 4 was available.

16.1.6 Anatomy of a Flake

Abstractions are easier to believe with one worked example, so here is a flake taken apart. It is real, it is measured again in §16.6.4, and it is the shape most concurrent flakes take.

Two backends are queried in an errgroup. Both fail. The test asserts on the error.

// Illustrative snippet — not a complete program
func query(ctx context.Context) error {
    g, _ := errgroup.WithContext(ctx)
    g.Go(func() error { return errSlow })
    g.Go(func() error { return errFast })
    return g.Wait()
}

// Illustrative snippet — not a complete program
// ✗ BROKEN: asserts on which goroutine returned first
func TestFastBackendErrorWins(t *testing.T) {
    err := query(context.Background())
    if !errors.Is(err, errFast) {
        t.Fatalf("got %v, want errFast", err)
    }
}

Now walk it through the people it passes.

The author ran it, it passed, they shipped it. At a 1–2% failure rate there is roughly a 90% chance that running it five times locally shows nothing.

Review saw an assertion on an error value, which is the most ordinary thing in a Go test. Nothing in the diff mentions concurrency.

CI fails one build in fifty. A large suite has many tests, so “one build in fifty fails somewhere” reads as infrastructure rather than as a specific bug.

The on-call engineer re-runs the build and it passes. The evidence is now gone: no seed, no output, no record that the failure named a different error than expected.

Six months later the assertion is deleted or wrapped in a retry, and with it goes the only signal anyone had that errgroup's first-error selection is a race the service’s error handling depends on.

Nothing in that chain is unreasonable, which is the point. (The split is lopsided rather than even for a specific scheduler reason, which §16.6.4 measures and names.) The failure is in the first step, and it is a specification failure: errors.Is(err, errFast) was never true, only usually true. §16.6.4 places a seam and the same measurement becomes 200 of 200, deterministically.

16.1.7 What a Flake Is

One policy question, because it decides whether any of this machinery gets used.

A flaky test is a test that passes and fails on the same code. The instinct is to treat it as a defective test. Sometimes that is right — §16.1.4's schedule assertions are genuinely defective and the fix is in the test. But the other possibility is that the test is fine and it is telling you about a real race that fires rarely, which is precisely what §16.1.1 measured: a correct assertion, a real bug, a 3-in-200 symptom.

You cannot tell which from the failure rate. You have to look.

The working policy that follows: a flake is a bug report against the system until proven otherwise, and the proof is a mechanism, not a re-run. Quarantine it so it stops blocking the queue, keep it running so the data accumulates, and require someone to name the cause before it is deleted or retried. A retry wrapper added without that step converts a bug report into silence, which is the worst possible outcome — the information was there and you paid for it and then discarded it.

16.1.8 Common Mistakes

Summary: Why Concurrent Tests Lie

A sequential test fixes the execution by fixing the inputs. A concurrent test cannot, because the scheduler contributes to every run and never records what it chose. So a pass is evidence about one interleaving, and the interleaving was not selected by you.

That produces three failure modes with three different fixes. A test can pass while the bug is present — measured, 197 passes in 200 runs against a real data race, where -race caught it 10 times in 10. A test can fail somewhere else entirely, because an orphaned assertion goroutine reports against whichever test is running when it wakes: measured, — PASS: TestOrphanA followed by a panic inside the innocent test that came after, or nothing at all if the binary exits first. And a test can simply stop, invisible to the runtime’s deadlock detector, until the binary’s own alarm fires — measured, five seconds to learn what §16.4 learns instantly.

What unites them is asserting on a schedule instead of an invariant. The way out is the determinism ladder: eliminate the concurrency from the assertion, or own the clock, or own the ordering, and treat real-time sleeps as what they are — a bet on hardware you do not control.

Self-Check Questions: Why Concurrent Tests Lie

16.2 The testing Package Under Concurrency

The previous section assumed that when a test asserts, the assertion counts. For concurrent tests that assumption is often false, and the ways it fails are not documented anywhere a reader is likely to look. This section comes second because everything after it depends on it: there is no point owning the schedule if the assertion that checks the result cannot fail the test.

Four behaviours, each measured, each with the same root cause — testing.T is bound to one goroutine, and Go gives you no help remembering which.

16.2.1 t.Fatal Does Not Stop What You Think

The documentation for FailNow says it plainly, and almost nobody has read it:

Fatal and Fatalf call FailNow. So this does not do what it looks like:

// Illustrative snippet — not a complete program
// ✗ BROKEN: the Fatal does not stop the test
func TestFatalInGoroutine(t *testing.T) {
    var wg sync.WaitGroup
    wg.Go(func() {
        t.Log("child: about to call t.Fatal")
        t.Fatal("failing from a child goroutine")
    })
    wg.Wait()
    t.Log("REACHED: line after the child called t.Fatal")
}

The test does fail — Fatal marks it failed and that part works. What does not happen is the stopping. FailNow terminates its goroutine by calling runtime.Goexit, and the goroutine it terminates is the child. The test body carries straight on to the next line.

What it does stop is the child. FailNow ends its goroutine with runtime.Goexit, which runs that goroutine’s deferred calls and then destroys it — silently, as far as the rest of the test is concerned. That combination is worse than either half alone:

// Illustrative snippet — not a complete program
// ✗ BROKEN: the Fatal kills the child before it can send
func TestFatalInChildHangs(t *testing.T) {
    results := make(chan int)
    go func() {
        conn := pool.Get()
        if conn == nil {
            t.Fatal("no connection")  // Goexit: no send happens
        }
        results <- serve(conn)
    }()
    t.Log("status:", <-results)
}

The guard fires, the child dies, and the send on results never happens. The test goroutine is now blocked forever on a receive that has no sender left alive.

A clear failure with a written-out message has been converted into §16.1.3's third failure mode — a hang with no output — by a line whose entire purpose was to report the failure clearly. And the message itself does not survive either.

So the failure message is not hard to find. It is discarded, by the very mechanism that was supposed to report it.

The fix is t.Error plus an explicit return, which is honest about what is happening, or better, moving the assertion out of the goroutine entirely (§16.2.7).

16.2.2 The Assertion That Blames Another Test

§16.1.2 measured the symptom. Here is the mechanism, because it explains a class of CI failures that otherwise look supernatural.

testing.T carries a done flag, set when the test function returns. Every call that records a result checks it:

The panic is deliberate and it is the right design — the alternative is silently discarding a real failure. But notice what it costs. The panic fires on the orphaned goroutine, at whatever moment it happens to run, and it takes down the process. The test that was executing at that moment is the one whose output you will be reading.

TestOrphanA              TestLaterB
┌───────────────┐        ┌──────────────────────────────┐
│ spawns ───────┼────────────────┐                      │
│ returns       │        │       │  200ms               │
└───────────────┘        │       ▼                      │
  reported PASS          │    t.Error  ──▶  panic       │
                         └──────────────────────────────┘
                               the run dies here,
                               in the innocent test

The name in the panic message is the only link back to
the test that actually caused it.

The practical advice: when a CI run dies with Fail in goroutine after X has completed inside test Y, the bug is in X and Y is a bystander. Read the name in the message, not the name in the header. It is the one case in Go testing where the reported test and the responsible test are reliably different.

16.2.3 defer, t.Cleanup, and t.Parallel

t.Parallel changes when a subtest body runs, and that breaks the most natural way to write teardown.

The mechanism: calling t.Parallel signals that the subtest should be paused and resumed later, after the parent test function has returned. The parent’s t.Run calls therefore return immediately, the parent body finishes, and only then do the parallel subtests actually execute.

defer is scoped to the parent function. t.Cleanup is scoped to the parent test, which now ends later.

// Illustrative snippet — not a complete program
// ✗ BROKEN: the fixture is closed before the subtests use it
func TestDeferVsParallel(t *testing.T) {
    open := true
    defer func() { open = false }()

    for _, n := range []string{"a", "b"} {
        t.Run(n, func(t *testing.T) {
            t.Parallel()
            t.Logf("sub %s sees open=%v", n, open)
        })
    }
}

The defer runs fourth, before either subtest. The t.Cleanup runs last, after both. Every parallel subtest sees a fixture that has already been torn down, and if the fixture were a database handle rather than a boolean, the subtests would fail with an error that mentions a closed connection and says nothing about t.Parallel.

t.Run("a")   ──────┐    paused
t.Run("b")   ──────┤    paused
                   │
parent body ends   │
                   │
defer   ───────────┤    ✗  fixture closed HERE
                   │
subtest a   ───────┤       sees a closed fixture
subtest b   ───────┤       sees a closed fixture
                   │
t.Cleanup   ───────┘    ✓  the correct place

t.Parallel pauses the subtest until the parent function has
returned. defer is scoped to the function; t.Cleanup is
scoped to the test.

So: inside any test that runs parallel subtests, teardown belongs in t.Cleanup, not in defer. This is not a style preference. The two run at different times and only one of them is correct.

There is a second interaction worth knowing, because it fails loudly rather than subtly. Some testing helpers mutate process-global state and are therefore incompatible with parallelism outright.

A panic, not a silent misbehaviour — which is the right trade, and worth contrasting with the defer case above, where the same category of mistake produces no diagnostic at all.

16.2.4 t.Context and the End of the Test

Go 1.24 added T.Context, which answers a question every previous chapter had to answer by hand: how does a test-owned goroutine know to stop?

“Just before” is load-bearing, and it is exactly the ordering you want.

And the consequence that makes it useful — a worker parked on <-t.Context().Done() has already returned by the time cleanup runs, so a wg.Wait() in a t.Cleanup does not hang:

That is the whole pattern for a test-owned goroutine:

// Illustrative snippet — not a complete program
func TestWorkerStops(t *testing.T) {
    var wg sync.WaitGroup
    wg.Go(func() {
        <-t.Context().Done()
        // ... shut down ...
    })
    t.Cleanup(wg.Wait)

    // ... the actual test ...
}

Compared with a hand-rolled context.WithCancel plus defer cancel(), this is shorter and, more importantly, it composes with the previous subsection: t.Cleanup is already the correct place for teardown under t.Parallel, and t.Context is cancelled to line up with it.

16.2.5 Assertion Helpers, and Where a Verdict Lives

Concurrent tests accumulate helpers faster than sequential ones — waiting for a condition, draining a channel, checking an invariant across several goroutines — and helpers are where §16.2.1's bug survives review, because a helper looks like a function rather than like an assertion.

Start with the mechanical part. t.Helper() marks a function as a helper so failures are reported at the caller’s line:

// Illustrative snippet — not a complete program
func mustReceive(t *testing.T, ch <-chan int) int {
    t.Helper()   // report at the CALLER's line, not this one
    select {
    case v := <-ch:
        return v
    case <-time.After(5 * time.Second):
        t.Fatal("nothing on the channel after 5s")
        return 0
    }
}

Without it, every failure in a suite that uses mustReceive points at the same line of the same file, which tells you nothing. Two details make it reliable: call t.Helper before anything that can fail, including before a select that might time out, because it is not retroactive; and remember that it changes only the reported line, not which goroutine may call it.

Which is the part that matters here. The signature is the giveaway. A helper taking *testing.T announces a verdict and belongs on the test goroutine. A helper returning error computes a verdict and may be called from anywhere:

// Illustrative snippet — not a complete program
func checkDrained(p *Pool) error {
    if n := p.Pending(); n != 0 {
        return fmt.Errorf("%d jobs still pending", n)
    }
    return nil
}

Now it composes. A goroutine can call it and send the result back; the test goroutine asserts. When you are unsure which form you want, write the second — it converts to the first in one line and never the other way round.

Some codebases encode this in names, so a caller can tell without reading the body: require* uses Fatal and is test-goroutine-only, check* uses Error or returns a value and is safe anywhere. Whatever the convention, the point is that “may I call this from a goroutine?” should be answerable from the call site.

16.2.6 What go vet Catches, and What It Misses

There is a vet analyzer for exactly the bug in §16.2.1, which is good news with a sharp edge on it.

The first is go func() { t.Fatal(...) }(). The second is go check(t), where the Fatal is one level down inside a helper — and vet names the helper. The third is go fn() where fn is a closure variable. It does not flag t.Errorf from a goroutine, which is correct: Errorf does not call FailNow and is safe there.

Now the edge.

// Illustrative snippet — not a complete program
var wg sync.WaitGroup
wg.Go(func() { t.Fatal("not flagged") })       // Go 1.25

var g errgroup.Group
g.Go(func() error { t.Fatal("not flagged"); return nil })

The analyzer looks for go statements. sync.WaitGroup.Go and errgroup.Group.Go start goroutines by calling a method, so there is no go statement in the test’s syntax tree and nothing to flag.

That gap matters more than it would have two releases ago. wg.Go arrived in Go 1.25 and is now the idiomatic way to start a tracked goroutine — this book switched to it in Chapter 14 — and errgroup has been the recommended way to run a group of failing-capable tasks since Chapter 14 introduced it. So the two constructs this book has spent three chapters teaching you to prefer are precisely the two that defeat the check.

And one more, easy to be caught by:

Taken together, the practical position is that vet is a useful backstop for the shape a newcomer writes and no help at all for the shape an experienced Go programmer writes. Run go vet ./... in CI as its own step rather than relying on go test to do it — and do not let a clean vet run substitute for the rule below.

16.2.7 The Rule

Four behaviours, one rule:

Everything in this section is a consequence. t.Fatal from a child does not stop the test because testing.T belongs to the test goroutine. An orphan blames another test because it outlived the goroutine it was bound to. Cleanup ordering is a question about when the test ends, not when a function returns.

In practice:

// Illustrative snippet — not a complete program
func TestHandlerRejects(t *testing.T) {
    type result struct {
        status int
        err    error
    }
    results := make(chan result, 1)   // buffered: never block a child

    go func() {
        status, err := serve(req)
        results <- result{status, err}
    }()

    select {
    case r := <-results:
        if r.err != nil {
            t.Fatal(r.err)         // on the test goroutine: works
        }
        if r.status != 503 {
            t.Fatalf("status = %d, want 503", r.status)
        }
    case <-time.After(2 * time.Second):
        t.Fatal("handler did not return")
    }
}

Three things are deliberate. The channel is buffered, so a child that finishes after the test has given up does not block forever holding a goroutine — an unbuffered channel here converts a test failure into a goroutine leak. The select has a timeout arm, used as a failure detector rather than for ordering (§16.6.3). And every t.Fatal is lexically inside the test function, where it does what it says.

The pattern generalises to any number of children: send a result per child, receive that many, and assert once you have them all. §16.6.2 shows the barrier version for the cases where you need to control when a child proceeds rather than just collect what it produced.

16.2.8 Common Mistakes

Summary: The testing Package Under Concurrency

testing.T is bound to the goroutine running the test, and the language provides no reminder. Four consequences follow, and none of them is discoverable by reading test code that looks correct.

t.Fatal from a child marks the test failed but does not stop it — measured, the line after the child’s Fatal executed. A guard clause written that way does not guard. An assertion from a goroutine the test never joined lands wherever it happens to land: measured, — PASS for the guilty test and a panic inside the innocent one that followed, or total silence when the binary exits first. Under t.Parallel, subtests run after the parent body returns, so measured, a defer teardown fires before every subtest and a t.Cleanup fires after all of them. And t.Context is cancelled just before cleanup, which makes t.Cleanup(wg.Wait) the complete idiom for a test-owned goroutine.

There is a vet analyzer for the first of these, and it has a gap worth knowing. Measured, go vet reports t.Fatal inside go func(), go helper(t) and go fn() — but not inside wg.Go or errgroup.Go, because it looks for go statements and those start goroutines by calling a method. Those are the two forms this book has taught since Chapter 14. And measured, go test does not run that analyzer at all.

One rule covers all of it: assertions live on the test goroutine, children report over a buffered channel.

Self-Check Questions: The testing Package Under Concurrency

16.3 The Race Detector as a Test Instrument

Chapter 8 taught the race detector as a tool: how to turn it on, how to read a WARNING: DATA RACE report, what the two stack traces mean, how to wire it into CI. This section asks a different question, and it is the one that decides how you use it: what is a clean -race run actually worth as evidence?

The answer is unusually precise, which is rare in this subject. The detector’s guarantees and its blind spots are both exact, and knowing them changes what you do with the rest of your testing budget.

16.3.1 What a Clean Run Proves

The race detector implements a happens-before algorithm. It maintains a vector clock per goroutine, records every synchronisation event that establishes an ordering — channel operations, mutex operations, WaitGroup waits, atomic operations — and for every memory access it checks whether the previous access to that address is ordered with respect to this one. If two accesses touch the same address, at least one is a write, and no happens-before edge connects them, it reports a race.

Two properties follow, and they are asymmetric.

No false positives. If the detector reports a race, there is a race. It is not a heuristic, and a report is never something to argue with — it is a proof that two unordered accesses occurred. This is a stronger guarantee than almost any other tool in the Go toolchain provides, and it is the reason a -race report should always outrank a passing test in your priorities.

Abundant false negatives. A clean run proves only that the accesses that executed on this run were properly ordered. It says nothing about accesses that did not execute. This is not a limitation of the implementation; it is inherent to a dynamic analysis, and it is where every practical mistake with the tool comes from.

The next three subsections make that asymmetry concrete, because the intuitions people bring to it are usually wrong in specific ways.

16.3.2 It Does Not Care About Timing

The most common misconception is that the race detector catches races when two goroutines happen to collide, and misses them when the timing is lucky — that it is a sampling tool whose hit rate depends on how tight the window is.

That is false, and it is worth measuring rather than asserting.

// Illustrative snippet — not a complete program
// Two writes to x, 50ms apart in wall-clock time.
func TestRaceWideApart(t *testing.T) {
    x := 0
    var wg sync.WaitGroup
    wg.Go(func() { x = 1 })
    wg.Go(func() {
        time.Sleep(50 * time.Millisecond)
        x = 2
    })
    wg.Wait()
    _ = x
}

The two writes are fifty milliseconds apart. On a 3.8 GHz machine that is roughly two hundred million cycles — they could not overlap if they tried.

The reason is the whole design. A time.Sleep is not a synchronisation operation, so it creates no happens-before edge. The detector does not ask “did these overlap in time?” — it asks “is there a chain of synchronisation events ordering these two accesses?” There is not, so they race, and the fifty milliseconds are irrelevant.

This has a direct practical consequence that surprises people: you cannot hide a race from the detector by adding sleeps, and you cannot help the detector find one by removing them. Timing pressure is the wrong lever entirely. §16.3.3 shows what the right one is.

One nuance keeps that statement honest, and it is a genuinely useful fact in its own right. Detection is not timing-sensitive — but enabling the detector changes which interleavings your program takes, because the race build deliberately perturbs the scheduler:

Under -race, the runnext fast path from §16.6.4 is skipped at random half the time.

A structural 99% bias becomes a near coin toss. So -race is two tools in one wrapper: a happens-before analyser that reports what it sees, and a partial scheduling fuzzer that changes what there is to see.

It is worth being precise about which of the two earned §16.1.1's result, because the tempting answer is wrong. Run that racy counter under -race and print the total alongside the report:

The counter is correct every time — no update was lost, so the two goroutines never meaningfully overlapped — and the race is reported anyway, six times out of six. None of that 10-of-10 came from the fuzzer. It came from the analyser, doing exactly what §16.3.2 said it does: reporting two unordered accesses because they executed, regardless of how far apart.

Which places the fuzzer correctly. It buys you nothing for data races, where the analyser already sees everything that runs. It buys you the other half of the chapter’s bug population — the race conditions of §16.3.4 that contain no data race, and the interleaving search of §16.7 — because those depend on ordering and nothing else in the toolchain perturbs it.

It also means a -race run is not a slower version of the same execution. It is a different sample of the schedule, which is a second, independent reason to run it.

16.3.3 It Only Sees What Ran

Here is the blind spot, in the form it actually takes in a real codebase.

// Illustrative snippet — not a complete program
type cache struct {
    mu   sync.Mutex
    hits int
    cold int  // written without the lock, on the miss path only
}

func (c *cache) get(miss bool) {
    if miss {
        c.cold++      // RACE -- but only on the miss path
        return
    }
    c.mu.Lock()
    c.hits++
    c.mu.Unlock()
}

A cache with a correctly-locked hot path and a racy cold path. The test suite, like most test suites, mostly exercises the hot path.

Fifty clean runs. Now one run of the miss path:

One run finds instantly what fifty could not find at all.

This is the single most important fact about the race detector as a test instrument, and it inverts the usual advice. -count=50 bought nothing here — not because fifty was too few, but because repetition was the wrong axis. The detector needed the miss path to execute once. It needed coverage, not iterations.

                      executed           not executed
                                    │
    ordered           clean         │
                      (correct)     │       INVISIBLE
                                    │
    unordered         REPORTED      │       invisible
                      (always)      │

The left column is a proof. The right column is why a clean
run is not one. Move code from right to left with COVERAGE,
not repetition.

The practical rule: treat -race as a multiplier on your test coverage, not as a test in itself. A package with 40% branch coverage running under -race is checking 40% of its code for races. The way to find more races is the same as the way to find more bugs of any other kind — execute more of the program.

16.3.4 What It Cannot See At All

Coverage explains the races the detector misses. There is a second category it cannot see at any coverage level, and §8.2 named it: a race condition is not a data race.

// Illustrative snippet — not a complete program
func (a *account) withdraw(n int) {
    a.mu.Lock()
    ok := a.balance >= n    // check
    a.mu.Unlock()
    if !ok {
        return
    }
    a.mu.Lock()
    a.balance -= n          // act
    a.mu.Unlock()
}

Every access to balance is under the mutex. There is no data race here, and the detector is correct to say so. There is still a bug: the check and the act are separate critical sections, so two goroutines can both read a sufficient balance before either subtracts.

Clean under the detector, wrong in a few percent of rounds — and the rate moves with load, which is exactly the property that makes it survive review.

This is the boundary of the tool, stated as sharply as it can be stated: -race verifies that your synchronisation is present. It cannot verify that your synchronisation is sufficient. Holding a lock for each individual access is exactly what the detector checks for, and exactly what is wrong here — the invariant needs one critical section spanning both operations, and no dynamic analysis of memory access ordering can know that.

Race conditions of this shape are found by the techniques in §16.7.4, by tests written against an invariant rather than an access pattern, and by code review (Chapter 18). Not by -race.

16.3.5 What It Costs

“Always run -race in CI” is good advice given as a slogan, which means the people who most need to weigh it get no help. The cost is real, it varies by an order of magnitude across workloads, and the variation is predictable.

The top row is the one that decides budgets, and it is the row usually left out. Code that shares nothing pays nothing — a loop doing arithmetic on local variables costs the same instrumented as it does plain. Your JSON parsing, your business logic, your template rendering: free.

The rest of the ordering is the opposite of what most people guess. The cheapest synchronising operation shows the worst multiplier.

The three synchronising factors move with machine load — the contended row has measured anywhere from 9× to 14× across sessions — so treat them as an order of magnitude rather than a constant. The ordering is the load-bearing part and it reproduces every time.

The reason is that the detector’s cost is roughly fixed per synchronisation event and per memory access, not proportional to what your code does between them. An uncontended Lock/Unlock pair around a single increment is almost pure synchronisation — there is no real work to dilute the instrumentation, so the overhead is the whole measurement. Channel ping-pong already spends most of its time in scheduler machinery that the detector does not multiply, so the same instrumentation is a much smaller fraction.

Which gives a rule you can apply without measuring your own code first: -race costs the most on the code that synchronises most per unit of work, which is exactly the code you most want to run under it. The tax is highest where the value is highest.

Memory matters too, and is less often mentioned: the detector allocates shadow memory proportional to the memory your program touches, so a test with a very large working set can need several times its usual footprint.

16.3.6 Exit Codes and CI

One detail that decides whether your CI actually enforces any of this.

The 66 is the one that catches people. It is the race detector’s own exit code, chosen to be distinctive, and it appears when you run a -race binary directly — which is what a CI job does when it builds an artifact and executes it, and what an integration-test harness does when it launches your service as a subprocess.

A shell script that checks if [ $? -eq 1 ] will conclude that a 66 means success. A supervisor that restarts on non-zero will restart in a loop without reporting why. And a docker run whose healthcheck only inspects stdout will see a crash with no explanation, because the WARNING: DATA RACE report goes to stderr.

The practical form: check for non-zero, not for a specific value, and make sure stderr from -race binaries is captured and surfaced. If you run your service under -race in an integration environment — which is an excellent idea and finds races no unit test will — this is the difference between the practice working and it silently doing nothing.

For the per-PR versus nightly split that follows from §16.3.5's cost data, see §16.7.7, which sets it out alongside the other knobs.

16.3.7 Common Mistakes

Summary: The Race Detector as a Test Instrument

The detector’s guarantees are asymmetric and both halves matter. A report is a proof — no false positives, never worth arguing with. A clean run is not a proof of anything beyond the accesses that actually executed.

Two intuitions need correcting. It is not timing-sensitive: measured, two writes fifty milliseconds apart with no happens-before edge are reported on the first run, because a sleep is not a synchronisation event. And its blind spot is coverage: measured, fifty -race runs of a hot path found nothing, while one run of the cold path found the race instantly. Repetition is the wrong axis; coverage is the right one.

It also cannot see race conditions that contain no data race. Measured, a check-then-act withdrawal with every access correctly locked drove the balance negative in 5–13 rounds out of 200 and reported PASS. The detector verifies that synchronisation is present, not that it is sufficient.

The cost ranges from nothing — code that shares nothing pays 1.02× — up to about 26×, ordered inversely to how much real work each operation does, so it is most expensive exactly where it is most valuable. And a -race binary exits 66, not 1, which is how the practice silently stops working in CI.

Self-Check Questions: The Race Detector as a Test Instrument

16.4 Determinism: testing/synctest

Everything so far has been about living with non-determinism: recognising it, measuring how much a green test proves, working out what the race detector does and does not cover. This section is about removing it.

testing/synctest went stable in Go 1.25. It is the largest change to testing concurrent code in Go’s history, and the reason is a single design decision: inside a synctest bubble, the runtime knows when every goroutine is blocked, so it can advance a fake clock instead of waiting. A test with a thirty-second timeout runs in no time at all, and it runs the same way every time.

Chapter 15 used it for one measurement and forwarded the rest here. This is the rest.

16.4.1 The Bubble

The entire API is three functions, and two of them were the whole package until Go 1.27.

// Illustrative snippet — not a complete program
func Test(t *testing.T, f func(*testing.T))
func Wait()
func Sleep(d time.Duration)   // Go 1.27

That is all of it. Test runs f inside a new bubble; every goroutine started within it joins the bubble. Wait blocks until every other goroutine in the bubble is durably blocked. Sleep, added in Go 1.27, advances the bubble’s clock by d and then waits for quiescence — time.Sleep(d) followed by Wait() in one call. Everything below is written in terms of Test and Wait; where a listing sleeps and then waits, Sleep is the shorter spelling.

The *testing.T handed to f is not quite the one you are used to, and the differences all follow from the bubble being a closed world:

• T.Cleanup functions run inside the bubble, immediately before Test returns — so cleanup gets virtual time too
• T.Context returns a context whose Done channel belongs to the bubble
• T.Run, T.Parallel and T.Deadline must not be called

That last restriction is worth pausing on. A bubble is a single deterministic world; subtests and parallelism would mean several, which is precisely what determinism rules out. Table-driven tests still work — you write the loop outside, with one synctest.Test call per case (§16.6.6).

16.4.2 Virtual Time

Inside a bubble, the time package is backed by a fake clock. Each bubble has its own, and it starts at the same instant every time.

The rule that makes it useful: time only advances when every goroutine in the bubble is durably blocked. When that happens, the clock jumps straight to the next moment that would unblock someone. Nothing waits.

// Illustrative snippet — not a complete program
func TestVirtualTime(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        start := time.Now()
        time.Sleep(30 * time.Second)
        t.Logf("virtual elapsed=%v", time.Since(start))
    })
}

Thirty seconds of virtual time in 0.00s of wall clock, and time.Since reports exactly 30s — not “about 30s”, not 30.0001s. The clock is not being sped up; it is being skipped.

That exactness is worth as much as the speed. A test of a backoff schedule can assert the delays were exactly 1s, 2s, 4s, 8s, rather than asserting they fell in tolerance bands that have to be widened every time CI gets busier. A fifteen-minute retry policy becomes a test that runs instantly and asserts on precise values:

// Illustrative snippet — not a complete program
func TestBackoffSchedule(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        var delays []time.Duration
        start := time.Now()
        for attempt := range 5 {
            retryAfter(attempt)             // sleeps internally
            delays = append(delays, time.Since(start))
            start = time.Now()
        }
        want := []time.Duration{1, 2, 4, 8, 16}
        for i, d := range delays {
            if d != want[i]*time.Second {
                t.Errorf("attempt %d: %v, want %v",
                    i, d, want[i]*time.Second)
            }
        }
    })
}

Every chapter in this book that measured a timeout — Chapter 13's context deadlines, Chapter 14's group cancellation, Chapter 15's grace budget — could have been tested this way for the parts that do not touch a socket.

16.4.3 Durably Blocked

Everything depends on that phrase, so it is worth getting exactly right.

The definition is about who can wake you. If the only thing that can unblock a goroutine is another goroutine in the same bubble, then the bubble knows the complete set of possible futures, and it can reason about them. If something outside could wake it — the network, the OS, another thread — the bubble cannot know when, so it cannot safely advance the clock.

The operations that block durably:

all durably blocked?
        │
        ├── someone called Wait   ──▶   Wait returns
        │
        ├── a timer could fire    ──▶   clock jumps to it
        │
        └── neither               ──▶   DEADLOCK, reported

A goroutine that is blocked but NOT durably -- on a mutex,
a socket, a syscall -- satisfies none of the three. The
bubble can neither advance nor conclude. That is the hang.

When every goroutine in the bubble reaches one of the “yes” states, exactly one of three things happens: Wait returns if someone called it; otherwise the clock advances to the next timer; otherwise there is no possible future at all, and that is a deadlock, which Test reports as a failure (§16.5.6).

16.4.4 What Is Not Durably Blocked

The “No” rows in that table are not footnotes. They are where tests go wrong, and the mutex row is the one that will bite first, because a mutex feels like exactly the kind of thing a bubble should understand.

It is not, and the reason is precise: a sync.Mutex has no bubble identity. Any goroutine anywhere in the process can hold it, so the bubble cannot prove that only a bubbled goroutine will release it.

Here is what that costs:

// Illustrative snippet — not a complete program
// ✗ BROKEN: this test hangs forever
func TestMutexIsNotDurable(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        var mu sync.Mutex
        mu.Lock()
        go func() {
            time.Sleep(time.Second)
            mu.Unlock()
        }()
        mu.Lock()   // parks on a mutex: NOT durable
        t.Log("acquired")
    })
}

Trace it. The root goroutine holds the lock and then tries to take it again, so it parks on the mutex. The child needs the clock to move a second forward before it can unlock. The clock will move only when every goroutine is durably blocked — and the root is parked on a mutex, which does not qualify. So the clock never advances, the child never wakes, the lock is never released, and nothing happens. Ever.

This is the most important trap in the whole package, and it deserves a blunt statement: the bubble’s deadlock detector has exactly the same blind spot as its clock. Detection requires every goroutine to be durably blocked. A goroutine parked on a mutex is blocked but not durably, so the bubble cannot tell “deadlocked” from “waiting for something outside” — and it must assume the latter, because assuming the former would fail correct tests.

So synctest gives you an instant, precise deadlock report (§16.5.6) for code that coordinates through channels, and degrades to a plain timeout for code that coordinates through mutexes. That is a real argument for channel-based coordination in code you intend to test deterministically, and one of the few places in this book where testability should influence a design choice that Chapter 9 and Chapter 12 would otherwise call a toss-up.

There is a related trap with WaitGroup, documented as a technical limitation:

// Illustrative snippet — not a complete program
var pkgWG sync.WaitGroup      // ✗ Wait() is not durably blocking
var ptrWG = new(sync.WaitGroup) // ✓ associates with the bubble

16.4.5 synctest.Wait

The second function solves the problem §16.6.2 otherwise solves with barriers: how does a test know a goroutine has got far enough?

// Illustrative snippet — not a complete program
func TestWaitNoBarrier(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        done := false
        go func() { done = true }()
        synctest.Wait()
        t.Log("after Wait, done =", done)
    })
}

Wait returns when every other goroutine in the bubble is durably blocked, which includes finished — a goroutine that has exited is not going to do anything else. So it is a general “let everything that can run, run” operation, which is precisely the primitive a test wants and which nothing outside a bubble can provide.

Note what this means for the data race in that listing. Writing done from one goroutine and reading it from another with no synchronisation is a data race by every rule in Chapter 8 — and synctest.Wait establishes a happens-before edge, so it is not. The bubble is not suspending the memory model; Wait is a real synchronisation operation, and §16.4.9 confirms the detector agrees.

The typical use is checking that something has not happened yet, which is normally the hardest kind of concurrent assertion:

// Illustrative snippet — not a complete program
synctest.Test(t, func(t *testing.T) {
    ctx, cancel := context.WithCancel(t.Context())
    fired := false
    context.AfterFunc(ctx, func() { fired = true })

    synctest.Wait()
    if fired {
        t.Fatal("AfterFunc ran before cancellation")
    }

    cancel()
    synctest.Wait()
    if !fired {
        t.Fatal("AfterFunc did not run after cancellation")
    }
})

Outside a bubble, the first assertion is untestable — “it has not happened yet” is always true if you check soon enough, and always suspect. Inside, Wait makes it exact: everything that could have run has run, so if it did not happen, it was never going to.

When the trigger is time rather than an explicit action, Go 1.27’s synctest.Sleep(d) is the same pattern with the sleep folded in: it advances the clock by d and then Waits. The package documentation gives the reason to prefer it over a bare time.Sleep: if the test and the system under test sleep for the same duration, which of the two wakes first is unpredictable, and the test almost always wants the system under test to settle first — which is exactly what the trailing Wait guarantees.

16.4.6 Contexts Use the Bubble’s Clock

A question comes up immediately, because so much Go code expresses its timing through context rather than through time directly: does a context deadline respect the bubble?

// Illustrative snippet — not a complete program
func TestContextUsesBubbleClock(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        ctx, cancel := context.WithTimeout(
            t.Context(), 10*time.Minute)
        defer cancel()
        start := time.Now()
        <-ctx.Done()
        t.Logf("ctx done after virtual %v; err=%v",
            time.Since(start), ctx.Err())
    })
}

It does, and it follows from the mechanism rather than from special handling. context.WithTimeout is built on time.AfterFunc, which is built on the runtime timers the bubble controls. Anything layered on the time package inherits the virtual clock for free.

That is more consequential than it first sounds, because it means the assertion can straddle the deadline:

Both sides, exactly, in no real time. Chapter 13's whole context tree, Chapter 15's grace budget, every per-request deadline and every WithTimeout in a retry loop becomes assertable at its production value. A ten-minute deadline is as cheap to test as a ten-millisecond one, which removes the last honest reason anyone had for parameterising timeouts down for tests — a practice that quietly means the number you ship is the one number you never tested.

16.4.7 Isolation Rules

A bubble is a closed world, and the runtime enforces it. Three rules, all of which produce immediate failures rather than subtle misbehaviour.

Bubbled channels, timers and tickers belong to their bubble. Operating on one from outside panics. This is what stops a bubble from leaking determinism into the rest of your suite, and it means a channel created inside a bubble cannot be used as a bridge to the outside — which is occasionally what you want to do and always wrong.

A WaitGroup associates with a bubble on its first Add or Go. After that, calling Add or Go on it from outside is a fatal error. §16.4.4 covered the package-variable limitation this creates.

sync.Cond.Wait is durably blocking, and waking a bubbled waiter from outside is a fatal error. The symmetry with WaitGroup is deliberate.

One more, easy to miss and occasionally the answer to a puzzling failure:

16.4.8 The Socket Boundary

The largest limitation follows directly from §16.4.3 and Chapter 15 already measured it.

Network I/O parks a goroutine on the runtime’s network poller, waiting for the operating system. The OS is outside the bubble. So a goroutine blocked on a socket read is not durably blocked, the clock cannot advance, and any test that puts a real network connection inside a bubble does not run — it hangs.

Measured in §15.7.7: a real net/http server inside a bubble never advanced; the test was still blocked after five seconds of real time.

That is not a bug to be fixed in a later Go release. It is inherent to real sockets: virtual time means the bubble decides when time passes, and it cannot decide that for a peer in another process.

INSIDE                          │  OUTSIDE
                                │
bubbled channels                │  a real socket
bubbled timers                  │  a mutex anyone else
Cond.Wait                       │    could unlock
bubbled WaitGroup.Wait          │  any system call
time.Sleep                      │  finalizers
                                │
virtual, exact, instant         │  the real world, on
                                │  its own clock

Cross the line and the clock stops advancing, because the
bubble can no longer know who might wake you. Bubble the
logic, not the transport.

The way through is to replace the network rather than to work around the clock. net.Pipe gives you a full net.Conn implemented entirely with in-memory channel operations — which are bubbled, and therefore durably blocking:

// Illustrative snippet — not a complete program
func TestProtocolHandshake(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        client, server := net.Pipe()
        t.Cleanup(func() { client.Close(); server.Close() })

        go handle(server)          // the code under test

        // ... write a request, read a response, assert ...
        // Every operation is a bubbled channel operation, so
        // timeouts inside handle() run on virtual time.
    })
}

This is the technique the standard library itself uses for exactly this purpose — the synctest documentation’s own worked example tests HTTP’s Expect: 100-continue handling over a fake network for the same reason.

For HTTP specifically the standard library now ships that fake network ready-made. Go 1.27’s httptest.NewTestServer(t testing.TB, handler http.Handler) returns an httptest.Server whose listener is in-memory rather than a loopback socket — the package documentation now says most tests should use it, and names testing/synctest as the reason. Create it inside synctest.Test, use srv.Client() as usual, and a handler-plus-client round trip runs on virtual time. The rule is unchanged: bubble the logic, not the transport — NewTestServer simply makes HTTP’s transport bubble-safe without you writing a net.Conn. (It is also the one 1.27 symbol the new default stdversion vet check flags: in a module still declaring go 1.25, go test refuses to build a file that uses it, so the go.mod bump and the listing land together.)

The decision rule is short. Bubble the logic, not the transport. A shutdown sequencer expressed in channels and contexts belongs in a bubble; the socket it eventually closes does not. Chapter 15's exercise used real short timeouts and hand-counted goroutines precisely because it kept a real listener; this chapter’s exercise strips the socket and gets determinism in exchange.

16.4.9 -race Still Works

One assumption worth killing before it costs someone their coverage: making a test deterministic does not blind the race detector.

That result is not obvious. A bubble controls scheduling closely enough that you might expect it to serialise everything and make races impossible to observe. It does not, and it should not: the detector works on happens-before edges (§16.3.1), and the bubble does not manufacture edges that the program did not create. synctest.Wait creates one because it genuinely is a synchronisation operation; two bare writes to a variable do not, bubbled or otherwise.

So -race composes with everything in this section, and the two are complementary rather than alternative: the bubble gives you a deterministic schedule, the detector checks the ordering of the memory accesses that schedule produced.

16.4.10 Common Mistakes

Summary: Determinism: testing/synctest

The package is three functions. Test runs a closure in a bubble; Wait blocks until every other bubbled goroutine is durably blocked; Sleep (Go 1.27) advances the clock and then Waits. Run, from the Go 1.24 experiment, no longer exists, and most published examples still use it.

Inside a bubble the time package is fake and per-bubble, starting at midnight UTC 2000-01-01, and it advances only when every goroutine is durably blocked — jumping straight to the next timer rather than waiting. Measured, thirty seconds of virtual time in 0.00s of wall clock, with time.Since reporting exactly 30s, which turns tolerance-band assertions into equality assertions.

“Durably blocked” means only a goroutine in the same bubble can wake you. Bubbled channel operations, Cond.Wait, bubbled WaitGroup.Wait and time.Sleep qualify. Mutexes, I/O and syscalls do not — and the consequence is sharper than a missing feature. Measured, a contended mutex inside a bubble hangs until the test binary’s alarm fires, because the bubble’s deadlock detection has the same blind spot as its clock. The socket boundary is the same rule in its most common form, already measured in §15.7.7, and net.Pipe — or, for HTTP, Go 1.27’s httptest.NewTestServer — is the way through.

synctest.Wait makes “this has not happened yet” a testable assertion for the first time. And measured, -race still reports races inside a bubble: determinism and detection compose.

Self-Check Questions: Determinism: testing/synctest

16.5 Leak Detection: Heuristic and Proof

Chapter 2 introduced goleak and built an exercise around it. Chapter 10 used it again in the deadlock-detection toolkit. Both chapters pointed here, and Chapter 2's Further Reading was specific about what was owed:

“Properly” is the whole assignment. Anyone can add defer goleak.VerifyNone(t) to a test; the reason that is not enough is that goleak answers a question it cannot actually answer, and the gap between what it reports and what is true has a measurable size.

16.5.1 What Chapter 2 Left On the Table

goleak works by comparing goroutine profiles. It takes a snapshot, filters out the ones it knows about, and reports whatever is left.

// Illustrative snippet — not a complete program
func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m)   // once, for the whole package
}

func TestOne(t *testing.T) {
    defer goleak.VerifyNone(t) // or per-test
}

That much is Chapter 2. The choice between them is about blast radius: VerifyTestMain runs one check after the entire package, which is cheap and catches everything, but tells you only that something in the package leaked. VerifyNone runs per test and names the culprit, at the cost of a check per test — and §16.5.2 shows what that cost actually is.

The part Chapter 2 could not cover, because it needed the vocabulary of this chapter, is the nature of the answer. A goroutine profile is a snapshot of a moving system. A goroutine that is about to exit and a goroutine that will never exit look identical in it. There is no field that distinguishes them, and there cannot be — the difference is about the future.

So goleak cannot observe the property it is named after. What it does instead is wait, and retry, and eventually give up. That makes it a heuristic with a tunable threshold, and everything surprising about it follows from where that threshold sits.

16.5.2 goleak Is a Retry Loop

The threshold is not documented in the package’s README, but it is three constants in options.go:

// Illustrative snippet — not a complete program
// options.go, verbatim -- the whole of goleak's patience.
const _defaultRetries = 20          // and maxSleep: 100ms

func (o *opts) retry(i int) bool {
    if i >= o.maxRetries {
        return false
    }
    d := time.Duration(int(time.Microsecond) << uint(i))
    if d > o.maxSleep {
        d = o.maxSleep
    }
    time.Sleep(d)
    return true
}

Twenty attempts. The delay starts at one microsecond and doubles, capped at one hundred milliseconds. Summing that schedule: attempts 0 through 16 contribute 1µs, 2µs, 4µs … 65.5ms, which totals about 131 ms; attempts 17, 18 and 19 are each capped at 100 ms, adding 300 ms.

That number is worth carrying around for a practical reason before a conceptual one. VerifyNone in a package with 200 tests, in the ordinary case where nothing leaks, is fast — the first profile is usually clean. But VerifyTestMain on a suite that does leak pays 431 ms once, and a per-test VerifyNone in a suite where one goroutine lingers pays it repeatedly.

16.5.3 The False Positive, Measured

The conceptual consequence is more important: the threshold is the entire definition. goleak does not report leaks. It reports goroutines that outlived 431 ms of patience.

That is testable directly. Start a goroutine that is definitely going to exit, at various delays, and ask goleak about it:

// Illustrative snippet — not a complete program
func TestGoleakFalsePositive(t *testing.T) {
    for _, d := range []time.Duration{
        100 * time.Millisecond,
        400 * time.Millisecond,
        600 * time.Millisecond,
    } {
        done := make(chan struct{})
        go func() { time.Sleep(d); close(done) }()
        err := goleak.Find()
        t.Logf("exits after %-6v -> leak: %-5v", d, err != nil)
        <-done
    }
}

A clean cliff edge, exactly where the arithmetic put it. Not one of these three goroutines is a leak — all three exit, on their own, without intervention. The third is reported as one because it took 600 ms and the tool waits 431.

attempt  0 ─ 16       1us, 2us, 4us ... 65ms      ≈131ms
attempt 17 ─ 19       100ms each (capped)         ≈300ms
                                                  ───────
                                        total     ≈431ms

a goroutine that exits after
    100ms   ──▶   not reported        ✓ correct
    400ms   ──▶   not reported        ✓ correct
    600ms   ──▶   REPORTED            ✗ false positive

None of the three is leaking. The third is only slower
than the tool is patient.

This is what “heuristic” means in practice, and it explains every goleak false positive anyone has ever filed, without needing a catalogue of special cases. The usual suspects — http.Transport's idle connections, a pending time.AfterFunc, a package’s initialisation goroutine, a background flusher on a ticker — are not separate phenomena. They are all the same phenomenon: something that will exit, or would exit if asked, taking longer than 431 ms to do it.

It also explains the property that makes goleak failures so unpleasant to debug: they get worse on slower machines. A goroutine that exits in 300 ms on a developer laptop can take 600 ms on a loaded CI runner, and the tool flips from silent to failing with no change to the code. The same test, the same binary, a different verdict — which is §16.1's opening problem, arriving inside the tool meant to solve it.

16.5.4 Counting Goroutines by Hand

Chapter 14's exercise counted goroutines with runtime.NumGoroutine() “so the mechanism stays visible”, and Chapter 15's fourth gate did the same. Now that the mechanism is visible, here is why neither chapter recommended it as a practice.

// Illustrative snippet — not a complete program
// ✗ BROKEN as a general technique
before := runtime.NumGoroutine()
doTheThing()
if after := runtime.NumGoroutine(); after != before {
    t.Errorf("leaked %d goroutines", after-before)
}

Three problems, in increasing order of how long they take to diagnose.

It races with correct shutdown. A goroutine that is exiting but has not yet been descheduled still counts. That is goleak's problem from §16.5.3 without goleak's 431 ms of patience — the same false positive, arriving sooner and far more often.

It counts the whole process. The test framework’s goroutines, the garbage collector’s, anything a background package started at init, and — crucially — anything a parallel sibling test started or finished during your measurement. Under t.Parallel the number is close to meaningless.

It reports a number, not a stack. “Leaked 1 goroutine” gives you nothing to act on. goleak prints the offending stack; the bubble prints the source line that created it.

What hand-counting has going for it is that it is dependency-free and completely obvious, which is exactly why the two earlier exercises used it — the goal there was to make a leak visible, not to establish a practice. In a real suite, use goleak or a bubble.

16.5.5 Ignoring, and When It Is a Lie

The tool’s answer to false positives is to ignore things.

// Illustrative snippet — not a complete program
func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m,
        goleak.IgnoreTopFunction("internal/poll.runtime_pollWait"),
        goleak.IgnoreAnyFunction("example.com/stats.(*worker).start"),
    )
}

IgnoreTopFunction matches on the goroutine’s topmost frame; IgnoreAnyFunction matches anywhere in the stack; IgnoreCurrent snapshots what is running now and excludes all of it, which is the blunt instrument for a package with unavoidable background goroutines.

These are necessary and there is nothing wrong with using them. But each one is an assertion, and it is worth being explicit about what you are asserting: “a goroutine sitting in this function is not a leak.”

That claim is sometimes true and sometimes a lie, and the test that tells them apart is not whether the failure went away. It is whether you can answer two questions about the goroutine:

• Who owns it? Name the object or subsystem responsible for its lifetime.
• What makes it exit? Name the specific event — a closed channel, a cancelled context, a Close call.

If you can answer both, the ignore is honest: you have a goroutine with a real lifecycle that happens to be slower than 431 ms or genuinely process-lifetime. If you cannot, you have just told the tool to stop reporting a leak you do not understand — and IgnoreTopFunction on a frame like runtime_pollWait is especially prone to this, because it covers every goroutine blocked on any network read anywhere in the program, including the one that is actually leaking.

16.5.6 The Bubble’s Version

Everything above is a consequence of not knowing whether a goroutine will exit. Inside a bubble, the runtime does know.

§16.4.3 established the mechanism: when every goroutine in the bubble is durably blocked, the bubble advances the clock to the next timer — and if there is no timer, then no future exists in which anything makes progress. That is a deadlock, definitionally, and it is detectable rather than heuristic.

Test uses this in two ways:

So a bubbled goroutine that never exits is not a goroutine that survives the test. It is a goroutine that stops the test from finishing — and the bubble notices immediately.

// Illustrative snippet — not a complete program
func TestBubbleLeak(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        ch := make(chan int)
        go func() { <-ch }()   // nobody will ever send
    })
}

Compare that against goleak, line by line. It took 0.00s rather than 436 ms. It required no dependency. It reported a proof rather than a threshold verdict — there is no interpretation under which that goroutine was going to exit. And it named the source line where the goroutine was created, which is the information you actually need and which a goroutine profile gives you only indirectly.

The goroutine state annotation is worth reading closely: [chan receive (durable), synctest bubble 2]. The runtime tags both the durability of the block and the bubble identity in ordinary goroutine dumps — and, from Go 1.27, any pprof labels the goroutine carries, after the bracket, for modules declaring go 1.27 or later. When you are debugging a bubbled test that is behaving strangely, kill -QUIT (§10.4) shows you exactly which goroutines the bubble considers durably blocked — which, per §16.4.4, is the first thing you want to know.

16.5.7 The Leak That Is Armed But Not Running

It would be tidy to stop there, and it would be wrong. The bubble has a blind spot of its own, and it is worth measuring because the obvious mental model gets it backwards.

// Illustrative snippet — not a complete program
// The leak is armed, not yet running.
func armLater() {
    time.AfterFunc(2*time.Second, func() { select {} })
}

At the moment the test ends, nothing is leaking. Something is scheduled to.

Both tools miss it, for the same reason from two directions: neither was asked about the future. Now add one line that costs nothing:

// Illustrative snippet — not a complete program
func TestBubbleCatchesArmedLeak(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        armLater()
        time.Sleep(3 * time.Second)   // virtual: free (synctest.Sleep)
    })
}

Still 0.00s of wall clock. The rule that falls out is short and almost nobody writes it down:

16.5.8 Adopting Detection in a Suite That Already Leaks

Everything above assumes a clean starting point. Most real suites do not have one: the first time anyone adds goleak.VerifyTestMain, thirty tests fail, and the change is reverted within the hour. That outcome is avoidable, and the sequence matters.

Do not start with VerifyTestMain across the whole tree. It produces a single failure with no attribution and a wall of goroutines, most of which turn out to be the same two causes.

Start at the leaves instead — the packages that own goroutines and have the fewest dependencies — one at a time, with an explicit exception list:

// Illustrative snippet — not a complete program
func TestMain(m *testing.M) {
    goleak.VerifyTestMain(m,
        // Owner:   package-level pool, created in init
        // Exits:   never — process lifetime
        // Tracked: ISSUE-4412
        goleak.IgnoreTopFunction("myapp.(*Pool).reap"),
    )
}

Every ignore carries three facts in a comment: owner, exit condition, tracking reference. That is §16.5.5's honesty test, written down where the next person will read it. An ignore with all three is documentation. An ignore with none is where the leak you are looking for will eventually hide.

Then work the list. Two causes account for most of a first run:

A test that starts a goroutine and returns. §16.2.2's orphan. The fix is t.Cleanup(wg.Wait), and it usually fixes several tests at once.

http.Transport idle connections. Not really a leak — the transport is holding a keep-alive connection open on purpose. Give each test its own client with t.Cleanup(client.CloseIdleConnections), or use httptest, whose Close handles it.

Finally, adopt in the order that preserves signal: VerifyTestMain in the package to prove it is clean, VerifyNone on the tests that own goroutines so future regressions are attributed to the right test, and a bubble for anything that can move into one.

16.5.9 Choosing

Two tools with a clean division, and the boundary is §16.4.8's boundary rather than a matter of preference — plus, since Go 1.27, a third instrument that sits between them.

Go 1.27 adds a third instrument that sits between the two. The runtime’s goroutineleak profile — pprof.Lookup("goroutineleak"), or /debug/pprof/goroutineleak — reports goroutines blocked on a channel, mutex, Cond or similar primitive that the garbage collector can prove no runnable goroutine will ever reach. That is a proof in the bubble’s sense (no threshold to tune, no Ignore list), it covers the whole process in goleak’s sense, and it costs nothing until you ask for it. Its blind spots are the mirror of the bubble’s: it cannot judge a goroutine parked on I/O or a syscall, and it can miss a primitive still reachable through a global or a runnable goroutine’s locals — so it is an instrument for the socket-holding boundary and for production, not a replacement for either column. It shares the bubble’s other blind spot too: §16.5.7’s goroutine that is armed but not yet running is parked on nothing, so a reachability walk cannot see it either. And §16.5.8’s suite that is too leaky to gate on is a good fit for it, because a profile you read is not a check that fails. Experimental in Go 1.26 behind GOEXPERIMENT=goroutineleakprofile, standard in 1.27; Chapter 19 (§19.3.6) covers how it works.

The rule that follows: use the bubble for everything you can put in one, and goleak for the boundary — and, from Go 1.27, the goroutineleak profile wherever you want the runtime’s own proof, including in production (Chapter 19). Concretely, in a package that has both kinds of code:

• Logic that coordinates through channels, contexts and timers goes in bubbles, and gets leak detection for free with no extra line of test code
• Code that holds sockets, files or subprocess handles is tested outside, and goleak.VerifyTestMain guards the package as a whole
• The Ignore options apply only to the second category, where they belong — inside a bubble there is nothing to ignore, because there is no threshold to be wrong about

That last point is the one worth internalising. Every Ignore in your codebase exists because a tool had to guess. Moving code into a bubble does not make the guessing more accurate; it removes the need to guess.

16.5.10 Common Mistakes

Summary: Leak Detection: Heuristic and Proof

goleak cannot observe whether a goroutine will exit, because a goroutine about to exit and one that never will look identical in a profile. So it waits: twenty retries with an exponential backoff capped at 100 ms, which is a derived budget of about 431 ms and a measured 436 ms against a real leak.

That threshold is the whole definition, and it produces a clean cliff. Measured, goroutines exiting after 100 ms and 400 ms were not reported; the same goroutine exiting after 600 ms was reported as a leak. Every known goleak false positive is that one phenomenon wearing different clothes, and it explains why the tool gets less reliable as machines get slower — the same failure mode §16.1 opened with, inside the tool meant to fix it.

The Ignore options are necessary and each one is an assertion. The honest test is whether you can name the goroutine’s owner and its exit condition; if you cannot, you have silenced a leak you do not understand.

Inside a bubble none of this applies, because the runtime knows. Measured, a bubbled goroutine parked on a channel nobody will send to produced an immediate panic: deadlock, in 0.00s, naming the creation line, with the goroutine tagged [chan receive (durable), synctest bubble 2]. Proof rather than threshold. Use the bubble wherever code will fit in one and goleak for the socket-holding boundary that will not.

Self-Check Questions: Leak Detection: Heuristic and Proof

16.6 Seams: Making Time, Schedule and Failure Injectable

A bubble is the best tool in this chapter and it does not reach everything. Anything holding a socket, a file, a subprocess or a contended mutex stays outside it, which in most real services is a substantial fraction of the interesting code. §16.1.5's ladder called that rung 2: you cannot own the clock, so you own the ordering instead.

A seam is a place where a test can substitute its own decision for one the program would otherwise make. Michael Feathers named the idea for sequential code; concurrency adds three things worth seaming that sequential code does not have — when something happens, in what order things happen, and which of several possible failures occurs first.

16.6.1 Clock Seams

The oldest technique in this section, and still necessary, because a bubble cannot reach code that talks to a socket while it waits.

The seam is an interface with one method:

// Illustrative snippet — not a complete program
type Clock interface {
    Now() time.Time
    After(d time.Duration) <-chan time.Time
}

type realClock struct{}

func (realClock) Now() time.Time { return time.Now() }
func (realClock) After(d time.Duration) <-chan time.Time {
    return time.After(d)
}

The code under test takes a Clock; production passes realClock{}; the test passes something it can drive. This is the standard shape and there are several good third-party implementations of the fake side, so the interesting question is not how to write one but when it is still the right answer now that virtual time exists.

Two cases, and they are narrower than they used to be:

When the component must hold a real connection while it waits. A connection pool that expires idle connections after five minutes genuinely needs a socket in the test to be worth anything, so its timer cannot be a bubbled timer. A clock seam lets the test jump five minutes without waiting, while the connection stays real.

When the code is not yours to change. A dependency that calls time.Now internally cannot be bubbled usefully — the bubble will supply a fake clock, but the dependency’s own timers and the code’s expectations may be built around wall-clock assumptions the bubble breaks in confusing ways. A seam at your boundary is more predictable than a bubble around someone else’s code.

Outside those, prefer the bubble. A clock seam costs you an interface in your production API, an implementation to maintain, and the risk that the fake and the real clock diverge in behaviour — while a bubble costs nothing and is exact.

16.6.2 Schedule Seams

The more common need. The test does not want to control when, it wants to control what happens before what.

The wrong tool is a sleep, and the reason is §16.1.4: a sleep asserts on a schedule. The right tool is a channel that makes the ordering explicit, and there are three shapes worth naming because they get confused with each other.

A one-shot gate blocks a goroutine until the test lets it proceed:

// Illustrative snippet — not a complete program
gate := make(chan struct{})
go func() {
    <-gate           // wait for the test's permission
    doTheThing()
}()
// ... set up whatever must be true first ...
close(gate)          // release it

A barrier blocks the test until a goroutine has reached a point:

// Illustrative snippet — not a complete program
reached := make(chan struct{})
go func() {
    setup()
    close(reached)   // announce arrival
    doTheThing()
}()
<-reached            // the test waits here

A rendezvous is both: each side waits for the other, which is what an unbuffered channel does natively.

Chapter 15's exercise needed the second kind and the reason generalises. Its handler recorded whether it had been told to stop, and the test read that record — but the handler wrote it from its own goroutine, and the shutdown report could arrive first. The fix was a returned channel that the handler closes on exit, received from before reading the record, and the reason is written into that exercise’s own source:

The fix was not a longer sleep. It was making the ordering explicit, so that the read cannot happen before the write regardless of how the scheduler feels.

The temptation to reach for something cheaper is strong, and there is one candidate that looks like it should work:

// Illustrative snippet — not a complete program
// ✗ BROKEN: Gosched is not synchronisation
func TestGoschedIsNotSync(t *testing.T) {
    done := false
    go func() { done = true }()
    runtime.Gosched()
    if !done {
        t.Fatal("the goroutine had not run")
    }
}

runtime.Gosched yields the processor, which sounds like “let the other goroutine run”. It offers no guarantee that any particular goroutine runs, let alone that it finishes.

Two to three percent is the worst possible number. It is low enough that the pattern looks like it works, and high enough that a suite running it a few hundred times will fail somewhere, occasionally, with no pattern. It is §16.1.1's 39-out-of-40 problem, self-inflicted.

The rule: Gosched is a scheduling hint, never a happens-before edge. If you need one goroutine to have finished, use a channel. The race detector agrees — a Gosched between two conflicting accesses does not stop -race reporting them, for exactly the reason §16.3.2 gave.

16.6.3 Timeouts, Honestly

Some tests must use real time. The question is how to do it without building §16.1's flakiness back in.

The answer is an asymmetry that §16.1's self-check introduced and that is worth stating as a rule:

ASSERTION                 SLOW MACHINE    VERDICT
──────────────────────────────────────────────────────────
"finishes within X"        → fails        ✗ unsound
(X as synchronisation)                      as ordering

"has not finished          → still true   ✓ sound
 after X"                                   (absence)

"eventually finishes;      → slower       ✓ sound
 X is the giving-up           but the       as a failure
 point"                       same verdict  detector
──────────────────────────────────────────────────────────

A timeout is a failure detector. It is never a device for
establishing ordering.

The middle and bottom rows are both fine and they are different. The bottom row is the one you use constantly: a select with a generous timeout arm, where reaching the timeout means the test fails. Its only cost on a slow machine is that a failing test takes longer to fail, and passing tests are unaffected because they never reach the arm.

That yields two practical rules. Make failure timeouts generous — seconds, not milliseconds. A 50 ms timeout and a 5 s timeout catch the same bugs; the first also catches a busy CI runner. And never derive an ordering from a timeout, which is what “sleep 100 ms, then assert it happened” does.

One standard-library behaviour is worth knowing here because it removes a timeout you might otherwise write.

That makes httptest a natural bridge from Chapter 15's subject: the test server does the graceful-shutdown wait for you, so a test that wants to assert “the handler completed before the server went away” needs no timing logic at all — just defer srv.Close() and an assertion afterwards. It is also a trap in the other direction: a handler that blocks forever makes Close block forever, turning a handler bug into §16.1.3's hang.

16.6.4 Testing Error Paths

Chapter 14 built error handling for concurrent code and named this as its outstanding debt:

Error paths are the hardest thing in this chapter to test, and the reason is specific. Success has one shape: everything worked. Failure has many, and which one you get is decided by a race — so an error-path test is a schedule assertion unless you make it something else.

The sharpest case is errgroup. §14.4 established that Wait returns the first non-nil error, where “first” means first to return, not most important. A test that asserts on which error comes back is therefore asserting on a schedule.

A 99-to-1 split. Not a coin toss — and that is worse, because a coin toss is obviously non-deterministic and a 198-to-2 split looks stable. A test written against the majority outcome passes 99% of the time, ships, and fails in CI at a rate low enough to be dismissed as flake. §16.1.1's problem again, now in the code that verifies error handling.

Compare that result with §16.1.6's, which printed the opposite winner from what looks like the same experiment. Both are correct, and the reason is worth knowing because it explains why the split is so lopsided rather than even.

The second goroutine registered wins, about 98–100% of the time, whichever error it carries. The mechanism is in the scheduler. newproc — the runtime function behind every go statement — finishes with runqput(pp, newg, true), and that final true means put this goroutine in the P’s runnext slot. runnext holds exactly one goroutine and it runs next, so each new goroutine displaces the previous one into the ordinary local queue.

Two g.Go calls in a row therefore do something that looks unfair and is entirely deterministic: the second one takes runnext, the first is pushed back, and the second runs first.

That is the whole explanation for the lopsidedness. There is no near-coin-toss here to be sampled — there is a structural bias with a little noise on top, which is precisely why the test looks reliable for fifty runs. §16.1.6's five parties were not unlucky; they were reading a real pattern and assuming it was a guarantee.

The fix is a seam that forces the order:

// Illustrative snippet — not a complete program
g, _ := errgroup.WithContext(context.Background())
g.Go(func() error { return errFast })
g.Go(func() error {
    time.Sleep(time.Millisecond)   // ordering, not synchronisation
    return errSlow
})
err := g.Wait()

That listing deserves a caveat, because it appears to contradict §16.6.2. The sleep here is not being used to establish that something has happened — it is being used to make one path deliberately slower than another, and the assertion is about which of two errors is selected. A slow machine slows both. It is the sound use, and it is still worth replacing with a gate when the code under test offers one:

// Illustrative snippet — not a complete program
release := make(chan struct{})
g.Go(func() error { return errFast })
g.Go(func() error { <-release; return errSlow })
// ... assert Wait() == errFast ...
close(release)

Three more error-path shapes, with the seam each needs:

Proving a pipeline error cancels upstream. The invariant is that the producer stops. Seam: give the producer a counter or a channel it writes to per item, assert the count stops growing after the consumer fails. Inside a bubble, synctest.Wait makes this exact (§16.4.5).

Proving context.Cause names the right failure. §14.4's WithCancelCause means the cause is set once, by the first failure — so the test needs the same ordering seam as the errgroup case, and then asserts on context.Cause(ctx) rather than on Wait's return.

Asserting on a panic recovered in a goroutine. Per §16.2, the assertion cannot live in the goroutine that recovers. The recovering goroutine sends the recovered value over a channel and the test goroutine asserts on it:

// Illustrative snippet — not a complete program
panicked := make(chan any, 1)
go func() {
    defer func() { panicked <- recover() }()
    riskyWork()
}()
if got := <-panicked; got == nil {
    t.Fatal("expected a panic")
}

Note the buffered channel and the unconditional send — recover() returns nil when there was no panic, so the test distinguishes “panicked with nil” from “did not panic” by always sending exactly once.

16.6.5 Fault Injection Without a Framework

A clock seam controls when; a schedule seam controls what order; the third seam controls what goes wrong. It needs no library, and the pattern is one closure.

// Illustrative snippet — not a complete program
// failN fails the first n calls, then succeeds. Atomic because the
// code under test may call it from any goroutine.
func failN(n int64) func() error {
    var calls atomic.Int64
    return func() error {
        if calls.Add(1) <= n {
            return errors.New("upstream unavailable")
        }
        return nil
    }
}

Point that at a retry with 1s/2s/4s backoff, inside a bubble, and a claim that would otherwise need seven real seconds and a tolerance band becomes an equality assertion:

// Illustrative snippet — not a complete program
func TestRecoversOnThirdAttempt(t *testing.T) {
    synctest.Test(t, func(t *testing.T) {
        start := time.Now()
        if err := Do(t.Context(), failN(2)); err != nil {
            t.Fatalf("Do = %v, want nil", err)
        }
        got, want := time.Since(start), 3*time.Second
        if got != want {
            t.Fatalf("recovered after %v, want %v", got, want)
        }
    })
}

The same closure shape covers most of what people reach for a chaos library to do. Three come up constantly:

A failing io.Reader. Wrap a real one and return an error after N bytes. This is how you test that a partial read is handled rather than assumed away.

An http.RoundTripper that fails on demand. http.Client{Transport: rt} with an rt you wrote is the single most useful seam in a service, because it turns every network failure — timeout, connection reset, 503, a body that stops mid-stream — into a value you choose rather than an outage you wait for.

A net.Conn from net.Pipe. §16.4.8's escape hatch, and it lets you close one side at a moment you pick, which is how you simulate a peer disappearing.

One rule applies to all of them: the injector’s own state must be safe for concurrent use. failN uses atomic.Int64 rather than a plain counter for exactly that reason. Getting it wrong produces a data race in your test, which -race will duly report — with a stack that points into the code under test, sending you to debug the wrong file.

16.6.6 Table-Driven Concurrent Tests

The standard Go idiom, with the concurrency-specific parts that actually matter in 2026.

// Illustrative snippet — not a complete program
func TestPolicy(t *testing.T) {
    cases := []struct {
        name    string
        workers int
        want    int
    }{
        {"single", 1, 100},
        {"parallel", 8, 100},
    }
    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            got := run(tc.workers)
            if got != tc.want {
                t.Errorf("got %d, want %d", got, tc.want)
            }
        })
    }
}

What may vary between cases: the concurrency level, the order operations are submitted in, the timing seam’s schedule. Those are the axes worth tabulating, and varying concurrency level across cases is the single most valuable column — running the same assertion at 1, 2 and 8 workers catches a surprising number of bugs that a fixed count does not.

What must not vary, and this is the failure mode: a shared fixture. If the cases run in parallel and touch the same object, the table has stopped being a set of independent tests and become one concurrent test with a confusing structure. Each case needs its own instance, created inside the subtest.

The cleanup interaction from §16.2.3 applies with full force here, because table-driven tests are where t.Parallel gets used most:

// Illustrative snippet — not a complete program
// ✗ BROKEN: every parallel subtest sees a closed store
func TestPolicy(t *testing.T) {
    store := open()
    defer store.Close()
    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            t.Parallel()
            store.Query(tc.input)   // store is already closed
        })
    }
}

t.Cleanup(store.Close) fixes it — but the better fix is usually one store per subtest, which removes the sharing that made the ordering matter.

Two smaller notes. t.Setenv anywhere in the loop body means no case can call t.Parallel (§16.2.3), which is a common surprise in tables that configure behaviour through environment variables. And a bubble cannot contain t.Run, so a table of synctest cases puts the loop outside:

// Illustrative snippet — not a complete program
func TestPolicyTable(t *testing.T) {
    for _, tc := range cases {
        t.Run(tc.name, func(t *testing.T) {
            synctest.Test(t, func(t *testing.T) {
                // one bubble per case
            })
        })
    }
}

And the bug that is not on this list: loop-variable capture. Before Go 1.22 the loop variable was shared across iterations, so subtests that ran after the loop finished — which is exactly what t.Parallel causes — all saw the final case. Go 1.22 gave each iteration its own variable and the bug is gone. It remains the most-warned-about hazard in table-driven concurrent testing, which now costs readers attention they should be spending on shared fixtures.

16.6.7 Common Mistakes

Summary: Seams: Making Time, Schedule and Failure Injectable

Everything a bubble cannot contain is tested by owning the ordering instead of the clock. A seam is where the test substitutes its own decision, and concurrency adds three things worth seaming: when something happens, in what order, and which failure occurs first.

Clock seams still matter, but in a narrower range than before — when a real connection must stay open across the wait, or when the code is not yours to change. Otherwise a bubble is exact and free. context.AfterFunc is often better than either, because cancellation is testable without any clock at all.

Schedule seams are gates, barriers and rendezvous, and the thing they replace is the sleep. runtime.Gosched is not a substitute: measured, the value was invisible in 42–52 rounds out of 2000, a 2–3% failure rate that is exactly bad enough to look like it works.

Timeouts stay honest as long as they are failure detectors rather than ordering devices, and generous rather than tight. Measured, httptest.Server.Close blocks until outstanding requests finish — 250 ms for a request with 250 ms left — which removes a timing assertion rather than adding one.

Error paths are the hardest case because failure has many shapes and a race picks one. Measured, two errgroup goroutines failing simultaneously produced a 198-to-2 split, which is more dangerous than a coin toss because it looks deterministic; forcing the order made it 200-to-0.

Self-Check Questions: Seams: Making Time, Schedule and Failure Injectable

16.7 Stress: What More Runs Actually Buy

“Run it a thousand times and see if it breaks” is the oldest advice in concurrent testing, and it is given without qualification often enough that it has become folklore. It is worth keeping, but not for the reason usually offered — and the difference decides whether a nightly stress job is finding bugs or burning CPU.

16.7.1 The Reframe

§16.3.3 measured the fact that reorganises this section: fifty -race runs of a covered path found nothing, and one run of an uncovered path found the race immediately.

That result rules out the intuitive model. Repetition does not make the race detector more sensitive, because the detector is not sampling — it reports every unordered pair of accesses that executes, on the first run they execute (§16.3.2). Running the same code path again gives it nothing it did not already have.

So what does -count=1000 buy?

Interleavings, for the bugs -race cannot see. §16.3.4's check-then-act withdrawal has no data race and drove a balance negative in 5–13 rounds out of 200. That bug is invisible to the detector at any coverage level, and it is a function of scheduling — so more runs, on more cores, genuinely does increase the chance of observing it.

Paths, when the repetition varies something. -count alone re-runs identical code. -cpu changes GOMAXPROCS, -shuffle changes test order and therefore the state each test inherits, and a randomised test body chooses different operations. Those are the flags that change what executes.

The distinction in one line: -race needs coverage; stress needs interleavings. They are different problems with different tools, and running -race -count=1000 on a package with 40% coverage spends a thousand runs on the same 40%.

With one addition from §16.3.2: because randomizeScheduler is on in a race build, -race -count=N explores a genuinely wider set of orderings than -count=N alone. That is worth nothing for data races — the analyser already sees those on the first run — and a great deal here, because the race build is the only scheduling fuzzer the toolchain ships and this is the half of the bug population that needs one.

16.7.2 The Knobs

Five flags, and what each actually varies.

Two of these deserve more than a row.

-cpu 1 is the most under-used flag in Go testing. Running at GOMAXPROCS=1 does not remove concurrency — goroutines still interleave at every preemption point — but it removes parallelism, and that changes which bugs are reachable. Some races need true parallelism and vanish; others become far more likely, because a goroutine that would have run on another core now has to wait for a scheduling point, widening windows that are microseconds wide at -cpu 16. Running both ends is worth more than running either one a hundred times. Chapter 1's exercise made this point at the very start of the book; it is the same lever.

-shuffle is how you find tests that depend on each other, which is a large fraction of “it only fails in CI” once the §16.2.2 orphans are dealt with. Crucially, it reports its seed:

That number is the whole point. A failure under -shuffle=on is reproducible with -shuffle=1788712755386749000, which pins the order exactly. Without capturing the seed a shuffled failure is nearly useless; with it, it is an ordinary bug report. Make sure your CI prints and preserves that line.

16.7.3 Triage: From Flake to Fix

The techniques above are only worth having if there is a path from “CI went red once” to a fix. Chapter 10 has an incident workflow for production; this is the test-suite equivalent, and the first step is the one people skip.

Capture before you do anything else. The full output, the seed if -shuffle was on, the iteration number, the -cpu value, the toolchain version, the commit. Re-running destroys all of it, and re-running is the single most common way a real finding is lost. Paste it into an issue before touching anything.

That instinct — run it again to see it — is exactly wrong. A failure that reproduces one time in five hundred will not survive the debugging process; every change made to observe it changes the timing, and adding a print statement is enough to hide it for a week. The one in front of you is likely the best evidence you will get for months.

Then classify it, using §16.1.6's tell. A plausible alternative — a different error, a reversed order, a count off by one — points at a schedule assertion, and the fix is in the test. Garbage values, a nil dereference or a torn struct point at a data race or a lifecycle bug, and the fix is in the program. The two get completely different treatment from here, so thirty seconds spent on this saves hours.

Then reproduce with the cheapest tool that works.

-count=1                 does it fail deterministically?
     │ no
-count=100 -race         a rare interleaving on a path
     │ no                  you already cover
-cpu 1, then -cpu 16     parallelism-dependent
     │ no
-shuffle=<seed>          order-dependent; needs a sibling
     │ no
-run <this test alone>   passes alone? then the bug is
                         BETWEEN tests ──▶ §16.2.2's
                         orphan is suspect one

The last rung is a diagnosis, not a failure to reproduce.

The last rung is worth stating plainly because it is so often misread as defeat. A test that fails in the suite and passes alone is telling you something specific: the bug is between tests. Shared global state, an orphaned goroutine from §16.2.2, a t.Setenv sibling, an unclosed listener on a fixed port.

Then build a deterministic reproduction, before fixing anything. This is the step that decides whether the rest of the work means anything, because a failure you can only reproduce statistically cannot be confirmed fixed — §16.7.5 puts a number on exactly how little a clean run buys you. The techniques in §16.4 and §16.6 exist to turn a statistical failure into a deterministic one: a bubble that pins the timing, a gate that forces the guilty ordering, a seam that makes the failing branch execute.

Then fix the class, not the instance. A schedule assertion means finding the invariant or placing a seam. A leak means adding the check that would have caught it at its source. A data race means asking which other paths have the same shape — a race in one method of a type is rarely alone.

Diagnosing the failure once it is captured and reproducible — reading the dump in depth, pprof, the execution tracer, delve — is Chapter 19's subject. This chapter’s job ends at a test that fails on demand.

16.7.4 Randomised Model Checking

One technique that finds a class of bug nothing else in this chapter reaches: race conditions in the §16.3.4 sense, where synchronisation is present but insufficient.

The idea is to stop asserting on specific outcomes and start asserting on an invariant, against a reference implementation that cannot be wrong.

// Illustrative snippet — not a complete program
func TestConcurrentMapAgainstModel(t *testing.T) {
    real := NewConcurrentMap()
    model := map[string]int{}   // the reference: a plain map
    var mu sync.Mutex           // guarding the model only

    ops := randomOps(t, 500)    // seeded from -shuffle or a flag

    var wg sync.WaitGroup
    for _, op := range ops {
        wg.Go(func() {
            switch op.kind {
            case put:
                real.Put(op.key, op.val)
                mu.Lock()
                model[op.key] = op.val
                mu.Unlock()
            case del:
                real.Delete(op.key)
                mu.Lock()
                delete(model, op.key)
                mu.Unlock()
            }
        })
    }
    wg.Wait()

    // The invariant: every key the model has, the real map has,
    // with the same value -- regardless of the order they ran in.
    for k, want := range model {
        if got, ok := real.Get(k); !ok || got != want {
            t.Errorf("key %q: got (%v,%v), want %v", k, got, ok, want)
        }
    }
}

Three properties make this worth the effort. The assertion is an invariant, so it holds under every interleaving and the test is not asserting on a schedule. The reference model is obviously correct, because it is a plain map under one lock — you are not testing two implementations against each other and hoping. And the operation sequence is randomised and seeded, so a failure is reproducible from the seed, exactly as with -shuffle.

The design rules that make it work: keep the model trivial, because a model with a bug produces failures you will chase in the wrong code; log the seed on failure without exception; and prefer commutative operation sets or an invariant that does not depend on order, since otherwise you must model the ordering too and the model stops being obviously correct.

This is a small, hand-rolled corner of a large field — property-based testing, linearisability checking, and formal model checkers all live here and all do more. The version above needs nothing beyond the standard library and finds check-then-act bugs that -race structurally cannot.

16.7.5 The Statistics of Zero Failures

A stress job passes. What has it established? The question has a numerical answer, and knowing it prevents a common and expensive mistake.

If a bug manifests with probability p on each run, then N clean runs are consistent with any p small enough to have plausibly hidden. The standard statistical result — the “rule of three” — is that observing zero failures in N independent trials gives a 95% confidence upper bound of roughly 3/N.

Read the second row carefully, because it is the one that matters in practice. One hundred clean runs are consistent with a bug that fires 3% of the time. That is a bug which would take down a service handling a hundred requests a second roughly three times a second.

clean runs        still consistent with a failure rate of
    10      ──▶     30%
   100      ──▶      3%
 1,000      ──▶      0.3%
10,000      ──▶      0.03%

Report the bound, not the run count. "It passed 1,000
times" sounds conclusive and is not; "we can rule out
rates above 0.3%" is the same fact, stated so a reader
can judge whether it is enough.

The same arithmetic run forwards says how many runs you need to have a reasonable chance of seeing a known-rare bug at all:

Now put that beside §16.1.1. That data race manifested at about 1.5% — three runs in two hundred. To catch it by repetition with 95% confidence you would need around 199 runs. Under -race, it was reported 10 times out of 10.

That comparison is the argument for this whole chapter in two numbers. Repetition is the most expensive way to find a concurrency bug and the least conclusive. Every other technique here — the detector, the bubble, a barrier, an invariant, a model — replaces a statistical argument with a structural one, and the structural one is both cheaper and stronger.

Which gives the honest way to report a stress result. Not “we ran it 1,000 times and it passed”, which sounds conclusive and is not, but “we ran it 1,000 times and can rule out failure rates above roughly 0.3%”. If 0.3% is not good enough for the system in question, the answer is not more runs — it is a technique that does not rely on runs.

16.7.6 Benchmarks as Correctness Tests

Benchmarking belongs to §11.2 for reading RunParallel output and to Chapter 20 for optimisation. Two things fall outside both.

A benchmark is a test. go test -race -bench . runs your benchmarks under the race detector, and benchmarks routinely reach code that tests do not: higher concurrency, larger inputs, longer runs, and the contended paths that only open under real load. Per §16.3.3 the detector’s limit is coverage, and a benchmark suite is coverage you already wrote. It is close to free to run occasionally and it finds things.

b.Loop supersedes b.N, and the reason is a correctness problem rather than an ergonomic one. Go 1.24 added it:

// Illustrative snippet — not a complete program
// The form every benchmark written before Go 1.24 uses:
func BenchmarkOld(b *testing.B) {
    for i := 0; i < b.N; i++ {
        result = expensive(i)   // needs a sink to survive
    }
}

// Go 1.24+:
func BenchmarkNew(b *testing.B) {
    for b.Loop() {
        expensive(1)            // kept alive automatically
    }
}

b.Loop resets the timer on its first call and stops it when it returns false, so setup and cleanup fall outside the measurement without ResetTimer/StopTimer bookkeeping. More importantly, the compiler keeps arguments, results and assigned variables inside the loop body alive, which retires the KeepAlive-and-sink folklore that every benchmarking guide teaches.

The function under test, so the numbers below are reproducible:

// Illustrative snippet — not a complete program
func mix(a, b int) int { return a*31 + b*17 }

Read rows one and two together, because that pair is the proof: calling mix costs exactly what calling nothing costs. 0.214 either way. The call is not being measured, it is being deleted, and the benchmark reports a number for work that never happened.

Row three shows why the sink was invented and why it is not free — assigning to a package variable keeps the call alive, and adds a store that the function itself never does, so the figure is now real but inflated.

Rows four and five are the surprise, and they correct an obvious misreading of the table. b.Loop reports the largest number not because it is the only form measuring the call, but because b.Loop has a per-iteration floor of roughly a nanosecond — the empty control costs 1.12, which is more than the mix measurement above it. At this scale mix is invisible inside that floor.

So the honest statement about b.Loop is narrower than the marketing and more useful: it is the only form that neither deletes your code nor charges you for a sink, and its floor of about a nanosecond means it cannot resolve operations faster than that. For anything sub-nanosecond, measure a batch per iteration and divide.

A benchmark that gets optimised away does not fail. It reports an impressively small number, and someone puts it in a commit message.

One incompatibility, and it is worth a mistakes row because the failure is confusing:

and 5 produced 15–16 repetitions of

The cause is that b.Loop manipulates the shared benchmark timer and RunParallel documents that the timer functions must not be used inside its body, because they have global effect. Which of the two symptoms you get depends on which parallel goroutine reaches the check first — a concurrency bug in the benchmarking harness for a chapter about concurrency bugs, whose own failure mode is non-deterministic. Inside RunParallel, keep using pb.Next().

16.7.7 What Runs When

Pulling §16.3.5's cost data and this section’s knobs into a policy.

The one non-negotiable is -race per PR on the packages that own concurrency. Per §16.3.5 it is most expensive on exactly that code, and per §16.1.1 it is the difference between a 1.5% symptom and a 10-out-of-10 report.

Beyond that, three properties matter more than the exact commands, and a pipeline that lacks any of them decays into decoration within a couple of months.

Every job has a named owner. A red nightly that belongs to nobody becomes background noise within two weeks, at which point you have converted a detectable bug into documented negligence — the failure is on record, visible, and ignored. That is strictly worse than not running the job, because it also manufactures the belief that stress testing is happening.

Failures carry their own reproduction. The job logs the seed, the iteration, the -cpu value and the full output automatically, per §16.7.3. If reproducing a nightly failure requires asking the person who wrote the job, it will not happen.

The expensive tiers cannot block a merge. The moment a weekly stress run gates a PR, someone will find a way to skip it, and they will be right to. That constraint is why the tiers exist at all: a single go test -race -count=100 on every push is worse than the whole pipeline in both directions — too slow to keep, and too shallow to find what the weekly run finds.

16.7.8 Common Mistakes

Summary: Stress: What More Runs Actually Buy

Repetition does not make the race detector more sensitive — measured in §16.3.3, fifty runs of a covered path found nothing that one run of an uncovered path did not find instantly. What -count buys is interleavings, which matters for the race conditions -race structurally cannot see, and nothing at all unless something varies between runs.

-cpu 1,2,4,16 varies parallelism and is the most under-used flag in Go testing; -shuffle=on varies order and measured, prints its seed as -test.shuffle N, which is what makes a shuffled failure reproducible rather than merely interesting.

When stress finds something, capture before investigating, and build a deterministic reproduction before fixing — because derived, confirming a fix for a 0.1% bug by repetition needs about 3,000 runs. The rule of three sets the honest bound: 100 clean runs are consistent with a 3% failure rate, and 1,000 with 0.3%. Against §16.1.1's 1.5% data race, repetition needs ~199 runs for 95% confidence where -race reported it 10 times out of 10.

Randomised model checking — random operations against an obviously-correct reference, asserting an invariant — reaches check-then-act bugs nothing else here does, using only the standard library.

And benchmarks are tests: measured, a discarded call under b.N cost 0.214 ns/op — exactly what an empty loop costs, which is what deletion looks like — while b.Loop keeps it alive at the price of a ~1 ns per-iteration floor.

Self-Check Questions: Stress: What More Runs Actually Buy