Chapter 21: Scheduler and Runtime Internals

21.1 Five Things You Have Already Seen

21.1.1 The Artifacts

This chapter does not open with a diagram of the scheduler, because you have already seen one in §1.4 and it did not let you answer any of the following.

One. The uneven run queues. §19.4.3 printed per-processor queue depths under GODEBUG=schedtrace and observed that processors 7 and 8 held 35 and 34 goroutines while processor 6 held none. It called that “work that is not being stolen effectively”. That explanation is wrong, and §21.3 replaces it — along with the belief about work stealing that made it seem right.

Two. More threads than processors. A schedtrace line reads gomaxprocs=16 … threads=19. Chapter 1 raised exactly this and deferred it — §1.4 notes that the runtime “may create additional OS threads when goroutines block on syscalls” and hands the detail to §1.5, and §1.3 says that when a goroutine blocks a thread “the runtime detects this and creates an additional thread so other goroutines keep running.” Neither says how many, which kinds of blocking count, or whether the threads come back. §21.4 answers all three, and the gap gets much wider than three.

Three. Two different costs for one goroutine. §2.5 says a goroutine starts with about 2 KB of stack. §14.4.4 measured 2.7 KB apiece for ten thousand parked goroutines. Both numbers are correct and they are measuring different things; §21.6 says which to use when.

Four. A goroutine that would not yield. §19.4.4 described a goroutine the runtime could not stop, and §10.4's table of what does and does not trigger the deadlock detector has rows for channels, mutexes, WaitGroup, select, sleeps and I/O — and no row for a goroutine that is simply running. §21.5 adds the row.

Five. A channel in the mutex profile. §5 taught that an unbuffered channel is a synchronisation point. §19.3.3 found channel operations attributed in the mutex profile. §21.6 explains why both are true, and corrects a plausible-sounding conclusion about channel performance along the way.

WHAT WAS PRINTED           WHAT EXPLAINS IT

 uneven per-P      ──►  21.3  and the explanation
 queues (19.4.3)              it was given is wrong
 threads=19 on     ──►  21.4  promised by Chapter 1
 gomaxprocs=16
 2 KB vs 2.7 KB    ──►  21.6  both correct
 a goroutine that  ──►  21.5  and a row is missing
 would not yield              from 10.4's table
 a channel in the  ──►  21.6  a channel is a lock
 mutex profile

21.1.2 The Test This Chapter Applies to Itself

An internals chapter can go wrong in one specific way: it becomes a tour of runtime source that the reader admires and never uses. The defence is a rule, stated up front and applied throughout:

That is also the honest answer to “when does internals knowledge matter?” — which your instinct probably files as the last section of a chapter like this. It matters at the boundaries where the abstraction stops being free, and every section below is one of those boundaries.

21.1.3 The Instrument

Chapter 19 used GODEBUG=schedtrace and mentioned its louder sibling in a single clause. That sibling is this chapter’s instrument, because it prints the model rather than describing it.

SCHED 105ms: gomaxprocs=16 idleprocs=7 threads=12
             spinningthreads=1 idlethreads=1 runqueue=0
             sysmonwait=false          [4 fields elided]
  P0:  status=1 schedtick=9 syscalltick=1 m=2 runqsize=0
       gfreecnt=0 timerslen=0
  P1:  status=0 schedtick=0 syscalltick=0 m=nil runqsize=0
       gfreecnt=0 timerslen=0
  [... P2 through P14 ...]
  P15: status=1 schedtick=5 syscalltick=2 m=0 runqsize=0
       gfreecnt=0 timerslen=1
  M11: p=7   curg=nil [...] spinning=true  blocked=false
  M10: p=8   curg=17  [...] spinning=false blocked=false
  M8:  p=nil curg=nil [...] spinning=false blocked=true
  G1:  status=4(sleep) m=nil lockedm=nil

── every field in here is a section below ──►

Read that block and you have the chapter’s table of contents. Every number in it is explained by a section below — including the two that look odd: threads=12 against gomaxprocs=16 (§21.4), and one processor owning the only pending timer in the process while fifteen own none (§21.3.5).

Two of those are also available as always-on counters, which is §19.4's habit applied here: /sched/threads/total:threads reports the live OS thread count and /sched/gomaxprocs:threads reports the current GOMAXPROCS — both from runtime/metrics, both for a counter read, and the second is a value that now moves (§21.2).

21.1.4 How to Read the Source Without Drowning

Every claim in this chapter is cited to a file and a line, and you should check them — not out of suspicion, but because the checking is the skill, and it outlasts any particular fact.

Four habits make $GOROOT/src/runtime tractable.

Start from a symbol, not a file. proc.go is over seven thousand lines and reading it in order teaches nothing. Start from something you already know exists — findRunnable, stealWork, newproc, chansend — and follow only what it calls.

Find the callers before reading the function. A function’s meaning is mostly its call sites. grep -n "stealWork(" proc.go returns two lines: the declaration and one call. That one fact — stealing has a single caller, and it is inside the function a P enters when it has nothing to do — is the whole of §21.3.2, and it took a grep rather than a reading.

**The comments are unusually good, and they explain the why.** The runtime is commented for people who will maintain it, so the interesting comments are the ones justifying a choice rather than describing an action. The 61-tick check in §21.3.1 carries a comment naming the exact starvation it prevents; the n > s.size branch of a package in Chapter 17 carried one explaining why a doomed acquire is parked outside the queue. Those are the sentences worth reading twice.

Check the version, always. Go’s runtime changes, and it changed twice recently in ways that invalidate most of what is written about it elsewhere: preemption in 1.14, GOMAXPROCS in 1.25. A file-and-line citation is only true for the toolchain it was read on, which is why every one in this chapter names go1.27.1.

21.1.5 Common Mistakes

Summary: Five Things You Have Already Seen

This chapter opens with five artifacts printed in earlier chapters and left unexplained: §19.4.3's uneven per-P run queues and its wrong explanation of them, threads=19 against gomaxprocs=16, which Chapter 1 raised and deferred, §2.5's 2 KB against §14.4.4's 2.7 KB, §19.4.4's goroutine that would not yield against a row missing from §10.4's table, and channel operations appearing in §19.3.3's mutex profile.

The test the chapter applies to itself is that an internals fact earns its place only if it explains an observation or changes a decision. That is also the answer to when internals knowledge matters: at the boundaries where the abstraction stops being free.

The instrument is GODEBUG=scheddetail=1, which prints the model rather than describing it — every field in its output is a section of this chapter, and two of them are also always-on runtime/metrics counters.

Self-Check Questions: Five Things You Have Already Seen

21.2 G, M, P — and Which One You Run Out Of

§1.4 names all three. This section is about the question the names do not answer, which is the one that matters in production: there are three resources here, they are scarce in wildly different degrees, and running out of each one looks completely different.

21.2.1 Three Resources, Three Ceilings

G  ┌────────────────────────────────────┐ millions
   │ spend freely; the book has said so │
   └────────────────────────────────────┘
        ↓ each needs a P to run

P  ┌──────────────┐ GOMAXPROCS
   │ queueing is  │ running out here is
   │ normal       │ what a scheduler is FOR
   └──────────────┘
        ↓ each needs an M to execute

M  ┌────┐ 10,000
   │ ⚠  │ running out here is
   └────┘ fatal error: thread exhaustion

Read that last column as the whole point of the table. Running out of Gs is a memory problem you can see coming on a graph. Running out of Ps is what a scheduler is for — it is the normal operating condition of a busy service, and §21.3 is about how the runtime handles it. Running out of Ms is not a degradation at all.

// Illustrative snippet — not a complete program
prev := debug.SetMaxThreads(20) // default is 10,000
// ... 64 goroutines each entering a blocking syscall

Not an error return, not backpressure, not a slow degradation: fatal error, and the process is gone. The documentation is blunt about it — “If it attempts to use more than this many, the program crashes” — and it names the three causes in the same paragraph: goroutines “blocked in system calls, cgo calls, or are locked to other goroutines.” That list is §21.4.

21.2.2 Why P Exists At All

The G and the M are the obvious two: the work, and the thing that can run work. If that were the whole design you would have a scheduler with a queue of goroutines and a pool of threads pulling from it — which is exactly what Go had before version 1.1, and it did not scale.

The problem is the queue. One queue shared by every thread needs one lock, and that lock is taken on every scheduling decision — every channel operation that blocks, every goroutine that finishes, every go statement. At four threads it is contended. At sixty-four it is the program.

P is the fix, and it is a simple one: make the queue local. A P owns a run queue that only it touches in the common case, so the scheduling decision that used to take a global lock now takes none at all. The M has to hold a P to run Go code, which is what bounds parallelism — and the count of Ps is GOMAXPROCS, which is why that variable controls parallelism rather than thread count.

That single idea explains the whole of §21.3. Every mechanism there — the local queue, the fast slot in front of it, stealing, the periodic check of the global queue — exists because the queue was made local and the consequences had to be managed.

21.2.3 How Many Ps You Have, and Who Decided

Here is the most consequential fact in this chapter, and the one most likely to be stale in whatever you read before this.

The rule everyone learned is that GOMAXPROCS defaults to runtime.NumCPU() — the number of logical CPU cores. §1.5 has already retired that: it prints GOMAXPROCS = min(logical CPUs, affinity mask, cgroup quota) for Go 1.25 and later, and ends at “on a current toolchain, do nothing.”

The old rule has not been true since Go 1.25, and the documentation is where the precise version lives. go doc runtime.GOMAXPROCS on go1.27.1 says the runtime “selects an appropriate default value from a combination of the number of logical CPUs on the machine, the process’s CPU affinity mask, and, on Linux, the process’s average CPU throughput limit based on cgroup CPU quota, if any.”

Three inputs, not one. And three consequences that follow from it, each verified in the same documentation.

It is container-aware now. A process in a cgroup with a 2-CPU quota on a 64-core host gets 2, not 64. That is the problem §1.5 spends a diagram on and recommends a third-party library for — and the runtime now solves it.

It is gated on your own go.mod. GODEBUG=containermaxprocs=0 restores the old NumCPU behaviour, and “is default for language version 1.24 and below”. Two engineers running identical source on identical toolchains get different defaults if their modules declare different language versions. This is the detail most likely to waste an afternoon, and it is the one thing here you can check in one command — the gating is baked into the binary and go version -m prints it:

go.mod says: go 1.24
  $ go version -m ./prog
    build  DefaultGODEBUG=containermaxprocs=0,[...],
           updatemaxprocs=0
    (on go1.27 the list is longer; asynctimerchan,
     removed in 1.27, is no longer in it)

go.mod says: go 1.27
  $ go version -m ./prog
    (no DefaultGODEBUG line at all)

── same source, same toolchain, one go.mod line ──►

It moves while your program runs. “The Go runtime periodically updates the default value based on changes to the total logical CPU count, the CPU affinity mask, or cgroup quota.” If your container’s quota is raised or lowered — by an operator, by a vertical autoscaler, by a noisy-neighbour policy — GOMAXPROCS follows, without a restart.

And then the sharp edge, which turns §1.5's advice into a trap:

§1.5 keeps two cases where you still set it yourself — CPU shares with no quota, where there is nothing for the runtime to read, and a deliberate value below the quota. Both are now decisions to take knowingly rather than by default, and §21.7 is about the advice that did not survive the change.

21.2.4 Reading the Value Without Changing It

There is one more asymmetry, and it is in the same shape as a finding from Chapter 19.

runtime.GOMAXPROCS both reads and writes, distinguished by the sign of its argument. Measured, from $GOROOT/src/runtime/debug.go:

// Illustrative snippet — not a complete program
func GOMAXPROCS(n int) int {
    lock(&sched.lock)                 // :75
    ret := int(gomaxprocs)
    if n <= 0 {
        unlock(&sched.lock)
        return ret                    // :77-79 — returns FIRST
    }
    sched.customGOMAXPROCS = true     // :83
    unlock(&sched.lock)
    // ... then the no-op check at :90, and the STW at :96
    return ret
}

GOMAXPROCS(0) is a pure read. The n <= 0 branch returns before the flag is set, so querying the value does not disable automatic updates. That is worth knowing precisely because the write path does.

If that pattern feels familiar it is because §19.3.5 found the same shape in a different package: SetMutexProfileFraction(-1) reads the current rate without setting it, and SetBlockProfileRate offers no reader at all. One function, two behaviours, selected by a sentinel argument — a convention the standard library uses more often than it documents.

But look at line 75 again. Reading GOMAXPROCS takes sched.lock — the one global lock the entire P design exists to avoid. It is cheap in absolute terms and it does not scale:

NumCPU does not register above the floor at all, and the floor row is why it is printed. NumCPU is return int(numCPUStartup) (debug.go:154-156) — one load of a variable the runtime sets at process startup, inlined into the caller. Dividing the middle row by it would produce a confident ratio out of the benchmark harness’s own loop overhead, so the honest statement is the qualitative one: reading NumCPU is free, and reading GOMAXPROCS(0) is a lock acquisition costing about 13 ns.

It also gets worse under parallelism — nearly threefold at 16-way — which is the signature of a global lock and the exact pathology P was introduced to remove.

21.2.5 Changing It Stops the World

One more fact about the write path, and it is sharper than the lock.

Setting GOMAXPROCS is not a variable assignment. Measured, from $GOROOT/src/runtime/debug.go:

// Illustrative snippet — not a complete program
if n == ret {                        // :90
    // sched.customGOMAXPROCS set, but no need to actually STW
    // since the gomaxprocs itself isn't changing.
    return ret                       // :93
}

stw := stopTheWorldGC(stwGOMAXPROCS) // :96
newprocs = int32(n)
startTheWorldGC(stw)                 // :104

Changing the number of processors stops the world. Every goroutine halts, procresize adds or removes Ps — moving any work stranded on a removed P to the global queue — and the world restarts. SetDefaultGOMAXPROCS does the same at :135.

The source says why it cannot be done live, at proc.go:6077: “gcworkbufs must not be being modified by either the GC or the write barrier code, so the GC must not be running if the number of Ps actually changes.”

Two consequences, and note how the first composes with §21.2.4's finding.

A same-value pin costs no pause but still sets the flag. Line 90 returns before the stop-the-world — so pinning to the value you already had is free in wall-clock terms and permanent in effect. That is the worst possible combination for detection: nothing to measure, and automatic updates gone.

And GOMAXPROCS(n) on a hot path is far worse than it looks. It is not a lock acquisition; it is a stop-the-world pause. The runtime’s own automatic updates pay this too, which is part of why they are rate-limited and skipped when the value has not moved — a container whose CPU limit is adjusted repeatedly buys a pause each time it actually changes.

21.2.6 Common Mistakes

Summary: G, M, P — and Which One You Run Out Of

Three resources with three wildly different ceilings. Gs are millions and running out is a memory problem. Ps are GOMAXPROCS and running out is the normal condition of a busy service. Ms top out at 10,000 and running out is fatal error: thread exhaustion — demonstrated by lowering the limit to 20 and watching the process die.

P exists to make the run queue local. Before Go 1.1 one global queue meant one global lock on every scheduling decision, and that is what did not scale. Every mechanism in §21.3 follows from making the queue local and managing the consequences.

GOMAXPROCS is container-aware since Go 1.25 — derived from logical CPUs, the affinity mask and the cgroup quota — it updates while the program runs, and its behaviour is gated on the language version in your own go.mod. That retires the NumCPU rule everyone learned, and makes setting the value yourself a trap rather than a fix, because doing so disables the automatic updates that now do the job.

And GOMAXPROCS(0) is a pure read that nonetheless takes the global scheduler lock: 12.8 ns serial and 36.3 ns at 16-way, against a NumCPU that does not measure above an empty benchmark loop.

Self-Check Questions: G, M, P — and Which One You Run Out Of

21.3 Where a Goroutine Waits

A goroutine becomes runnable — you called go, or a channel receive completed, or a timer fired. Between that moment and the moment it executes, it sits somewhere. This section is about the three places, and about a claim Chapter 19 made concerning them that turns out to be wrong.

21.3.1 Three Places, in Priority Order

┌──────────────┐
│  runnext     │  one slot, per P
│  (1 slot)    │  checked FIRST
└──────┬───────┘
       │ displaced
       ▼
┌──────────────┐
│ local runq   │  256 entries, per P
│  [256]       │  lock-free for its owner
└──────┬───────┘
       │ full: half of it moves
       ▼
  global runq     unbounded, one lock

Read in $GOROOT/src/runtime at go1.27.1:

runnext is a single-goroutine fast path, and it exists for one pattern in particular. When goroutine A sends on a channel and unblocks goroutine B, B goes into A’s runnext — so the very next scheduling decision on that P runs B, without touching a queue. That is what makes a request/response handoff between two goroutines fast: the value is copied straight into B’s frame (§21.6.4) and B is then the next thing scheduled, with no queue and no lock between the two events.

The local queue is 256 entries and is accessed without a lock by its owning P. That bound is the point: an unbounded local queue would let one P hoard work that other Ps could be running.

Overflow moves half, not one. When the local queue is full, runqputslow moves a batch of len(runq)/2 + 1 — 128 goroutines taken from the 256-entry queue, plus the one being pushed — to the global queue in a single lock acquisition. Moving one at a time would mean taking the global lock on every subsequent go statement; moving half amortises it and leaves the P with 128 to work on.

The global queue is checked on every 61st scheduling decision, whether or not the local queue has work:

// Illustrative snippet — not a complete program
// proc.go:3458
if pp.schedtick%61 == 0 && !sched.runq.empty() {

That is a starvation valve, and overflow is only half of what it protects. The other half is preemption: when sysmon stops a goroutine that has held its processor too long, that goroutine does not go back on the local queue. goschedImpl puts it on the global one (proc.go:4352), keeping it local only when the stop was for a garbage collection. So on a busy machine the global queue fills continuously with preempted goroutines, and without the 61st-tick check a processor with a permanently non-empty local queue would never look at them again. That is the loop between §21.3 and §21.5 closing: the mechanism in one section is what makes the valve in the other necessary.

The number itself is arbitrary — a prime, chosen to avoid resonating with other periodic behaviour — and it is exactly the kind of constant §21.7 tells you not to memorise.

21.3.2 Stealing Is Conservation, Not Balancing

Now the correction, and it changes what a figure printed in Chapter 19 means.

§19.4.3 showed per-P queue depths under schedtrace — [14 3 14 15 2 4 0 35 34 …] — and wrote: “Persistent imbalance like that points at work that is not being stolen effectively, usually because goroutines are being created in a pattern the scheduler cannot rebalance.”

That is wrong, and the source says why in one line. stealWork has exactly one caller in the entire runtime:

and that call site is inside findRunnable (proc.go:3404) — the function a P enters when it has nothing to run. There is no path by which a P that is currently executing a goroutine steals anything from anyone.

So look again at the line §19.4.3 printed. It reads idleprocs=0. Every P was busy. Not one of them was in findRunnable, so not one steal was ever attempted — and the imbalance is not evidence of stealing failing, because stealing was never invoked.

That distinction is worth holding onto because it inverts the diagnosis. Uneven queues with idleprocs=0 are fine: everyone is working. Uneven queues with idleprocs greater than zero would be the anomaly, because it would mean a P went looking and came back empty-handed while work existed — and that is a much narrower and more interesting situation.

21.3.3 Measured: The Difference Stealing Makes, and When It Cannot

The mechanism is directly observable. Two hundred short CPU-bound tasks, GOMAXPROCS=8, differing only in whether the other seven Ps are free to steal or are held by goroutines that are genuinely running:

With every P free, the burst finishes in about an eighth of the serial time — the tasks were created by one goroutine, landed on one P’s queue, and the other seven Ps pulled them across. That is stealing working exactly as intended.

With seven Ps held by running goroutines, the same burst takes the full serial time. Nobody stole, because nobody was in findRunnable, so all two hundred tasks ran on the single P that created them.

An important qualification, because the first arm is easy to over-read: this is not a benchmark showing stealing is fast. It is a demonstration that stealing is pull-based by idle Ps, using the presence or absence of idle Ps as the only variable.

21.3.4 How a P Looks for Work

findRunnable is where a P goes when its own queue is empty, and its order tells you the priorities:

1  this P's timers      before anything else, at :3430
2  local runq           its own, lock-free
3  global runq          shared, one lock
4  netpoll              ready network goroutines,
                        non-blocking
5  steal from other Ps  stealTries = 4 randomised passes
6  park the M

── cheapest source first; stealing is fifth ──►

Step 1 is easy to miss and it is the reason §21.3.5 works. pp.timers.check(0, nil) (proc.go:3430) is the first statement after the safe-point checks at the top of findRunnable — before the trace reader, before the GC worker, before the 61st-tick global check. A P looking for work looks at its own clock first.

Two details in step 5 repay knowing. The order in which other Ps are visited is randomised, so that many Ps entering findRunnable at once do not all attack the same victim. And the last of the four passes will also steal a P’s runnext and its timers — stealTimersOrRunNextG := i == stealTries-1 at proc.go:3850, a concession that only applies once the earlier, gentler passes have failed.

Step 6 is where the M parks, and it connects to §21.4: a P with no work releases its M rather than spinning forever, which is why an idle Go program does not burn CPU.

21.3.5 Where Timers Live

timerslen appears in every line of §21.1's scheddetail output, and no chapter of this book has said what it counts. Every chapter since Chapter 4 uses time.After, time.NewTimer or context.WithTimeout, and none of them said where the timer actually runs.

Timers live on the P. Each processor owns a heap of pending timers, and the scheduler checks it as the first thing it does when looking for work — step 1 of §21.3.4's list. There is no timer thread and no timer goroutine; a fired timer simply makes a goroutine runnable on the P that owned the timer.

That placement has three consequences worth carrying.

A timer is nearly free until it fires. Creating one is a heap insertion on the local P, with no lock in the common case and no goroutine. This is why the book has been able to recommend context.WithTimeout on every request without ever discussing cost — Chapter 13 gave it away for a reason.

Timers are stolen along with goroutines. The last of §21.3.4's four steal passes takes timers as well as goroutines. An idle P will take over another P’s pending timers rather than let them fire late, which is what stops a busy P from delaying every timeout it happens to own.

And a timer is a reason for the scheduler to expect progress, which is §21.5.4's point about the deadlock detector arriving from the other direction. checkdead looks for pending timers precisely because a timer will make something runnable without any other goroutine’s help — so one time.Ticker anywhere in the process suppresses deadlock detection for the whole program.

21.3.6 Common Mistakes

Summary: Where a Goroutine Waits

A runnable goroutine sits in one of three places: runnext, a single per-P slot checked first and used for the channel handoff that makes request/response fast; the local run queue, 256 entries and lock-free for its owner; or the global queue, which the local one spills half of itself into when full, which a preempted goroutine is put on (proc.go:4352), and which is checked every 61st scheduling decision so neither source can starve.

The correction: work stealing is conservation, not balancing. stealWork has one caller, inside findRunnable, so a busy P never steals. §19.4.3's uneven queues had idleprocs=0, which means no steal was ever attempted — the imbalance was the expected steady state, not evidence of a scheduler failing.

Measured, 200 tasks that finish in 13 ms with all Ps free take 101 ms against a 100 ms serial bound when seven of eight Ps are held by running goroutines. That is not stealing performing badly; it is stealing not happening, correctly.

findRunnable checks this P’s timers first, then the local queue, the global queue, the netpoller, and four randomised steal passes — the last of which will take another P’s runnext and timers — before parking the M.

Self-Check Questions: Where a Goroutine Waits

21.4 Threads You Did Not Ask For

Chapter 1 raised this and deferred it. §1.4 says the runtime “may create additional OS threads when goroutines block on syscalls” and hands the detail to §1.5; §1.3 says that when a goroutine blocks an OS thread “the runtime detects this and creates an additional thread so other goroutines keep running.” Neither says how many, or which kinds of blocking count.

Here is the detail, and the gap gets much wider than the threads=19 §21.1.1 opened with, or the threads=12 on sixteen processors that §21.1.3's block prints.

21.4.1 Measured: Two Kinds of Blocking

The same program, the same machine, GOMAXPROCS=16, two ways of having a thousand goroutines wait:

Five hundred blocked goroutines cost about one extra thread. Sixty-four cost about sixty-two. The duration of the individual calls is not the variable — §21.4.3 sweeps it and finds it barely matters.

That is roughly a five-hundred-fold difference in thread cost per blocked goroutine, and the only thing that varies is whether the wait is something the netpoller can express. §1.4 explains the netpoller itself — epoll, kqueue, IOCP, and the four-step flow — and this chapter does not repeat it. What §1.4 does not say is the price of the other branch, which is the entire content of this section.

And the threads do not come back. After the syscalls complete the count stays near 67. Ms are cached rather than destroyed, on the reasonable assumption that a program which needed 67 threads once may need them again. That is why thread count is a high-water mark rather than a current reading, and why a brief burst of file I/O leaves a permanent trace in your process.

21.4.2 The Mechanism

A goroutine that enters a syscall takes its M with it — the thread is now inside the kernel and cannot run Go code. If nothing else happened, GOMAXPROCS goroutines doing file I/O would stall the whole program.

So something else happens: the P is taken away from the blocked M and given to another one.

BEFORE                          AFTER
  M1 ─ P3 ─ G(syscall)      M1 ─────── G(syscall)
                                  ╲        still in kernel
  (other Ms idle)                  P3 ─ M7 ─ G'
                                     P reassigned to a
                                     new M; parallelism
                                     preserved

  The syscall keeps its thread. The P moves on.
  Threads: 1 -> 2. Parallelism: preserved.

Two consequences follow, and both are visible in scheddetail. The syscalltick field counts how many times a P has been handed over this way. And threads grows, because the M that was handed a P had to come from somewhere — either the idle pool or a fresh clone.

The netpoller is the contrast. A goroutine reading from a socket does not enter a blocking syscall. It registers with the poller, parks, and its M is free immediately to run something else. One poller thread watches every registered socket. That is why five hundred socket-blocked goroutines cost one thread and why a Go server can hold a hundred thousand idle connections on a handful of threads.

21.4.3 What Decides It Is Concurrency, Not Duration

Before the list, a correction to the impression §21.4.2 leaves. It describes what happens to a syscall that blocks. It does not say which syscalls pay, and the intuitive answer — the slow ones — is wrong.

When a goroutine enters a syscall the runtime marks its P as being in one and leaves it attached. Nothing is handed off and no thread is created. sysmon decides later whether to take the P, and its decision is mostly not about elapsed time. Measured, proc.go:6743-6761 in retake:

// Illustrative snippet — not a complete program
// Retake the P if it's there for more than 1 sysmon tick
// (at least 20us).
if syst := int64(pp.syscalltick); !sysretake &&
    int64(pd.syscalltick) != syst {
    pd.syscalltick = uint32(syst)
    pd.syscallwhen = now
    thread.resume()
    goto done
}

if runqempty(pp) && sched.nmspinning.Load()+
    sched.npidle.Load() > 0 &&
    pd.syscallwhen+10*1000*1000 > now {
    thread.resume()
    goto done
}

thread.takeP()

The first branch is the clock everyone quotes, and it is one sysmon tick — at least twenty microseconds, not ten milliseconds. The ten milliseconds is in the second branch, and that branch is an exemption rather than a threshold: the runtime leaves the P alone only when all three of its conditions hold — the P’s own run queue is empty, some other processor is idle or already hunting, and the call is younger than ten milliseconds. Nothing waiting, somewhere else to run it, and recent. Fail any one and the P is taken, which is where the thread comes from.

Note also that the ten milliseconds is a bare literal in that expression. It is not forcePreemptNS, and the two constants are unrelated despite being equal today.

Then hold the concurrency and sweep the duration straight across ten milliseconds:

Read the two tables against each other. A 128-fold increase in concurrency moved thread count 128-fold. A 400-fold increase in duration, straight across the number the folklore names, moved it by at most one thread. A fifty-microsecond call costs a thread — a hundred and twenty-eight of them, if a hundred and twenty-eight are outstanding. A twenty-millisecond call costs none when only two are. Thread count tracks how many blocking calls are in flight and is very nearly blind to how long each one takes.

The second table’s last row also shows the exemption doing its job, and it was the most stable measurement here — +0 in every run at every duration. At two calls in flight on four processors there is always an idle P and nothing queued, so the runtime declines to take anything and the thread count never moves. That is the case the ten-millisecond backstop exists for, and the reason a low-concurrency program can make slow syscalls all day for free.

21.4.4 What Actually Costs a Thread

Three categories, and it is worth being precise because “I/O” is not the dividing line.

Blocking syscalls the poller cannot express. Regular-file reads and writes are the common case: os.File operations on a local disk block the thread, because the OS offers no usable async interface for them on most platforms. So a service reading many files concurrently grows threads in proportion to its concurrency, while the same service reading many sockets does not.

cgo calls. Any call into C blocks its M for the duration, because the runtime has no way to interrupt or reschedule foreign code. A cgo-heavy service is a thread-heavy service, and the relationship is roughly one thread per concurrent cgo call in flight.

runtime.LockOSThread. This one is deliberate: it pins a goroutine to its M for as long as the lock is held, which some C libraries and OS interfaces require. The cost is that the M cannot be reused for anything else, and if the goroutine is long-lived the thread is permanently spoken for.

21.4.5 sysmon: the Thread With No P

Something has to notice that an M is sitting in a syscall and decide whether its P should be reassigned. That something is sysmon, a thread the runtime starts at boot and never schedules — it has no P and runs outside the Go scheduler entirely.

It appears in scheddetail as sysmonwait, and it does four jobs relevant to this book:

• Retakes Ps from goroutines sitting in syscalls, which is §21.4.2's mechanism and §21.4.3's arithmetic.
• Forces preemption of goroutines that have run too long, which is §21.5's.
• Polls the network if nobody else has for a while, so a program with no other activity still notices readable sockets.
• Runs the periodic GOMAXPROCS update introduced in Go 1.25 (§21.2.3), which is why that value can change while your program runs without your program doing anything.

sysmon appears zero times in the preceding twenty chapters, which is defensible — you cannot control it and never call it — but it is the answer to “who preempts the preempter” and to “who decided my syscall had cost me a thread”, and both questions arrive eventually.

21.4.6 Watching It, and the Ceiling

Two runtime/metrics counters make this observable without pprof, in the always-on style §19.4 argued for:

// Illustrative snippet — not a complete program
s := []metrics.Sample{
    {Name: "/sched/threads/total:threads"},
    {Name: "/sched/gomaxprocs:threads"},
}
metrics.Read(s)

/sched/threads/total is the live OS thread count — the number that reproduced §21.4.1's table without any profiling enabled. /sched/gomaxprocs is the current GOMAXPROCS, read without going anywhere near runtime.GOMAXPROCS and therefore without touching sched.lock (§21.2.4). Post-1.25 that second value moves, which makes it a genuinely useful thing to alert on: a service whose GOMAXPROCS silently halved has had its quota cut, and nothing else in your telemetry will say so.

The reason to watch the first is §21.2.1's ceiling. Thread growth is gradual, silent, and terminal:

21.4.7 Diagnosing Thread Growth

Thread growth is worth a procedure, because it is silent, monotonic, and terminal — and because the diagnosis is quick once you know the four questions.

First, confirm it is growth rather than a level. Threads are a high-water mark (§21.4.1), so a single reading tells you almost nothing. Sample /sched/threads/total twice, minutes apart, under comparable load. A flat number at 400 is a program that once did 400 concurrent blocking calls and is fine. A number rising steadily is the problem.

Second, find the blocked goroutines. Threads grow because goroutines block in ways the poller cannot express, so the goroutines are still there, parked. A debug=1 goroutine profile (§19.2.2) collapses them: if thread growth is real, one stack will have a count that tracks it almost exactly.

Third, classify the wait. The stack tells you which of §21.4.4's three categories you are in — a syscall.Read on a regular file, a cgo entry point, or a LockOSThread region. Those have different fixes and the stack distinguishes them immediately.

Fourth, bound it. The fix is almost never to make the blocking call faster; it is to limit how many can be in flight, which is Chapter 17's material. A semaphore sized from the thread budget you are willing to spend converts an unbounded climb into a queue you can see.

21.4.8 Common Mistakes

Summary: Threads You Did Not Ask For

Chapter 1 promised this section. Measured, on one machine at GOMAXPROCS=16: five threads at rest, six with five hundred goroutines blocked on sockets, and sixty-seven with sixty-four blocked in a syscall. Roughly a five-hundred-fold difference in thread cost per blocked goroutine, decided entirely by whether the netpoller can express the wait — and the threads do not come back afterwards, because Ms are cached.

The mechanism is that a syscall keeps its M and loses its P: the P is handed to another thread so parallelism is preserved, syscalltick counts the handovers, and threads grows.

What decides whether a given call pays is concurrency, not duration. sysmon takes a syscalling P after one tick — twenty microseconds — unless the P’s queue is empty and another processor is idle and the call is younger than ten milliseconds, which is an exemption rather than a threshold. Measured at GOMAXPROCS=4: fifty-microsecond calls cost one thread each at 128 in flight, while twenty-millisecond calls cost nothing at two in flight — a 128-fold concurrency change moved threads 128-fold and a 400-fold duration change moved them by at most one. Thread count is in-flight count, which by §17.1.3's Little’s Law is rate times duration — so a filesystem that slows down raises threads at an unchanged request rate.

Three things cost a thread: blocking syscalls the poller cannot express, cgo calls, and LockOSThread — the same three the SetMaxThreads documentation names.

sysmon is the thread with no P that retakes those Ps, forces preemption, polls the network as a backstop, and runs the Go 1.25 GOMAXPROCS update. It appears in zero earlier chapters and answers several questions that do.

And the ceiling is fatal: ten thousand threads, then fatal error: thread exhaustion, reachable by unbounded concurrency over file I/O or cgo.

Self-Check Questions: Threads You Did Not Ask For

21.5 Preemption, and the Ten Years It Took

§1.4 states that Go 1.14 introduced asynchronous preemption and that a tight loop no longer starves other goroutines. That is correct and it is an assertion. This section makes it observable, which is the difference between knowing the fact and having seen what it protects you from.

21.5.1 Measured: One Environment Variable, One Hang

GOMAXPROCS(1) — exactly one P, so anything that wants to run must take it from whoever has it. One goroutine runs a loop with no function calls, no allocation, and no channel operations. main sleeps briefly, then needs the P back in order to print.

// Illustrative snippet — not a complete program
runtime.GOMAXPROCS(1)
go func() {
    var x uint64
    for { // no calls, no allocation, no channel ops
        x++
        sink = x
    }
}()
start := time.Now()
time.Sleep(50 * time.Millisecond)  // main yields; the loop
                                   // takes the P
late := time.Since(start) - 50*time.Millisecond
fmt.Println("main resumed", late, "late")

Note that the loop never ends. That is deliberate and it is the whole experiment: main can only print if something takes the processor away from a goroutine that will never give it up.

One environment variable turns a working program into a permanent hang. That is precisely what Go 1.14 bought, reproducible on demand — and it is the world every Go program lived in before then.

The seven-to-ten milliseconds in the first row is not noise, and it is worth holding onto until §21.5.5 names it: it is the slice a goroutine gets before sysmon intervenes. Preemption is not instant; it is bounded.

21.5.2 The Cost Is Not Fairness, It Is Stop-the-World

The obvious reading of that measurement is about fairness on a single processor, and it is the less important one. Give the machine room — GOMAXPROCS=4, so three of four processors are idle — and ask the main goroutine to collect garbage instead of print.

Three idle processors did not help at all, and that is the finding. A garbage collection has to stop the world, and stopping the world means every goroutine reaching a safe point — not most of them, not the ones on busy processors. One goroutine that will never reach one makes the pause unbounded no matter how much of the machine is free.

That also gives §19.4.4 its mechanism. That section told you to compare /sched/pauses/stopping/other against /sched/pauses/total/other, and said that when stopping approaches total, “the collector is not slow — it is waiting on a goroutine that will not yield.” The measurement above is that sentence’s limiting case: stopping is the entire pause, and the pause never ends.

21.5.3 Why the Old Model Failed

The loop is deliberately austere. A function call is a cooperative preemption point, so a loop containing almost any call could always be preempted; that is why the pre-1.14 problem was rarer than it sounds and maddening when it appeared. The loops that hung were numeric kernels, tight for loops over slices, and busy-waits — code with no reason to touch the runtime.

cooperative (always)      asynchronous (since 1.14)
────────────────────      ────────────────────────
│ func entry              │ sysmon notices 10ms
│   ↓                     │   ↓
│ check preempt flag      │ send SIGURG to the M
│   ↓                     │   ↓
│ yield if set            │ handler parks the G
│                         │   at that instruction
works for any code        works for code with no
that calls functions      call sites at all

The second required the compiler to guarantee
every instruction is a safe point. That is the
decade.

21.5.4 A Row Missing From Chapter 10

This is the chapter’s second correction to a shipped page.

§10.4 has a table of what does and does not trigger the runtime’s deadlock detector. It lists channel send and receive, sync.Mutex.Lock, sync.WaitGroup.Wait, select, empty select, time.Sleep, <-time.After(), and network or file I/O.

There is no row for a goroutine that is simply running.

That is the case §19.4.4 met and could not name — the goroutine that “will not yield” — and it is the most common reason a production deadlock goes unreported. The detector fires only when every goroutine is blocked on something that requires another goroutine to release it. One goroutine executing a loop is not blocked at all, so the condition never holds, and the deadlock among the other thousand goes unannounced.

The distinction that makes it fit §10.4's framing: the other “✗” rows are cases where progress is possible without another goroutine — a timer will fire, an I/O completion will arrive. A running goroutine is the degenerate version, where progress is not merely possible but happening.

That last claim is not an inference. It is the final thing checkdead does before giving up, at proc.go:6512-6514:

// Illustrative snippet — not a complete program
// There are no goroutines running, so we can look at the P's.
for _, pp := range allp {
    if len(pp.timers.heap) > 0 {
        return
    }
}

Six lines below it is fatal("all goroutines are asleep - deadlock!"). One pending timer anywhere in the process is the whole distance between the two, which is why a service with a time.Ticker in it will hang silently where a toy program would have told you.

21.5.5 How the Interrupt Actually Happens

Cooperative preemption is a check. The compiler inserts a test at function entry — is this goroutine flagged for preemption? — and if so it yields. That covers any code that calls functions, which is nearly all code, and it costs a comparison.

Asynchronous preemption is a signal. sysmon (§21.4.5) notices a goroutine has held its P too long and sends the thread a SIGURG; the signal handler parks the goroutine at whatever instruction it had reached. That is what makes it work on a loop with no call sites.

The threshold is a constant:

// Illustrative snippet — not a complete program
// proc.go:6679
const forcePreemptNS = 10 * 1000 * 1000 // 10ms

That is the 10 ms §1.4 mentions without naming, and it is the maximum time a goroutine can monopolise a P before sysmon intervenes.

The mechanism is more delicate than “send a signal”: the runtime has to be able to stop at that instruction and later resume, which means the compiler must guarantee that register and stack state are recoverable at any preemptible point. That guarantee — asynchronous safe points — is why the feature took as long as it did, and why some regions remain non-preemptible.

21.5.6 What Preemption Still Cannot Interrupt

Two limits survive, and only one of them matters in practice.

A goroutine inside a syscall cannot be signal-preempted. The thread is in the kernel; the runtime cannot park it at an instruction it does not control. What happens instead is §21.4.2's mechanism: sysmon takes the P away and hands it to another M, so the goroutine keeps its thread and the rest of the program keeps running.

With a single P, the second goroutine could only run if the P had been taken back — so this measures the retake directly. The distinction worth holding: the goroutine is not preempted, but the processor is recovered, which is what the rest of the program actually needs.

Runtime-internal non-preemptible regions. Some code marks itself unpreemptible — parts of the scheduler, the allocator, and stack growth itself, because moving a stack while something is preempting it is not a thing that can work. These windows are microseconds and you will not observe them from user code, which is the right amount of attention to give them.

21.5.7 Common Mistakes

Summary: Preemption, and the Ten Years It Took

§1.4 asserts asynchronous preemption; this section demonstrates it. The same binary at GOMAXPROCS=1 with a call-free tight loop resumes 7–10 ms late by default — inside forcePreemptNS — and never resumes at all under GODEBUG=asyncpreemptoff=1. That is the pre-1.14 world, on demand.

And the important half is not fairness. At GOMAXPROCS=4 with three processors idle, runtime.GC() still hangs under the same flag — because stopping the world requires every goroutine to reach a safe point, and one that never will makes the pause unbounded however much of the machine is free. Async preemption is what bounds stop-the-world pauses, which is what makes the collector’s latency guarantees possible; fairness is a side effect. That is also the mechanism behind §19.4.4's stopping-versus-total ratio.

Chapter 10's deadlock-detector table is missing a row: a goroutine that is simply running suppresses the detector, because it is not blocked at all. That is §19.4.4's goroutine that would not yield, and it is why production almost never reports a deadlock — though the more precise rule is that the runtime suppresses whenever it can see a reason to expect progress, which a single pending timer also provides.

Cooperative preemption is a compiler-inserted check at function entry. Asynchronous preemption is a SIGURG from sysmon after forcePreemptNS, 10 ms. It required asynchronous safe points, which is why it took a decade.

Two limits remain. A goroutine in a syscall is not preempted — its P is retaken instead, measured directly at GOMAXPROCS=1. And a few runtime-internal regions are unpreemptible for microseconds. runtime.Gosched() is a pre-1.14 workaround that modern code rarely needs.

Self-Check Questions: Preemption, and the Ten Years It Took

21.6 Why the Cheap Things Are Cheap

Three mechanisms, one theme. Creating a goroutine is cheap, blocking one is cheap, and handing a value between two of them is cheap — and each is cheap for a specific, findable reason. This section is the positive half of the spine, before §21.7 spends what is left of it.

21.6.1 A Stack That Moves

§1.4 covers the shape: a goroutine starts with about 2 KB, the runtime grows it when a call would overflow, growth is a copy to a larger allocation, and the cap is around 1 GB. What that section does not say is what the copy costs or what it implies.

The way to isolate the copying is to do the same descent twice in the same goroutine. The first pays for growth; the second finds the stack already grown.

Same goroutine, same work, same code path. At depth 1,024 — a recursive parser on a nested document, say — the first call through costs 1.11 ms and every later one costs 42 µs.

Now read the ratio column downward: 48×, 33×, 26×, 17×, 11×. The overhead is proportionally largest at shallow depths and shrinks as the chain lengthens, which is the opposite of what “deep recursion is expensive” would predict.

It also says which case to worry about, and it is not the one people worry about. A goroutine recursing to 32,768 pays 11× on its first descent and is fast forever after. A goroutine recursing to 16 and then exiting pays 48× and never amortises anything. The expensive pattern is shallow growth repeated across many short-lived goroutines, not deep growth in one long-lived one.

The copy is why a goroutine’s stack address is not stable. When the runtime moves a stack it must find every pointer into that stack and rewrite it, which it can do because Go is type-safe and the compiler emits maps describing where pointers live. Two consequences follow that occasionally matter: you cannot hand a pointer to a Go stack variable to C and expect it to remain valid, and there is no way to observe the address change from Go, because everything that could observe it has been fixed up.

21.6.2 The Goroutine Is Pooled; the Stack Is Not

Which raises the obvious optimisation: if growth is the cost, reuse the grown stacks. Keep a pool of goroutines that have already paid, and hand work to them.

The runtime looks like it is doing exactly that. Each processor keeps a free list of dead goroutines — gfreecnt in §21.1.3's output. When a goroutine returns its g structure goes on that list rather than being released, and the next go statement on that P takes one back with no allocation and no scheduling setup. That is the other half of “creation is cheap”: most creations are a pop from a per-P free list rather than an allocation. The list caps at 64 (proc.go:5536) and drains half to a central one beyond that.

But look at what actually goes on the list. Measured, proc.go:5521-5527, in gfput:

// Illustrative snippet — not a complete program
stksize := gp.stack.hi - gp.stack.lo

if stksize != uintptr(startingStackSize) {
    // non-standard stack size - free it.
    stackfree(gp.stack)
    gp.stack.lo = 0
    gp.stack.hi = 0

The g is pooled. A stack that is not exactly the starting size is freed. The next goroutine to reuse that g starts at the starting size and grows all over again.

Which turns §21.6.1's measurement into a decision. If the work needs a deep stack, do it on a long-lived goroutine rather than a fresh one per item. A worker pool of ten goroutines each handling a thousand deep-recursion jobs pays for growth ten times. Ten thousand short-lived goroutines each handling one pays ten thousand times, and no amount of pooling at the goroutine level changes that — which is a concrete reason to prefer Chapter 7's worker pool over spawning per item, independent of every reason that chapter gave.

21.6.3 Reconciling 2 KB With 2.7 KB

§1.4 and §2.5 say a goroutine costs about 2 KB. §14.4.4 measured 2.7 KB apiece for ten thousand parked goroutines. Both are correct, and §21.1 promised to say which to use.

Note which number the division is on. Dividing the total by ten thousand gives 2,120 and quietly charges every goroutine a share of the runtime’s own stacks; dividing the difference gives 2,081, which is what one more goroutine actually costs — and which is §14.4.4's stack half to the byte. The remaining ~600 bytes in Chapter 14's 2.7 KB is the g structure and its scheduling bookkeeping, which lives on the heap rather than in StackInuse.

So:

Chapter 14 states the trap outright — “the commonly quoted 2 KB per goroutine is the stack half only” — and it is worth repeating here because the smaller number is the famous one and under-predicts memory by about a third.

21.6.4 The Channel: A Lock and a Queue

Now §21.1's fifth artifact — Chapter 5 called an unbuffered channel a synchronisation point, and §19.3.3 found channel operations in the mutex profile.

Both are true because a channel is a mutex around a ring buffer and two wait queues. Every operation — send, receive, close — takes that lock. That is why channel contention shows up where §9's material and §19.3's block and mutex profiles can see it, and why a heavily contended channel and a heavily contended mutex look similar in a profile: at that level they are the same thing.

But the two profiles name it differently, and one of them does not name it at all. Measured: sixteen goroutines contending on one buffered channel, with both profiles enabled at rate 1:

The block profile names the channel. The mutex profile does not name it anywhere in the flat column, because the leaf is where the contention was measured and that is literally runtime.unlock. The chansend and chanrecv frames are in the stacks, so they appear in the cumulative column and in a graph view — but a team that greps a flat mutex profile for chan, which is the first thing anyone does, finds nothing and concludes there is no channel contention. GODEBUG=runtimecontentionstacks=1 does not help; that knob is for the runtime’s own internal locks, and re-running under it left the leaf unchanged.

receiver waiting?  ──yes──► copy STRAIGHT into the
│                           receiver's stack, mark
│                           it runnable via runnext
no
│
buffer has room?   ──yes──► copy into the ring
│                           buffer, return
no
│
└──────────────────────────► park the sender in a
                             sudog on the send queue

All three take the channel's lock. Only the
third one blocks.

The interesting mechanism is what happens when the lock is held and a partner is already waiting.

A send to a waiting receiver copies directly into the receiver’s stack. No buffer slot is used, even on a buffered channel; the value goes from the sender’s frame to the receiver’s, the receiver is marked runnable, and — via runnext (§21.3.1) — it is very likely the next thing that P runs. The waiting goroutines are held in sudog structures, which are themselves pooled per-P, so blocking on a channel does not allocate in the common case.

That is a genuinely elegant path, and it is tempting to conclude something about performance from it that is not true.

21.6.5 Measured: What Direct Handoff Does Not Buy

The tempting conclusion is that direct handoff makes unbuffered channels as fast as buffered ones. It does not, and the mistake is worth showing because the mechanism is real while the consequence is false.

Unbuffered is roughly four times slower than the best buffered case. Direct handoff happens for buffered channels too whenever a receiver is already waiting, so it is not something unbuffered channels have and buffered ones lack. What unbuffered channels lack is the ability for the sender to proceed without a partner, and on a streaming workload that is the whole cost.

The curve also stops paying, and the last row is why the table goes past 64. Capacity 1024 is no better than capacity 64 — slightly worse here, and it was the noisiest row across the ten runs. Buffering buys the sender room to run ahead of the receiver; once the buffer is larger than the lead the sender can actually build, more capacity is a larger allocation and a colder ring. If you are choosing a number, the useful question is how far ahead the producer gets, not how big the queue could be.

The true statement about handoff is narrower, and it is the useful one for §5's “when to buffer” question:

The method matters more than usual here, so it is stated: run-to-run spread within a single capacity reached 30 % on this machine, several times the ~5 % spread between capacities. Capacity is not what this benchmark is measuring — which is the finding, and the contrast with the streaming table above, where capacity moved the number four-fold, is the whole argument.

21.6.6 Common Mistakes

Summary: Why the Cheap Things Are Cheap

Stacks grow by copying, which is why a goroutine’s stack address is not stable and why the runtime must rewrite every pointer into it. Measured, the same descent run twice in one goroutine costs 48× more the first time at depth 16 and 11× at depth 32,768 — a falling ratio, which is the signature of geometric growth and which says the expensive pattern is shallow growth across many short-lived goroutines rather than deep growth in one.

Creation is cheap partly because most creations pop a g from a per-P free list — gfreecnt in scheddetail — which makes creation cost bimodal and makes creation benchmarks measure the cheap path. But only the g is pooled: a stack that is not exactly the starting size is freed at proc.go:5523, and the starting size itself is recomputed at every collection from the average of the stacks scanned.

Measured, ten thousand parked goroutines add 2,081 bytes of stack each — §14.4.4's stack half to the byte, provided the arithmetic is on the difference rather than on the total. Use ~2 KB for “how deep can I go” and ~2.7 KB for “what will a million cost”, because the extra is the g structure on the heap.

A channel is a mutex around a ring buffer and two wait queues, which is why channel operations appear in the mutex profile — though the flat leaf there is runtime.unlock and the word chan appears nowhere in it, so the profile that names the channel is the block profile. A send to a waiting receiver copies straight into the receiver’s stack, and the waiters are pooled sudogs so blocking does not allocate.

But direct handoff is not why unbuffered channels are fast, because they are not: measured, streaming costs 184.2 ns at capacity 0 against 42.7 ns at capacity 64 — four times slower — and the curve then flattens, capacity 1024 buying nothing over 64. The true statement is that when the pattern forces a rendezvous, buffering buys nothing — ping-pong measured 355, 357 and 374 ns at capacities 0, 1 and 64, a spread smaller than the noise within any one of them.

Self-Check Questions: Why the Cheap Things Are Cheap

21.7 What This Changes About Code You Already Wrote

Every section so far has explained something. This one changes something, which is the test §21.1.2 set for the chapter and the debt it has to settle before ending.

21.7.1 Two Chapters of This Book Gave You a Number That Is Wrong in a Container

§14.4.4 says: “Choosing the limit. Start at runtime.NumCPU() for CPU-bound work and roughly double that for I/O-bound, then measure.” And §17.4.7 repeats it: “Chapter 14 gave the starting heuristic and it stands: runtime.NumCPU() for CPU-bound work, roughly double for I/O-bound, then measure.”

Not GOMAXPROCS. NumCPU. And runtime.NumCPU is documented as reporting the logical CPUs “usable by the current process”, queried from the operating system at process startup — the doc comment says so at debug.go:151-153 and the body is a single load of the variable it recorded — which on Linux does not account for a cgroup CPU quota.

That over-provisioning is not harmless. Thirty-two times the intended concurrency means thirty-two times the in-flight memory, thirty-two times the pressure on whatever downstream Chapter 17 told you to protect, and a queue at the CPU that the Go scheduler will handle correctly and slowly.

21.7.2 And a Second Bug, Which Only Appeared in Go 1.25

Fix the first bug naively — swap NumCPU() for GOMAXPROCS(0) at construction — and you land in the second.

Since Go 1.25 the runtime re-derives GOMAXPROCS from the quota while your program runs (§21.2.3). So a width read once, at startup is a value that can go stale: a service whose quota is cut mid-incident keeps scheduling at its old width, which is precisely the moment you would want it to narrow.

These are different bugs and they want distinguishing:

21.7.3 And a Third Trap, If You Try to Make It Stop Moving

The natural response to a value that moves is to pin it. That is the trap §21.2.3 named:

// Illustrative snippet — not a complete program
// ✗ BROKEN: pins the value AND disables automatic updates
runtime.GOMAXPROCS(n)

Calling GOMAXPROCS with n >= 1 sets sched.customGOMAXPROCS, and from then on the runtime stops re-reading the quota entirely. You have traded a value that tracks reality for one that does not, in order to avoid having to read it twice.

Worse, at debug.go:83 the flag is set before the n == ret early return at :90 — so pinning to the value it already had disables updates while changing nothing observable. There is no exported way to ask whether automatic updates are still on; sched.customGOMAXPROCS is unexported, and the /godebug/non-default-behavior/updatemaxprocs counter tracks only the GODEBUG path, not the call path.

NumCPU()          ──► 64 in a 2-CPU container
at construction       wrong from the first second
                      (the book's advice)
      │ fix naively
      ▼
GOMAXPROCS(0)     ──► right at startup, stale
at construction       after a quota change
                      (nobody's advice; you
                       arrive here by fixing)
      │ "make it stop moving"
      ▼
GOMAXPROCS(n)     ──► updates disabled forever,
to pin it             even pinning the SAME value
                      (unobservable from Go)

21.7.4 The Posture That Works

Three rules, in order of how much they matter.

Read GOMAXPROCS(0), not NumCPU(). It is the number the runtime will actually schedule, and it is container-aware. NumCPU remains correct for the question it answers — how much hardware exists — which is rarely the question you have.

Read it where the width is decided, not where the work is done. §21.2.4 measured GOMAXPROCS(0) at 12.8 ns serial and 36.3 ns at 16-way, because it takes the global scheduler lock. Once per batch, once per pool resize, once per period — fine. Once per item on a hot path — a global lock on your fastest code, and the one the P design exists to avoid.

If you pin it, pin it deliberately and say so. A benchmark, a reproduction, a workload with a measured better width: all legitimate. Acquiring the opt-out as a side effect of an environment variable somebody set in 2019 is not.

21.7.5 When to Stop

The last thing this chapter owes you is a boundary, because it has spent seven sections reading runtime source and the honest position is that most of it will not stay true.

Not durable. The 256-entry run queue, the 61-tick global check, four steal attempts, the 10 ms preemption slice, runnext as a single slot, sudog pooling, per-P gFree lists. Every one is an implementation detail of go1.27.1. Several changed within the last few releases and several will change again. A reader who memorises 61 has acquired trivia with a half-life.

Durable. The shape:

• Work queues are per-processor, because a global one does not scale.
• Idle processors pull work; busy ones do not push it. Stealing is conservation.
• Blocking is free when the runtime can express the wait and costs a thread when it cannot.
• Threads are the scarce, fatal resource; goroutines are not.
• A goroutine can be interrupted anywhere, at a cost the runtime paid in compiler machinery.
• GOMAXPROCS bounds Go execution, not threads and not resource use — and it is no longer a constant.

And the observables, which outlast both: GODEBUG=scheddetail, /sched/goroutines/*, /sched/threads/total, /sched/gomaxprocs, /sched/latencies, and the four trace-derived profiles from §19.5.2. Those tell you what the current runtime is doing without requiring you to know how this year’s version implements it.

21.7.6 Common Mistakes

Summary: What This Changes About Code You Already Wrote

Two chapters of this book told you to size pools from runtime.NumCPU(). That call is host-scoped and cgroup-blind, so in a 2-CPU container on a 64-core host it returns 64 — a pool thirty-two times too wide from the first second, with no staleness involved. That is the harder bug and it is the book’s.

Fixing it naively lands you in the second: since Go 1.25 GOMAXPROCS is re-derived while the program runs, so a width read once at construction can go stale exactly when a quota is cut.

And pinning the value to stop it moving is a third trap, because that disables the automatic updates — including when you pin it to the value it already had, since the flag is set before the no-op check. There is no exported way to ask whether updates are still on.

The posture: read GOMAXPROCS(0) rather than NumCPU(), read it where the width is decided rather than per item, and pin only deliberately.

And the boundary. The constants in this chapter are implementation details with a half-life. The shape — per-P queues, pull-based stealing, blocking that costs a thread when the poller cannot express it, threads as the fatal resource, preemption anywhere, GOMAXPROCS as a moving bound on Go execution — and the observables are what last.

Self-Check Questions: What This Changes About Code You Already Wrote