Chapter 22: What Held, and What Did Not

22.1 The Experiment, and How to Re-run It

Every figure in the previous twenty-one chapters is reproducible with go test. A chapter whose central artifact is a twenty-minute crawl could easily have broken that, and the design below exists to stop it.

22.1.1 Why the Crawl Is Not of the Internet

A crawl of real websites cannot be re-run by a reader: the targets change, the network changes, and the load is somebody else’s problem. So the target is an in-repo seeded site graph served by httptest, shipped as code/ch22/sitegraph/. It is a real loopback server rather than Go 1.27’s in-memory httptest.NewTestServer on purpose: §22.3.2’s thread count is only evidence if the fetches are real sockets. There is precedent — code/ch18/corpus/ is already a bundled fixture package.

The cost of a fixture is that somebody chose its parameters, and a chosen distribution can manufacture a chosen finding. They are therefore stated, and each is justified against a shape from the world rather than picked.

22.1.2 The Nine Series

Sampled at 1 Hz through runtime/metrics and the crawler’s own counters: goroutines, runnable goroutines, OS threads, heap in-use, allocation rate, cumulative GC pause, limiter-map length, in-flight fetches, and frontier depth — plus the counters §17.7.5 asks for.

One detail nearly cost the chapter. The limiter’s wait time was first recorded as a cumulative mean, which is dominated by its earliest samples and therefore cannot show a slope — the entire subject. It was changed to a windowed mean before the run.

22.1.3 The Window, Derived

A slope needs long enough to be a slope. Two constraints fixed the window before anything ran:

1  K ≫ R·T        the key space must not saturate
                  inside the window, or growth stops
                  for a reason that is not the code's
2  Δ(end,start)   ≥ 6σ of the series' own sampling
                  noise, or a trend cannot be told
                  from jitter

── twenty minutes at 1 Hz: 1,200 samples ──►

Both were checked in a twelve-second calibration, which also caught a metric that reads zero: /sched/threads:threads does not exist on go1.26.1 or go1.27 — the Go 1.26 release notes print it that way; the runtime does not. The name is /sched/threads/total:threads, and the first calibration recorded an entire thread series of zeros before anyone noticed.

22.1.4 Three Configurations, and Why All Three

The frontier — the queue of URLs still to fetch — is where the book gives more than one answer, and §22.2.2 is the fork. Rather than choose and report one, all three implementations were built and measured:

The twenty-minute run and its control both use Drop, because §22.2.2 argues it is the correct reading of the book. Grow and Block are measured separately in §22.5.2, and the three results together are the chapter’s most useful finding.

22.1.5 Pre-registration

§20.7.9 says to write the number down before you start. Applied to a system, that means the predictions exist before the run and are published whatever they say.

predictions.md was committed with a timestamp and a SHA-256 before the twenty-minute run. Twelve entries, each carrying four fields — an entry missing any of them is not a prediction:

The file also carries a section headed what no prediction covers, so that a finding could not later be presented as having been anticipated.

Disclosure, because pre-registration without it is worthless. Two calibration runs of ten and twelve seconds preceded the file, to fix metric names and sampling. Twelve seconds cannot show a twenty-minute slope, so nothing in §22.5 was visible at registration time — but the calibration happened, and saying so is part of the method.

22.1.6 Running It Yourself

Twenty minutes, and it writes the nine series to a CSV. CH22_HOSTS=200000 reproduces §22.4.3's control, CH22_SWEEP=off disables §17.7.2's sweeper so you can watch the limiter map grow instead of converge, and CH22_OUT names the output file.

The runs this chapter reports ship too, so no figure here rests on a file you cannot open:

Two of those are independent runs, and they are the evidence for the callout below. soak_registered.csv is the default configuration run again, separately from the one this chapter reports. It stalls at t=564 against t=561, ends at 901,656 URLs seen against 900,219, and excludes 64.0% against 64.2% — while reporting 18 OS threads where the headline run reports 23. The stall instant reproduced to within three seconds and the excluded fraction to within two-tenths of a point; the absolute that §22.3 prints in a table moved by 22%. That is also why P2 was registered as at or below 25 rather than at 23 — a bound survives a move that a value does not, which is §22.4.5 seen from the side where the form was chosen correctly. soak_sustained.csv is the same check from the other direction: a graph on which the frontier never drains, 3,702,167 URLs seen in twenty minutes and 81.3% of them excluded, against the control’s 80.9%.

The rest run in seconds: frontier_variants_test.go is §22.5.2's three configurations, mapdrain_test.go is §22.5.4's two, and gates_test.go is the exercise.

A unit convention, stated once. Throughout this chapter MB means 10⁶ bytes, which is what runtime/metrics reports and what the harness divides by. 72.6 MB and 80.6 bytes an entry are consistent with each other on that basis; read as MiB the second figure would be 84.6.

22.1.7 Common Mistakes

Summary: The Experiment, and How to Re-run It

The chapter’s central artifact is a twenty-minute time-series, which is the one thing in this book a go test could easily fail to reproduce — so the target is a seeded in-repo graph, every fixture parameter is stated and justified, and the window was derived from two constraints rather than chosen. Twelve predictions were registered in a form that admits falsification, the calibration that preceded them is disclosed, and where the book gave two answers all three implementations were built rather than one being reported.

Self-Check Questions: The Experiment, and How to Re-run It

22.2 Every Rule the System Was Built From, and What Each One Predicts

This section is the audit trail. It exists so that §22.4 can score the book rather than the author’s foresight, and it is short by design — the rules are already taught, and each row is a citation rather than a lesson.

22.2.1 The Audit Trail

Fifteen decisions, fifteen citations, and the whole of §22.4's meaning rests on that being exactly true rather than approximately true. Two were wrong on the first pass and corrected before the run: the seen-set’s compound update is §11.3.3's check-then-act rather than Chapter 12's sync.Map material, and one citation turned out to have no owner at all (§22.2.3).

22.2.2 The Fork, and Why It Has No Correct Answer

One decision mattered more than the rest, and the book genuinely gives more than one answer.

A crawler’s frontier is the case that separates them. Eight links are discovered per page fetched and one page is consumed, so arrivals exceed service by roughly eight to one permanently. That is not a burst, so §5.2's rule does not apply and §5.3's does — the frontier is bounded and drops what does not fit.

That is the correct reading, and §22.5.3 is what it costs. But the reason this fork gets its own subsection is that the third reading is also available and also loses, which §22.5.2 measures. Record the fork here; the arithmetic of who is right is in §22.5.

22.2.3 Where No Rule Applied

Three decisions had no owner, and naming them is the honest half of an audit trail.

Draining the response body. Before Go 1.27 a body not read to completion and closed left the connection unusable, so the transport opened a new one and §17.7.1's connection pool bought nothing; from 1.27 Close drains a bounded amount itself, and a body above that bound still costs the connection. No chapter covers either. It is two lines on which the entire throughput figure depends — and this crawler would read the body anyway, because the links are in it.

What to do with a URL the frontier rejected. §5.3 says drop. Nothing says whether to record the drop, retry it, or treat it as data loss. §22.5.3 is what that silence costs.

Whether the dedup set has a lifetime. Every other structure in this book has one — §12.3's pool is emptied by the collector, §17.7.2's limiter map is swept, contexts are cancelled. The seen-set is the only thing deliberately permanent, and nothing prices it.

22.2.4 The Twelve Predictions

Registered before the run in the four-field form. Abbreviated to source and claim; tolerances and falsifiers are in predictions.md.

P11 is the only one derived rather than quoted, and it is Little’s Law doing real work. L = λW. The fixture’s latency is log-normal at median 15 ms with σ = 1.1, so its mean is 15·e^(σ²/2) ≈ 27.5 ms — not 15, and the difference is the whole prediction. With sixteen workers each holding one fetch, λ = 16 / 0.0275 ≈ 580 fetches per second.

22.2.5 What No Prediction Covers

Registered alongside the twelve, so that nothing found later could be presented as anticipated:

• What fraction of a corpus a bounded frontier discards, and whether anyone should watch it. P7 predicts that drops happen and that the rate is stable. No chapter says the fraction matters, names a threshold, or puts it on a dashboard.
• Which series moves first when something goes wrong. §17.7.5 says what to instrument and §19.4.6 says what to add; neither orders them.
• Anything whose signal is a slope. That is the chapter’s thesis and the reason for the twenty-minute window.

22.2.6 Common Mistakes

Summary: Every Rule the System Was Built From

Fifteen decisions, fifteen citations, two of them wrong on the first pass — which is the argument for opening the section rather than recalling it. One fork mattered: §5.2's buffer-absorbs-the-burst against §5.3's and §17.4's bound-it, and a crawler’s frontier is permanently over-fed rather than bursty, so §5.3 wins on the arithmetic. Three decisions had no owner at all, and all three are about lifetime in a system, which is the one thing a chapter about a mechanism structurally cannot teach.

Self-Check Questions: Every Rule the System Was Built From

22.3 What Happened

Twenty minutes, 1,200 samples, one crawler, one fixture. This section is reportage: what the nine series did, before any of it is interpreted.

22.3.1 The Run, at Seven Instants

And the work, over the same instants.

And the two sets the system keeps.

The last two rows are identical in every column that counts work.

22.3.2 The Six That Never Moved

goroutines   110 ─────────────────────────── 94
                 16 workers + net/http's own

threads       15 ──┐
                   └──── 23 ────────────────  23
                 §21.4: sockets, not syscalls

runnable       0 ─────────────────────────────  0
                 never above 1

GC pause    0.084 s total over 1,200 s
                 = 0.007% of wall clock

error rate  2.06% ──────────────────────── 2.06%
                 the fixture's own 2%

heap        72.6 MB ────────────────────  72.6 MB
                 across the final third

── all six correct; none of them knows the
   crawler stopped at t=562 ──►

Threads settling at 23 is §21.4.1 arriving exactly. That section measured 500 goroutines blocked on sockets costing six threads, against 64 in a genuinely blocking syscall costing 67, with the netpoller as the difference. Every fetch here is a socket.

22.3.3 The Two Rates, and the One That Warned

Fetched and dropped are counters, so their levels rise by construction. Their rates are the series worth reading.

t=61     fetch ████████████ 570   drop ████████████████ 1,880
t=181    fetch ████████████ 566   drop ██████████ 1,204
t=301    fetch ████████████ 584   drop ███████ 780
t=361    fetch ████████████ 583   drop ████ 544
t=440    fetch ████████████ 547   drop ██ 256
t=500    fetch ████████████ 547   drop  0
t=555    fetch ████████████ 551   drop  0
t=562    fetch  0                 drop  0

── the left column is flat until the instant it
   is zero; the right one had been falling for
   three minutes ──►

The fetch rate is what a throughput dashboard shows. It read 551 per second at t=555, 520 at t=558, and zero at t=562. Put a threshold under it and you get two samples of notice, both of which look like ordinary variance against a series that has been between 547 and 597 all run — which is why §22.6.1 scores it as no warning at all. The drop rate had been falling since t=361.

22.3.4 The One That Converged

The keyed limiter did exactly what §17.7.2 promised, and it is the clearest held prediction in the chapter.

The map peaked at 7,006 keys against a key space of 20,000 hosts, held a plateau near 5,000 while the crawl was active, and emptied when the work stopped. 46,827 evictions, and the sweeper’s two-phase discipline never appeared in any latency series.

22.3.5 The Heap, and the Column That Explains It

Heap in-use is the noisiest series and the easiest to misread: 2.2 MB at the start, a peak of 135.8 MB, 72.6 MB across the final third. Read against the collector it means nothing. Read against the dedup set it is arithmetic — and §22.4.4 is what happens when you try to state that arithmetic as a band on a value.

The number that matters is not the total. It is that two-thirds of those entries are URLs the crawler never fetched, and that after the crawl stopped the 72.6 MB did not come back for 639 seconds.

22.3.6 Common Mistakes

Summary: What Happened

The crawler ran at 565 fetches per second for nine minutes, discovered 900,219 URLs, fetched 315,945, dropped 577,780, and then stopped and ran for a further 639 seconds doing nothing. Six health signals stayed flat throughout — goroutines, threads, runnable, GC pause, error rate and heap — all six correct and none aware. The keyed limiter converged and emptied exactly as §17.7.2 said, with 46,827 evictions and no visible cost. The heap is explained entirely by the dedup set, and two-thirds of that set is URLs never fetched.

Self-Check Questions: What Happened

22.4 What Held, and What Did Not

Twelve predictions, registered with a falsifier each. This section publishes the score whatever it says, which was the condition of registering it.

22.4.1 The Scorecard

Nine held, three failed. P6 and P10 are one event — the crawl stopping at t=562 — observed in two series. P4 is separate and is the interesting one, and P7 held for a reason worth its own subsection.

22.4.2 What Eight Out of Twelve Establishes

The temptation is to read a high score as a victory lap. It is worth being precise.

It establishes that the book’s mechanism-level advice composes. Nine predictions were derived from eight different chapters and the fixture’s own configuration, applied to a system none of them had seen, and came true within tolerances registered in advance. §21.4's thread arithmetic, §17.7.2's sweep, §17.1.3's Little’s Law and §20.6's allocation model all transferred without adjustment. The common failure of a book of rules is that each is true and the set is jointly useless; that is not what happened.

It does not establish that the system worked. It fetched 35% of what it found and stopped. Every prediction that held was about a mechanism, and the mechanisms were fine — which is exactly the chapter’s problem, because the failures were not in any mechanism.

22.4.3 The One That Was Withdrawn

P6 predicted the frontier would stay saturated. It drained at t=562 and the crawler stopped.

The tempting report is that a crawler built by the book silently dies. Before writing that, one question had to be answered: the graph has a million pages and the crawler had seen 900,219 of them. Was the stop a failure, or the fixture running out?

The frontier never drained and the crawler never stopped. The termination was the fixture, not the crawler, and the finding is withdrawn.

That is the most useful five minutes in this chapter. A control took a dramatic result away — and in the same table made the surviving finding worse, because exclusion rose from 64% to 81% as the graph grew.

22.4.4 P4, Which Failed for the Interesting Reason

P4 is the one worth its own subsection, and this chapter’s first pass scored it held — by checking the endpoint rather than the samples. On the stricter reading its own falsifier registered, it fails.

And the claim inside it is correct.

as a value   heap within +-50% of seen x 100 B
             96 / 560 samples outside
             ── FALSIFIED ──

as a slope   heap = 7.5 MB + 105.7 B x seen
             R2 = 0.796, coefficient inside the
             predicted 100 +- 50
             ── CONFIRMED ──

── one hypothesis, one dataset, two verdicts ──►

22.4.5 P7 Held, and Its Falsifier Was Wrong

P7 was scored Half on this chapter’s first pass, and there is no such verdict. Inventing a third category after the run is exactly what pre-registration exists to prevent, so it is worth doing properly.

And the prediction was nearly useless, because the falsifier only looked one way.

drop rate rises   ──► falsified   ✓ anticipated
drop rate flat    ──► held        ✓ anticipated
drop rate falls   ──► held        ✗ not considered
drop rate zero    ──► held        ✗ and it is the
                                    failure itself

── the series that gave 201 s of warning is the
   one this prediction was blind to ──►

22.4.6 What the Scorecard Cannot Tell You

Three things survived every check, and none of them was predicted, because no chapter is about them:

• The exclusion is real and scales the wrong way. 64% of found pages never fetched at a million; 81% at ten million.
• The dedup set is real. 900,219 entries, 72.6 MB, retained 639 seconds after the last fetch.
• The instrumentation gap is real. In both runs the error rate reads 2.06% and §17.7.5's Four Numbers are healthy while most of the work is being discarded.

22.4.7 Common Mistakes

Summary: What Held, and What Did Not

Nine of twelve predictions held, drawn from eight different chapters and the fixture and applied to a system none had seen — §21.4's thread arithmetic, §17.7.2's sweep, §17.1.3's Little’s Law and §20.6's allocation model all transferring unmodified. Two of the three failures were one event, and a five-minute control showed that event was the fixture exhausting rather than the crawler failing, so the finding was withdrawn — while the same comparison made the surviving one worse, from 64% to 81%. The fourth failure is the chapter’s thesis arriving inside its own scorecard: a correct hypothesis, refuted because it was stated as a band on a value and confirmed at R² 0.796 when stated as a slope.

Self-Check Questions: What Held, and What Did Not

22.5 The Failures That Only Exist in Time

Everything that survived the control has one shape, and it is the shape no instrument in this book can see. A benchmark measures a function. A profile samples an instant. Neither can express a quantity that is fine at every moment and wrong across an hour.

22.5.1 The Queue That Feeds Itself

The frontier is a queue whose producers are its own consumers: every fetch removes one URL and adds up to eight. Its steady state is growth proportional to the fan-out minus one, permanently.

What makes that a finding rather than an obvious consequence is that the book tells you to bound it in three separate places, and every one of those places assumes something this queue does not have.

§5.1  an unbuffered channel gives backpressure
      ── assumes the producer can be made to wait

ch7   a pipeline is bounded by construction
      ── assumes a DAG; no stage feeds one behind it

§17.5 capacity = deadline x drain rate
      ── assumes a caller with a deadline

the frontier: the producer IS the consumer, the
graph is a cycle, and there is no caller at all

── all three preconditions fail at once ──►

Backpressure needs a producer you can block, and here blocking the producer blocks the consumer, because they are the same goroutine. A pipeline is bounded because work flows one way; this graph is a cycle. And §17.5's sizing formula needs a deadline owned by somebody outside the system, and a crawler has no such caller.

22.5.2 Drop, Grow, or Deadlock — There Is No Fourth

Three implementations, all built, all measured.

Block. Remove the default: branch so a full frontier makes the sender wait. Measured — TestBlockingFrontierStalls, eight workers, capacity 64, fan-out 6:

Fourteen items on that run, fifteen on the next, and on every run fewer than one queue’s worth of a frontier holding 64 — and then nothing, for as long as you leave it running. Every worker is blocked on a send into the queue it is the only consumer of, and nothing drains it because everything that could drain it is blocked filling it. This is §10's circular wait with one participant class rather than two, and go vet and -race are both silent throughout — the goroutines are correctly synchronised and permanently stuck.

Grow. Keep an unbounded frontier. Measured — TestGrowingFrontierIsUnbounded, the same shape with a slice instead of a channel, for half a second:

The queue holds five entries for every item completed, and five is fanout − 1. That ratio is the part that travels: it came out at 5.0 to four figures on every run here, and on a slower machine that reached 6.7 million entries instead of 9.7 million it was still exactly 5.0. It does not converge, because nothing in the loop removes more than it adds. The absolute is not structural — it is whatever your machine completes in half a second — but the growth rate is (fanout − 1) × throughput whatever that number is, and at roughly 100 bytes an entry a machine of this speed reaches a gigabyte inside a minute. §17.4's slower failure, arriving exactly on the schedule the arithmetic predicts.

Drop. Bound it and discard the overflow. That is what this crawler does, and §22.5.3 is the bill.

grow    unbounded queue     ──► memory, without limit
drop    bounded + discard   ──► 64-81% of the corpus
block   bounded + wait      ──► deadlock, every time

── the book recommends "bound the queue"; two of
   the three ways to do that lose ──►

The reusable form: when a queue’s producers are its own consumers, the choice is not bounded or unbounded. It is drop, or grow. Blocking is not a bound — it is a deadlock with a capacity argument — and whichever of the other two you pick has to be counted.

22.5.3 The Ratio Nobody Exports

Measured, this run: 900,219 URLs discovered, 315,945 fetched, 577,780 dropped. 64% of everything the crawler found, it recorded permanently and fetched never — and 81% on the ten-million-page control.

The mechanism is two correct rules meeting, and it is six lines long.

// Illustrative snippet — not a complete program
func (c *Crawler) enqueue(u string) {
    c.mu.Lock()
    if _, ok := c.seen[u]; ok {
        c.mu.Unlock()
        return
    }
    c.seen[u] = struct{}{}   // recorded here
    c.mu.Unlock()

    select {
    case c.frontier <- u:
        c.stats.Enqueued.Add(1)
    default:
        c.stats.Dropped.Add(1)   // and dropped here
    }
}

§5.3 says bound the queue, and it is bounded. §11.3.3 says the check and the insert are one compound operation under one hold of the lock, and they are. Both rules are followed and neither is wrong.

The consequence lives in the gap between the two statements. A URL is written into seen before anything knows whether the frontier has room for it, so a URL discovered while the frontier is full is recorded permanently and never queued. It cannot be rediscovered, because the dedup set will reject it the next time a page links to it. The crawler has silently decided never to visit two-thirds of the graph, and the line that decided it is an unlock.

The repairs all trade something, and the book supplies none of them because each is about lifetime rather than mechanism. Mark on dequeue instead of enqueue, and the frontier holds duplicates. Keep a rejected set, and it is a second unbounded structure. Block, and §22.5.2 measured what that does.

22.5.4 The State You Cannot Delete

The dedup set is the one structure in the system with no lifetime, and that is not an oversight. It is what deduplication is: a set that forgets is a set that lets the crawler revisit.

Everything else here expires. §12.3's pool is emptied by the collector. §17.7.2's limiter map is swept — 46,827 evictions. Contexts are cancelled. The seen-set is deliberately permanent, and nothing in 546,000 words prices it.

And if you do decide to evict, one more measured surprise. Measured — TestMapDoesNotShrinkOnDelete and TestReplacingTheMapReturnsAllOfIt: two million string keys inserted into a map[string]struct{}, then either every key deleted or the map itself replaced, with forced collections between readings.

len(m) is 0 after the deletes. The map holds nothing. It is still 112.0 MB, and deleting every key returned 45.8% of what the map cost. Assigning a fresh map over it returns all of it.

A Go map does not shrink. Its buckets are allocated as it grows and are not released when entries are removed; only replacing the map returns the memory. So an eviction policy that deletes from the map it is bounding does not bound memory — it bounds len, which is not the number anyone cares about.

Both rows ship as tests, and getting the second to measure anything took §20.1.2's rung 4. Without an observation of the map between filling it and reading the heap, the collector is entitled to take it first — the “before” reading came back at 0.5 MB, and the experiment measured an empty program twice and reported 0% returned. Chapter 20's dead-code trap has a memory-profiling twin, and it is the quieter of the two, because a suspiciously fast benchmark at least looks wrong.

22.5.5 A Value Can Refute What a Slope Confirms

§22.4.4 has the measurement; this is what it generalises to.

P4 asserted a relationship between two growing quantities and stated it as a band on an instantaneous value. Across the crawling phase the band was violated in 17% of samples, and the relationship it asserted is true at R² 0.796 with a coefficient inside the predicted range.

heap_inuse  =  live set
            +  garbage not yet collected    <- GOGC
            +  a doubling map's old array   <- runtime

the second and third terms are amplitude, not
trend.  They average to nothing over the window
and they are tens of MB at any instant.

── a band on the instant tests the amplitude;
   a slope tests the claim ──►

The rule that survives: state a claim about accumulation as a slope with a coefficient. “Heap stays under 400 MB” is a claim about the collector. “Retention is 105 ± 50 bytes per remembered URL” is a claim about your data structure, it is refutable, and it is the one that transfers to a machine with a different GOGC.

22.5.6 Why No Instrument in This Book Can See Any of It

Chapter 19 taught four views and Chapter 20 taught benchmarking. Against this class each fails for a structural reason rather than a fixable one.

benchmark   ──► how long does this call take?
                 correct, and the call is fine
CPU profile ──► where is time going right now?
                 nowhere; the system is idle
heap        ──► what is resident right now?
                 72.6 MB, all of it reachable
dump        ──► where is everyone right now?
                 parked, exactly as designed
-race       ──► is access synchronised?
                 yes, and it always was

── every one asks about NOW; the failure is a
   quantity accumulated over an hour ──►

The common property is that every instrument in this book answers a question about an instant, and each of these failures is a property of an interval. That is not a gap in the tooling so much as a category difference, and naming it is the most portable thing in this chapter.

22.5.7 The Shape of the Class, and What to Do

Five failures, one shape:

• A quantity that only accumulates. Excluded URLs. Correct at every instant.
• A quantity that should fall and does not. Retained dedup memory after the work ends — and delete returns less than half of it.
• A rate whose zero looks like health. Frontier depth zero, in-flight zero.
• A claim whose form cannot survive the runtime’s amplitude. P4 as a band on a value (§22.4.4).
• A claim that could only be wrong in one direction. P7's falsifier, blind to the collapse that was the failure (§22.4.5).

The last two are failures of measurement rather than of the system, and they belong on the list because they arose the same way: an instrument shaped for a value, pointed at an interval.

The diagnostic is a question you can ask of a design before you run it: does this system have a quantity whose correct value depends on the whole run rather than on the moment? If yes, it needs a series, not a gauge.

Three rules follow:

Export at least one rate. A level’s zero is ambiguous between idle and dead. A rate’s zero is not.

Export what you discard. Anything a system drops, sheds, rejects, evicts or times out is work that left without a trace. Here that one series would have given 201 seconds of warning; §17.7.5's four gave 54.

Give every retained structure an owner and a bound, or write down that it has neither. §17.7.2 does exactly that for the limiter map, and it is the one structure here that behaved.

22.5.8 Common Mistakes

Summary: The Failures That Only Exist in Time

The frontier is a queue whose producers are its own consumers, and all three of the book’s bounding rules assume a precondition it violates: backpressure needs a producer you can block, a pipeline needs a DAG, and §17.5's sizing needs a caller with a deadline. So the choice is drop, grow, or deadlock — and blocking, measured, stalls short of one queue’s worth with -race silent. Dropping costs 64% of the corpus at a million pages and 81% at ten million, because §5.3's bound and §11.3.3's compound check together record a URL before discovering there is no room for it.

The dedup set is the one structure with no lifetime — 900,219 entries, 80.6 bytes each, two-thirds for pages never fetched, still resident 639 seconds later — and measured, deleting every key from a two-million-entry map returns 45.8% of its cost while len reads zero, and replacing the map returns 100%. The fix is to replace rather than delete, which is §21.4.1's cached-threads insight wearing different clothes.

And a correct hypothesis about all of it was refuted because it was stated as a band on a value; as a slope it holds at R² 0.796.

Self-Check Questions: The Failures That Only Exist in Time

22.6 Which Signal Moved First

§17.7.5 names four numbers to instrument. §19.4.6 adds the runtime’s. Neither says which one moves first, and that is the question the person on call actually has. This section does not issue a third prescription; it reports an ordering.

22.6.1 The Ordering, Measured

The run stopped fetching at t=562. Working backwards through the nine recorded series:

drop rate     ████████████████████████████  201 s
frontier      ███████                        56 s
fetch rate    ▏                               0 s
everything    ▏                            never
else

── the longest warning is a series no chapter
   asks you to export ──►

Frontier depth is one of §17.7.5's Four Numbers and it bought 56 seconds, which is that section doing its job. Throughput — the number most services alert on — read 551 per second five seconds before it was zero.

The signal that led by 201 seconds is the drop rate, and it appears on no list in this book. That is not a criticism of §17.7.5: a chapter about rate limiting cannot know that your system discards work somewhere else. It is §22.2.3's gap in its most expensive form.

22.6.2 It Moved the Wrong Way

There is a second, sharper problem with the one signal that did warn.

§17.7.5 argues for queue depth as a way to see a service accumulating an unbounded backlog — the difference between a service at its limit and one falling behind, which have identical admitted-rate graphs. That framing makes a rising queue the signal.

Here it fell. From 4,095 to 0, and the fall was the warning.

Only the last row is a problem, and it is the only one a depth alarm never fires on.

Which is why the four readings above need two numbers rather than one. Depth alone is ambiguous in both directions. Depth paired with a rate resolves all four, and the pairing costs nothing because both numbers are already exported.

22.6.3 What the Other Signals Were Worth

Six of the nine series never moved at all, and it is worth being precise about what that means rather than dismissing them.

They were not useless — they were exonerating. Goroutines flat ruled out a leak in the pool. Threads flat ruled out §21.4's blocking-syscall growth. Runnable at zero ruled out processor starvation, which §19.4.6 pairs with the limiter precisely to distinguish. GC negligible ruled out collector pressure. Each answered a real question, correctly, in the negative.

The mistake is not exporting them. It is treating a set of green exonerations as a positive statement about health. Six correct “not this” readings do not add up to one “working”.

22.6.4 Common Mistakes

Summary: Which Signal Moved First

The drop rate turned 201 seconds before the failure; §17.7.5's frontier depth turned 56 seconds before it; throughput turned zero seconds before it, reading 551 per second five seconds out. Six of the nine series never moved at all. The signal with the longest warning is on no list in this book, and the one that did warn moved in the direction nobody alarms on — a falling queue rather than a rising one, which is indistinguishable from spare capacity unless it is paired with a rate.

Self-Check Questions: Which Signal Moved First

22.7 Where This Book Stops

This is the last section of the last chapter, and its job is to say what the preceding 546,000 words do not cover — deliberately, so that the boundary is a decision rather than a gap you discover.

22.7.1 What This Chapter Did Not Test

The experiment is narrow and its limits should be stated before its conclusions travel.

One machine. Every figure is from one Intel i7-10700K at GOMAXPROCS=16. §22.3's absolute numbers are that machine’s; the directions and the ratios are what transfer, which is Chapter 20's rule applying to itself.

One workload shape. A crawler discovers more work than it completes. A system whose work arrives from outside — a server, a consumer — has a different frontier problem, and §22.5.1's three-preconditions analysis is exactly the tool for deciding whether yours is the same shape.

One fixture. §22.1.1 states its parameters precisely so that a reader can judge, and §22.4.3 withdrew a finding that turned out to belong to it. There may be others; the parameters ship so you can look.

Twenty minutes. Long enough for the constraints in §22.1.3, and not long enough for anything whose period is hours. A daily cycle, a weekly cache warm, a monthly key rotation: all invisible here, all real.

22.7.2 The Boundary, Named

one process, many goroutines   ── chapters 1-21 ──►

─────────────── the boundary ───────────────

many processes, one cluster    consensus, replication,
                               partial failure, clocks

── the second is not more of the first.  It is a
   different subject with different primitives ──►

Concurrency inside one address space has a memory model, a scheduler and a set of primitives this book has spent its length on. Distributed systems share none of them: there is no happens-before edge you can establish with a mutex, no scheduler you can read, and failure is partial rather than total. A sync.Mutex and a distributed lock share a name and nothing else, and treating the second as the first is one of the more expensive mistakes available in this industry.

This crawler is a good illustration of where the line falls. Every failure in §22.5 is inside one process. Run the same crawler on twenty machines and the dedup set cannot be a map at all — it becomes a distributed set, with a consistency model, a failure mode when a node dies holding uncommitted membership, and a completely different answer to “have we seen this URL”. Nothing in Chapters 1–21 helps with that.

That boundary is where this book stops, and the CoreLabs distributed-systems course is where the other side begins.

22.7.3 What the Appendices Owe You

Two things this chapter deliberately did not do, because they belong elsewhere.

A decision index. Every fork this book established — mutex against channel against atomic, buffered against unbuffered, context against done-channel, errgroup against collect-all, pool against stack — with the section that owns it and the measurement that decides it. That is a reference, not an argument, and putting it in a chapter would make it harder to find rather than easier. It is the appendices' job.

A translation from other languages. What a Go channel is and is not, for somebody arriving from threads and condition variables, or from async/await. Also reference, also the appendices'.

22.7.4 When to Stop

§20.7.9 answered this for optimisation. The system-level version is shorter and it is the last thing this book has to say.

Stop when the next thing you would measure has no decision attached to it. This chapter recorded nine series and used four. The other five were exonerations (§22.6.3) — worth having, not worth extending. A tenth series with no action behind it is a dashboard tile, not an instrument.

Stop when the failures you are finding are properties of your harness. §22.4.3 is the worked example: a striking result, five minutes of control, a withdrawal. When two consecutive findings evaporate under a control, the harness is what you are studying.

Stop when the remaining risk is not in this book’s subject. The crawler’s next real problem is not concurrency; it is politeness, robots.txt, content dedup and storage. Knowing that the concurrency is finished is itself a result, and it is the one this chapter was built to produce.

22.7.5 Common Mistakes

Summary: Where This Book Stops

The experiment is one machine, one workload shape, one fixture and twenty minutes, and each of those is a limit on how far its conclusions travel — the directions transfer and the absolutes do not. The boundary of the book is the process: everything here has a memory model, a scheduler and primitives that a cluster does not share, and a sync.Mutex and a distributed lock have a name in common and nothing else. The decision index and the cross-language translation belong to the appendices, deliberately. And the system-level answer to when to stop is that you stop when the next measurement has no decision attached, when your findings have started belonging to your harness, or when the remaining risk is no longer this book’s subject.

Self-Check Questions: Where This Book Stops