testing
Seven ways the harness lied to me (in one day)
One debugging day, seven instrument failures: a probe that double-counted its own bookkeeping into '93 duplicate requests', a WebGPU canvas that reads back as all zeros, a parity test that passed by comparing two empty frames, a screenshot taken 7.5 seconds too early, and three more. Each with the tell that gave it away and the cross-check that caught it — because the same day also proved that one of those 'flakes' was hiding a real bug.
By the X-GIS team 6 min read
Everything below happened in one working day on one codebase, while investigating a network-overload report, a blank hemisphere, and floating labels. None of these seven are engine bugs. All seven are the measuring apparatus — the Playwright harness, the readback path, the demo runner, my own eyes on a screenshot — reporting something false about the engine. Three lied toward panic, three lied toward comfort, and one managed both in the same afternoon.
I’m writing them down with the tell (what should have triggered suspicion sooner) and the cross-check that settled it, because the day’s real lesson is that every probe needs a second, independent witness.
1. The counter that counted itself
The network probe reported 93 requests, 93 duplicates — every tile
fetched exactly twice, which would have confirmed the “request queue fires
double” theory on the spot. The tell was the perfection: real duplication
bugs are ragged; a 100.0% duplication rate across three tile hosts is the
signature of the observer, not the observed. The cross-check was a second
instrument on the same run: a CDP Network.requestWillBeSent count, which
reported zero duplicates. The probe had registered both a route handler and
a response listener and recorded each request in both. A correction went to
the issue; every downstream byte number got halved.
2. The canvas that reads back as zeros
A pixel assertion failed: the readback was black, all channels, all pixels.
The frame was fine — the compositor screenshot (page.screenshot) showed
the full render. Under Chromium, drawImage/getImageData readback from a
WebGPU canvas yields transparent pixels; our e2e helpers even carry the
comment, which I had not re-read. The tell: an all-zero readback is not a
wrong image, it’s an absent one — bugs make pixels wrong, plumbing makes
them zero. Cross-check: judge only the compositor screenshot; treat in-page
canvas readback as a convenience that must first prove it can see anything.
3. The parity test that compared two empty frames
A backend-parity probe loaded a labels demo with ?e2e=1 and reported the
two backends pixel-identical. They were: both frames were empty.
?e2e=1 exists precisely to disable the demo runner’s fixture auto-push so
specs can register their own content — the probe borrowed the flag without
inheriting the obligation, captured two frames of background, and “passed”
at diffing nothing. The tell: a diff of exactly 0.0 between two different
GPU stacks is suspicious — even correct backends differ by antialiasing
noise. Cross-check: a content floor (assert non-background coverage) before
any comparison is allowed to mean anything.
4. The screenshot taken 7.5 seconds too early
A globe capture showed half a hemisphere missing. I nearly filed it as a render regression; a recapture with a 10 s settle instead of 2.5 s showed full coverage — the tiles simply hadn’t arrived yet. Timing flake, case closed.
Except the same silhouette came back later in the day as #1076, where a hemisphere genuinely never renders — fallback ancestors computed, uploaded, and then suppressed by a drape flag — and the proper verification had to induce the latency deliberately (a route handler delaying west-column tiles 20 s) to make the bug stable enough to observe. That’s the double lesson: a longer settle is the right cross-check for “is this loading or broken,” and it is not a licence to file every half-loaded frame under flake. A slow network and a dropped draw produce the same picture. One of them is a bug with a deadline; ask what the frame looks like when loading finishes, and force the race if you have to.
5. The downscaled screenshot that accused the wrong label
Reviewing the floating-labels fix, the after-frame — downscaled to fit review — showed Nigeria still floating, and I said so. Measurement disagreed: Nigeria’s anchor sat 11.7 px inside the projected limb, indistinguishable from healthy Kenya’s 11.8, and the ×5 native-resolution crop showed it flat on the disc. At review scale, the three grey pixels of a healthy near-limb label and a floater are the same three grey pixels. The same crop pass then found the real remaining floater (two-line Burkina Faso), so the cross-check paid twice: it cleared the innocent and indicted the guilty. Our verification rules already mandated native-res crops; I wrote the mandate and still eyeballed the thumbnail first.
6. The stale-server theory I never needed
Mid-investigation, results stopped making sense, and the config line
reuseExistingServer: true in playwright.config.ts became the suspect:
maybe an old dev server from a previous session was serving stale code. I
built a small theory on it before running the cheap check — was anything
listening on the port at session start? Nothing was. Fresh server, fresh
code, hypothesis dead in one command. The tell here is embarrassing in its
generality: the theory was attractive because it explained everything and
indicted nobody’s code. The cross-check for “stale environment” claims is
always the same and always cheap — verify the environment’s identity
(port, PID, build hash) before reasoning about its contents.
7. The green suite that refuted nothing
The floating-labels fix went through three rounds, and after rounds one and two the unit suite was fully green while the rendered frame still showed labels in the sky. Nothing was wrong with the tests; they proved exactly what they stated — the margin culled the band anchors, the limb inset culled the 2.9 px witness. What they stated was the round’s own theory of the bug. A test suite is the fix’s memory, not its judge; it can only refute what its author already suspected. The judge was the probe screenshot every time — which is why, on this engine, “the suite is green” has never been accepted as the last line of a fix’s evidence.
Instrument the instrument
The pattern across all seven: an instrument’s output was treated as ground truth because it was an instrument — numbers from a probe, pixels from a readback, green from a suite all arrive wearing lab coats. The failures sort into two families. False alarms (1, 2, 5, 6) burn time and, worse, spend your credibility on retracted findings. False comfort (3, 7) is the expensive one — it ships. Incident 4 sits in both, which is what makes it the keeper: the identical observation was a flake at 2.5 s and a real defect later, and only controlling the experiment (forcing the latency) separated them.
The working rule this day beat into the process: no finding is real until a second instrument that shares no code with the first agrees. Route handler and CDP. Readback and compositor screenshot. Downscale and native crop. Unit suite and rendered frame. The engine only has to be right once; the harness has to be right every single time you look through it — so look through two.
References
- “The pixel test that passed on the wrong GPU” — the ancestor of incident 3: an assertion that never pinned which system produced its pixels.
- “The boundary audit missed an edge because a regex ate a digit” — incident 1’s older sibling: the instrument artifact that reads as a finding.
- “The map that downloaded the world” and “The hemisphere that wasn’t there” — the two investigations these instruments were lying to.
Read next
rendering
7 min read
Draped at the wrong tile: a deferred write turned a shared uniform pool into a data race
A draped landcover polygon rendered hundreds of kilometres offshore — but only on WebGPU. Every vector slice of one source shares one Material uniform pool, and each slice resets the pool cursor to 0, so a later slice's tile i overwrote an earlier slice's tile-i buffer. WebGPU's queue.writeBuffer defers to submit (last-writer-wins), so every draw read the final bytes; WebGL2's immediate bufferSubData never showed it — and the differential test was structurally blind, because both frames aliased identically.
rendering
7 min read
The hemisphere that was never selected: a fast-path bbox that collapsed to a point
Half the globe rendered as background again — same symptom as the drape-suppression bug, opposite root cause. This time the trans-antimeridian tiles were never selected: the overzoom fast-path unprojects the viewport corners to a lon/lat box, but on a small globe disc the corners miss the sphere, the box collapses to the sub-camera point, and it emits the single tile column under the camera — which a contiguous longitude range can never wrap across the dateline. Told apart from the draw bug by seam location and persistence.
rendering
6 min read
The hemisphere that wasn't there: computed, cached, uploaded, drawn nowhere
Half the globe rendered as pure background, cut exactly at the 0-meridian z1 tile boundary. The fallback ancestors for the missing half were computed, CPU-cached, and force-uploaded to the GPU — then drawn nowhere, because a drape suppression flag was scoped to the renderer instance instead of one dispatch. Verified by making the network slow on purpose, after the first pixel readback lied.