~/studies/work 2026-11-09 · published 2026-11-21

Where the wall time went

hakoniwa was meant to run browser tests in half the time of plain Playwright. It used 37 to 39% less CPU and the same wall time. A phase profile shows why, and what the follow-up decisions changed.

The target and the setup

ADR-0011 set a goal for hakoniwa's front-end test path: at least 50% less wall time and at least 50% less CPU-seconds than plain Playwright with default settings, with the same verdicts. Until measured, the ADR calls every gain a hypothesis.

The benchmark app is a small React SPA with a Bun API, a second-origin widget and seeded faults. The suite has 17 specs. Each configuration ran in a container limited to 4 CPUs and 6 GB, with the Playwright v1.65.0 image. The machine is an AMD Ryzen 7 5700X with 16 threads and 47 GiB of RAM, Linux 7.2.5, Docker 29.7.2. Healthy runs are the median of 3. The fault run is a single repetition.

  • A: plain Playwright, 8 workers on 4 CPUs.
  • B: the engine on Chromium with every shortcut off, 4 slots, batch 5.
  • B+: B with the bench-only admitted shortcuts: session store, asset cache, CDP blocking, motion freezing.
  • C: B+ with Lightpanda admitted.

Result

ConfigWall svs ACPU-svs APeak MB
A5.3-16.4-1687
B5.1-4%10.4-37%817
B+5.2-3%10.2-38%816
C5.1-5%10.1-39%627

Verdicts matched A in every repetition. CPU fell by 37 to 39% and memory by about half. Wall time stayed within noise. The shortcuts cut response bytes from 3.9 to 1.0 MB, so the CDP blocking works, and they changed neither time nor CPU on this suite. The target was not met.

Where the time goes

A traced run of B, one repetition, shows the first step reported about 1.8 s after the lease loop started. The pieces the trace attributes: listing the test ids of the group takes about 0.26 s, the load inside the test run another 0.24 s, and forking Playwright workers and launching their browsers about 0.3 s. The 17 specs then run for about 2.5 to 3 s on four workers.

So roughly a third of a 5 s suite is a fixed start that the engine path pays and plain Playwright pays less of, because it forks workers directly. A suite this short cannot amortise it. The engine saves CPU by running one browser per worker, 4 workers against A's 8, and without a per-slot browser server or test server. Saving CPU does not shorten a start-up chain.

Under a seeded fault it was worse

ConfigWall svs ACPU-svs A
A9.3-17.1-
B15.0+61%15.5-9%
B+15.2+63%15.0-12%
C21.5+130%18.4+8%

The fault was a search that ignores its query, failing 3 of 17 specs, one repetition per configuration. The cause is a design choice in the execution path. Each failing spec was run again with its trace on, so a failure cost two attempts, each waiting out the 5 s expect timeout, plus a new worker and browser after each failure. C was slower still: Lightpanda attempts failed first, fell back to Chromium and ran again.

What the negative result changed

The result fed directly into ADR-0019, written the same day. It removes the rerun on failure: the first attempt keeps a lean trace and the failing run finishes, so flakiness is judged across runs and not inside one. The ADR lists keeping the rerun as a rejected option, because it made failing suites 61% slower than a baseline that does not rerun.

It also drops the separate listing pass, keeps one spare runner started and idle, and sweeps slot counts of 4, 6 and 8. All three savings are labelled hypotheses in the ADR until a heavier benchmark shows them. The only follow-up measurement in the records is on 60 specs, 2 repetitions: the lean trace cost 64.3 and 70.0 CPU-s against 68.2 and 71.0 with no trace, inside the run-to-run noise.

I have no record of the heavier benchmark meeting the 50% target. The claim today is lower CPU and memory at equal verdicts, not faster wall time.