Hazel performance analysis · ^report(6) · 2026-10-12

Catching performance regressions in CI

The second question from a meeting with Cyrus: ^report(1)–^report(5) found and fixed performance problems by hand, one profile at a time. What stops them coming back? This report covers what Hazel's CI checks today and why its two benchmark workflows never worked. It explains why a time can't fail a build on GitHub's runners and a count can, and describes the first count-based check (#2678). It also weighs building on #2161, an earlier benchmarking PR, against starting fresh.

Short answers.

Nothing in CI catches a performance regression today. No check is required to merge into dev at all; the tests run on every push, and a red run is a signal, not a gate.
The two benchmark workflows on dev, /perf and /bench-compare, never worked there. They came from #2161 without the scripts they call. /perf also ran a pull request's own code for anyone who commented. #2678 turns them off.
A time can't fail a build on GitHub's shared runners; a count can. The full test job's time varied by about 10% across the last five dev runs, and the quick job once ran 40% faster than usual. The same program does the same amount of work on every machine.
The first count-based check is in #2678: the evaluator's steps, held to budgets. A cold run takes 206,834 steps. A re-run after an edit takes 378, and must stay under 5% of a run. With reuse switched off, the re-run takes 206,834 steps and the test fails.
#2161 or fresh? We started fresh for the check, because #2161 measures time. Its phase-by-phase timings would suit a later report that informs but does not fail the build (below).

Update, later on 2026-10-12: a count is not a time. A count of evaluation steps has no noise, but it can disagree with the time, in either direction. In the woods, 26% fewer steps made the view only 6% faster. Turning the view's call memo off made a woods change about 25% faster, while a step count would have called it a regression. So a budget that fails is a reason to time the change, not a verdict on it. The numbers.

What CI runs today

Workflow	When	What it does	Time
`ci.yml` → `checks.yml`	Every push, to any branch	Fast checks: formatting, a release build, the quick tests. Full tests: the whole suite with coverage.	19–28 min
`fork-ci.yml`	Pull requests from forks	The same checks, without coverage
`deploy.yml`	Every push	A release build, published to hazel.org/build	4 min
`extended-tests.yml`	Sundays, or by hand	The property tests at 100 times their usual size	2–3 h
`perf.yml`	A `/perf` comment	Meant to benchmark a PR against its base	never worked on `dev`
`bench-compare.yml`	A `/bench-compare` comment	Meant to compare two commits' stored benchmarks	never succeeded
`update_deps.yml`	Daily	Updates dependencies in a PR	10–14 min

dev's branch protection lists no required checks. So even a failing test is information, not a block.

The two benchmark workflows

/perf builds the PR and its base and runs bench/build-and-run.sh on each. That script is only on #2161's branch, perf-bench. On dev, the workflow exits with "No such file or directory" on both sides; that was the last run, on 09-30. Its last two real successes were on #2161, on 2026-03-06. One on 08-12 "succeeded" with nothing to compare against: the base side had failed to build, so every row was new.
/bench-compare looks for stored results named bench-<sha>. Only #2161's branch has the workflow that stores them, and none exist now. It has never succeeded.
Neither could fail a build. The comparison script marks a change of more than 10% and 0.5 ms with an emoji, and always exits successfully.
/perf ran a PR's own code (npm install and the benchmark script) for any commenter, with a token that can write to the PR.

A CI audit noted the missing scripts on #2161 in August. #2678 turns both workflows off: they now start only by hand, and then their job is skipped. A comment in each file says why, and what reviving it would need.

Why not times

A regression worth catching is often 10–20%. That is the size of the noise on GitHub's shared runners. For the last five dev runs, with nearly the same tests (4,699–4,704):

the full test job took 1,110–1,453 s: a mean of 1,323 s and a standard deviation of 127 s, about 10%;
the quick job took 145–147 s four times, and 87 s once.

Each run was a different commit, so some of that spread is code. But a single timing per build cannot tell a 10% regression from a slow machine. Timing the base and the PR on the same runner, interleaved and repeated, narrows it. That costs a second build and a long job, and still gives a judgment, not a pass or fail. Times belong in a report.

Counts that can fail a build

The same program, on the same code, does the same work on every machine. A count of that work can be held to a budget with no false alarms. These counts exist or are close:

Update, later on 2026-10-12: no false alarms from noise, that is. A count can still disagree with the time, in either direction (below).

Count	What a rise means	Where
Evaluator steps actually taken	more evaluation, or reuse that stopped working	`dev`, with #2678
Statics analyses per expression	an expression analyzed again, as module and livelit definitions were, twice each, until #2651	#2635: `Test_StaticsPasses`, with per-file budgets
Work per edit: pieces reparsed, items re-analyzed	an edit recomputing more than it touched	#2635: `Test_EditLocality`, `Test_BenchStatics`
Bytes: an elaboration's size; the worker's messages	larger values crossing between the threads (`^report(5)`)	#2635: `Test_ElabSize`; messages measured by hand so far

What counts miss: a step that got slower, and the page's own costs (virtual DOM, layout), which need a browser. Those stay with timings.

Budgets as a ratchet. An exact count fails on improvements too. A count on #2635 fails now with "expected 2, received 1", although 1 is better. A budget with room above the count catches a rise. A floor a quarter below the budget catches an improvement, and the failure says to lower the budget, in the same change. So each budget stays close to the truth, and the history of the budgets is a history of the improvements.

The first check: #2678

Evaluator.evaluate_counting_work is evaluate, also returning the steps its trampoline actually took. A result taken from the cache counts as the one step that fetches it.

The evaluator already had a step count, but it is the wrong one for this. To keep probe samples' step numbers in line, it counts a reused subtree's steps again, so it is the same whether anything was reused or not.

A new test group, PerfBudgets, runs in the quick suite in under a second:

Case	Steps now	Budget
A cold run: recursion, list traversals and case matches (a smaller `bench/eval-bench.hz`)	206,834	227,000
A re-run after an edit to a literal nothing else reads; also under 5% of a cold run, whatever the budgets	378	420

It catches what it is for. With reuse switched off in IncrEval.reuse_check, the re-run takes 206,834 steps, and the test fails: "over its budget of 420. A performance regression". This is the failure ^report(4) and ^report(5) kept finding by hand, a change re-evaluating everything, invisible to tests that only check the answer. The counts are the same on every run.

#2161: build on it, or start fresh?

#2161, "Performance benchmarking", by David Moon, is a draft opened on 2026-03-05. Its last commit is from 2026-03-13, and it conflicts with dev. It has 48 commits: +2,376 and −323 lines in 30 files.

Its harness, bench/hazel_bench.re, times the editor's syntax and statics phases: zipping, parsing (MakeTerm), layout (Measured), type checking, elaboration. It runs them on let-chains of 100 and 500 bindings, in four scenarios: cold, warm, after a cursor move, and after an edit. Each is the median of 10 runs. It posted result tables on its own PR in March and August. The PR also replaces structural memoization with memoization by pointer, in MakeTerm, Measured, highlighting and elsewhere. The two workflows on dev came from it.

	Build on #2161	Start fresh: counts in the test suite (#2678)
Measures	Time per phase of the editor's syntax and statics	Work done: evaluator steps now; statics analyses and bytes next
Can it fail a build?	No. Its threshold (10% and 0.5 ms) is the size of the runners' noise, and its comparison never fails	Yes, with no false alarms: a count over budget fails the test
Covers where `^report(1)`–`^report(5)` found the costs	Partly: statics, yes; evaluation, reuse, the worker and the page, no	Partly: evaluation and reuse now; statics once #2635's counts reach `dev`; the page, no
Explains a regression	Yes: time by phase shows where it went	Partly: a count shows that work grew, and which test, not where the time goes
Sees a step getting slower	Yes, given enough repetitions	No
Programs	Synthetic let-chains	Small programs in the test; slides can be added
Cost per run	A separate job building base and head, about 11 minutes, on request	Under a second, inside the tests every push already runs
Safety	Ran a PR's code for any commenter	Only the normal CI
State	A conflicting draft that mixes the harness with unrelated memoization changes; it would need splitting first	One small commit on today's `dev`
Ownership	David's PR: reviving it is his call	Ours

The decision: counts for the check that fails a build, so we started fresh. #2161's timings by phase are the right shape for the other half: a report on a PR that shows where time went, run on one runner for base and head, repeated, and read by a person. If David wants that, the harness could be split out of #2161 for it.

Update, later on 2026-10-12: a count is not a time

Later the same day, work on the woods' picture view (^report(5)) measured steps and time side by side. Times are of the view's own evaluation on the page: a timer around it, on one build, comparing variants in turn. Changes in the whole redraw were too noisy to see.

Change	Steps	The view's time
The points strings made more directly (a function call fewer per number, no check per point)	99,621 → 74,055 (−26%)	780 → 735, 726 ms (−6%)
The view's call memo turned off, the woods, moving the figure to a new size	more: every reuse lost	about 700 → 555 ms (−25%)
The memo turned off, the woods, back to a size drawn before	more	about 230 → 600 ms
The memo turned off, Kids' Choice's face, an eye-size change	more	about 7–23 → 38–51 ms

Why they disagree. Steps are not equal: a variable lookup is cheap, a pattern match or a cast through the recursive Html type is not. And some work is no step at all. The call memo builds a key for each call it might reuse, putting the call's inputs in a comparable form and hashing them. In the woods those keys copied the whole picture three times on every change, about a quarter of the view's time, and none of it counted as a step. Statics, the page's drawing, garbage collection and memory are no steps either. Nor is copying. Giving a value a type rebuilt the whole value in one step: the woods' picture, 15,500 nodes, at every typed parameter it crossed. Removing that copying made the view 20–25% faster without changing its step count much (^report(4)'s update, #2679).

So a step budget can be wrong both ways.

A false improvement: a memo whose hits remove steps, while its keys cost more time than the hits save. More garbage for the collector, fewer steps, is another.
A false regression: removing such a memo, as in the woods above: more steps, less time. Splitting one expensive step into several cheap ones is another.

What it means for #2678: the budgets stay, since they are deterministic and catch the failure they are for (reuse that stops working). But a budget that fails is a prompt to measure, not a verdict, which the failure message already says: find it, or raise the budget with the reason. Decisions are made on time, with a timer around the part that changed and the variants compared in turn on one build. Two counts would make the budgets harder to fool: the memo's own work (the nodes its keys put in form), and memory allocated, if js_of_ocaml can report it.

What next

Statics counts on dev: the analyses-per-expression counter and Test_StaticsPasses, from #2635. Its corpus case runs only in the full suite. A small subset could go in the quick one.
The slides: a budget for the work after a figure move in the woods, where reuse failed before #2675.
Bytes: elaboration sizes on dev, and the worker's message sizes for a fixed interaction.
Make the check count: ask the maintainers to require the test jobs for merging into dev.
A timing report, informing, not failing: base and head on one runner, repeated, phase by phase, perhaps from #2161's harness.

How this was put together

The CI facts: the workflow files on dev (e12145c69b), dev's branch protection, and the logs of the runs named above, read with the GitHub CLI.
The noise: the "Test Successful in N s" lines of the last ten green dev runs (2026-09-22 to 10-09), of which the last five have nearly the same suite.
#2161: its description, files, comments and branch.
The counts: #2678's test, run twice, and once with reuse switched off; on the laptop, under node.
The numbers, for programs too: every table in this report, and what its numbers measure, is also in this page as JSON, in its <script type="application/json" id="report-data"> element, so a reader or an agent that fetches the page alone has the dataset. The same, with any raw files that survive, is in 006/data/.