Evening of 10-14: the autosave's print, B against C again, views in the worker

2026-10-14, Claude, while Matthew was away (nothing pushed). ^ticket(84), ^ticket(29), ^ticket(18).

Pushed, 2026-10-15 morning, on Matthew's word. #2635, now at 653ebc4adf: the autosave's print (ec9f125acd, was cbc585933b), the tests' startup (c053aec689, was 0dd613dbba) and the parser's count (653ebc4adf, was 3663ef22e6). #2597, now at f8ae6f2b9c: the same three (a8426b1893, 0322b09ca2, 7f62da773a), the undo compaction (d41c7a0298, was bc66ae5616) and the newest undo and redo entries kept whole (f8ae6f2b9c, was 3041fd0483). Each is the local commit named below, cherry-picked; the affected test groups pass on both branches. The views-in-worker prototype (^ticket(18)) is not pushed. Later that morning, #2635 too keeps its newest undo and redo entries whole (032fe3aca6): its undo and redo had taken about three edits' time, and now take under 0.3 s (^ticket(89)).

How it was measured

• #2635's head 9fb2bdf067, dev build, in ~/hazel-livelits-modular, served on 8341. Headless Firefox over WebDriver, a fresh profile per session.
• ^report(8)'s set-ups, again: the Full Body slide, the "arm length" slider, two warm-up changes and then twelve, each a real pointer drag to a length not drawn before. Release to redraw is from the pointerup to the first 200 ms poll at which the largest picture's markup differs. C: the whole program on screen. B: UX mode, every GUI row's chip off but arm_length's and the picture's (its row reads probe(drawn) while shown, drawn while hidden).
• ^report(8)'s harness did not survive, so it is rebuilt from the report's recorded method: fullbody_release.py, and profile18.py (a copy of scripts/profile_interaction.py with this interaction and UX mode added). Both are in ~/.cache/hazel-diag/ (with callers.py and callees.py, which read a profile's stacks around one function), and each session's runs in ~/.cache/hazel-diag/r29/. The numbers below are each session's median of twelve.

B against C, today (^ticket(29))

So on today's head B is slower than C again, by 0.6–0.7 s; ^report(8)'s fixed B (1.26 against 1.23 s) no longer holds. With ^ticket(84)'s fix the two are within run-to-run spread, B perhaps 0.1–0.2 s slower: livelit views run on the main thread in UX mode (live_html, about 380 ms of samples per change, profiled).

The autosave's print (^ticket(84))

• What: about a second after a change, the save alarm prints the program to text (PersistentZipper.to_string for a whole program, persist_divided for a divided one), through MarkerParse.to_text and Segment.to_string, with the zipper's manual refractors (the probes). Triggers.refractor_seg_to_seg_with was quadratic on the top-level let chain: ListUtil.sublist from the list's head at every node, and Skel.range down the Pre chain for every child. And Segment.to_string calls it again on every nested segment while any refractor is unplaced, building each segment's skeleton.
• Where it showed: in C the save lands just after the redraw (about 600 ms, delaying the next interaction); in B it lands inside the change (about 1 s), which was most of B's extra time.
• Fix: slices of an array, a skeleton's first or last index taken alone, and a segment with no refractor pieces at its own level returned at once (a term is wrapped only when its root piece, at that level, is a refractor's, so the output is the same). save_current per change, profiled: about 600 → 300 → 80 ms (the walk under it 1,594 → 710 → 38 samples over three changes).
• Tests: DocSlides.ReparseBackuptext (120, the slides printed with their probes and parsed back), DividedLaws (17), Parser.Segmented (6), FastParse (33) pass.
• Where: local branch perf/2635-autosave-print, commit cbc585933b. Not pushed: to #2635 on Matthew's word, then cherry-pick to #2597.

Views in the worker (^ticket(18))

• Today: half of each view already runs in the worker. For a divided program (UX mode), statics fold def.view(model) into the program for every livelit projector (UserLivelit.instrument_view, the projector ids unioned in at Workspace.re:591). The worker evaluates it to a ViewCmd and sends it back as the projector's sample. The page then runs that command to Html (ViewCmdRunner.run, in LivelitProj.live_html) on every redraw that misses its view cache, and livelits with splices are never cached. For a whole program, the projectors are not given to statics (CachedStatics.init_compositional takes probe_ids_of_zipper(z) without them), so there is no fold-in, and every view is evaluated on the page from its definition (memo_view).
• Done: MvuShape and ViewCmdRunner moved to the language library, so the worker, which links only language, can run views. haz3lcore keeps both names (include). MVU (69), UserLivelits (70), OpenValues, ResultView and IdentityMemo pass, and the timings are unchanged (B 1.40 s, C 1.34 s). Local branch perf/2635-views-in-worker, commit 61d031fd93, on top of ^ticket(84)'s fix.
• Tried, and much slower: the worker ran every view sample to Html before posting, as Pure(html), which the page's runner answers at once. B went from 1.40 to 14.2 s per change, and with whole programs given the projectors too, C went from 1.34 to 13.7 s. Profiled, 65–83% of the worker was that pass, for two reasons. A divided program targets every livelit projector, about forty here, where the page draws only the two on screen. And the cache, keyed on the command's identity, missed from post to post. Kept for reference on experiment/2635-views-in-worker-all (a7fbe23788, labelled slow, not for use).
• What it would need: the page sending the ids of the livelits it draws with each request, and the worker running only those; a cache keyed by the command's content (ValueHash), so that an unchanged view is never run again; and for whole programs, folding views in only for the livelits on screen. These are protocol and design calls for Matthew before going further.

Update, later the same evening: views in the worker, with reuse

Two more experiments on experiment/2635-views-in-worker-all, after the slow one above. All are measured against the same build with views on the page (perf/2635-views-in-worker: the autosave fix and the module move).

• Experiment 2 (9c7808c51e): per projector, the worker keeps its last command and Html, and reuses the Html when the new command is the same object or syntactically equal (Exp.fast_equal, which ignores ids, so a fresh run's rebuilt command still matches). Only views whose command changed are run.
• Experiment 3 (ede920b07e): whole programs fold their livelits' views into the worker's run too (CachedStatics.init_compositional given the projectors), so the page no longer evaluates views from their definitions (memo_view).

So: on a big picture, the main thread does about 540 ms (B) or 220 ms (C) less per change, and the redraw arrives no later. On a small slide, the redraw comes about 45 ms later, since the worker's run and the Html's trip to the page cost more than the page's own view did. Not checked: whether the page's optimistic entries still converge with the worker's Html (LivelitProj.user_view compares them with fast_equal); ordinary probes whose values happen to be view commands are converted too; the per-projector cache is never cleared. Questions for Matthew: run views in the worker for every livelit, or only for those whose view is costly (measured per view)? And the ids of the livelits on screen in the request, so the worker runs only those?

The recovering parser, and every test run's start (^ticket(46), ^ticket(85))

• Measured again: one MenhirCorpus case alone (QCHECK_LONG=1), #2635's head: eyes.hz 19.6 s and full-body.hz 76.3 s, not the 85–90 s and 452 s in the ticket. Those were inside the whole suite's process, or an older build; both PR branches have the token-at-a-time and segmented loading (from #2614 and #2610).
• Every test run's start (^ticket(85)): a CPU profile of that one case spent 20.7 of its 45.5 s in Str's matcher, at load: Test_Unproject.slides() picks the livelit slides with Str.search_forward, which is slow under js_of_ocaml. A plain substring scan replaces it; the same 42 slides are picked and Unproject's 47 cases pass. A one-test run went from 28.0 and 28.3 s to 6.4 and 6.4 s. Local branch test/2635-unproject-startup, 0dd613dbba.
• The segmented parser's count: after every insertion it took List.length of the whole rest of the text (2.9 s of eyes.hz's parse), so it was quadratic in the text's length. It now counts along the run. eyes.hz 19.6 → 15.8 s, full-body.hz 76.3 → 53.6 s. Parser.Segmented, DocSlides.ReparseBackuptext (120) and MenhirCorpus (119, each parse checked against Menhir's) pass. Local branch perf/2635-parser-count, 3663ef22e6.
• What is left of full-body.hz's 41 s parse: re-molding the sibling run on each insertion (remold 22 s), and reassembling it in unselect, even with an empty selection (14 s, from destroy_selection and directional_unselect). The second looks wasted for a parse, but the insert path may rely on it to pair a tile's shards, so it is left alone: a question for the editor's owners.

Memory per edit: #2635 against #2597 (^ticket(65), for ^ticket(58))

^report(14) found that a #2597 tab keeps about 92 MB more with every edit of Full Body's shirt-width slider, and asked whether that growth is #2597's own. The same driver (014/drivers/heap_per_edit.py: headless Chromium, two forced GCs after each edit, then the live JS heap) ran on #2635's exact head, 9fb2bdf067, rebuilt for the run.

• #2635 grows by a tenth as much, and falls back. It keeps 9.2 MB after the first edit and 93.6 MB after the tenth, then drops to 46.0 MB at the eleventh. It climbs to 108.3 MB by the 21st and drops to 53.1 MB at the 22nd. Over the same fifteen edits, #2597 keeps 1,388 MB and never falls back. An earlier 15-edit run on the same build of #2635 agrees with this one to within 0.1 MB.
• So the growth is #2597's own. Whatever keeps a version of the program per edit there is gone, or bounded, on #2635, which has dev and #2469's modular editors merged in. ^report(14)'s other leads now have a narrower search: what #2597 holds that #2635 does not.
• What falls back on #2635 is not yet known. The likely candidate is ShareCodec's tables of values shared with the worker, which start over at 200,000 entries; ^report(14)'s Figure 3 shows them rising and resetting on #2597 too. Confirming it takes ^report(14)'s probe build (patches/count_id_keyed_tables.patch) on #2635. The low points rose 7 MB in eleven edits; two low points are too few to call that a leak.
• The data: 014/data/update-2026-10-14/heap_2635_30.json (and the 15-edit run beside it), in the series' working copy, not yet pushed.

Update, near midnight: what #2597 keeps per edit is its undo stack (^ticket(58))

The answer. #2597's page-level undo stack (History.Model.undo_stack in src/web/app/History.re) keeps a whole page model for every edit, with that version's syntax, statics and layout. Its cap only applies when the setting cap_undo_stack is on, and that setting is off by default (Settings.re: cap_undo_stack: false). So ^report(14)'s undo test, which lowered the cap from 250 to 2, changed nothing: the cap never ran. dev has the same code. #2635 does not, because Andrew fixed it on the modular editors in 6010128b7d (2026-08-26, "memory: kill the exponential co_ctx leak + compact undo snapshots"): each snapshot drops its derived caches, they are rebuilt on undo, and the cap always applies.

• How it was found. First, what the browser itself holds does not grow: over six edits #2597's heap grew by 500 MB while its listeners (609), DOM nodes, detached ones included (47,923), observers and timers stayed the same. Next, ^report(14)'s sampled live heap, by source file, on #2635: of 37.5 MB kept over four edits, almost all is the worker codec, ShareCodec and ValueHash, against #2597's 390 MB of types, statics, layout and terms. A probe build of #2597 with ValueHash's memo living only within one synchronous run still grew 68 MB per edit, and made heap snapshots possible (2.3 GB of JSON, 75 s). Two snapshots in one session, after edits 3 and 6, give the objects allocated in between; a dominator tree of the second (29 million nodes, 4.6 s in C) shows three copies of one 35 MB structure, the Full Body editor's derived data. Two are the heads of undo_stack's first two entries (Logged.Model.current → undo_stack → each entry's page model → the slide list → Full Body). The third is under ProjectorView's view cache, which holds one.
• Confirmed. With the cap always on at 2, #2597 stops growing after the third edit: 284 MB kept after 12 edits, in two runs that agree within 10 MB, against 1,048 MB as built.
• The fix, ported. Local branch perf/2597-compact-undo, bc66ae5616, on #2597's head c4e411f9c2: Andrew's compaction (adapted to #2597's flat scratchpad list; the result kept through persist and unpersist, as #2635 now does), the cap of 250 always on, and the Undo tests checking the text, as on #2635. Full Body keeps 159 MB after 12 edits, level from the third edit. The six Undo tests pass. In headless Chromium, three runs of edit, undo, redo, a second edit and undo: every step restores the same picture and slider value as as-built #2597 does.
• Its cost, measured. An undo now rebuilds what the snapshot dropped. Five runs per build, alternating: with the compaction every redo took 6.0 to 6.5 s, about an edit's time, against 2.7 to 3.1 s as built, and half the undos did too (5 of 10 at 5.8 to 6.5 s; the rest 2.0 to 3.0 s, like every undo as built). Which undos stay fast was not examined; the whole-program memos may still hold the program being restored. The console's "HazelDOM: unrecognized attribute: Disabled" also appears in as-built #2597 (387 times a run), about twice as often on the port, which fits more redraws.
• An experiment that keeps the cost away. Local commit 3041fd0483 on perf/2597-compact-undo-keep-newest, on top of the port: the newest entry of each stack keeps its page model whole and is compacted only when the next is pushed above it. The latest undo and redo then take what they take as built (median 2.6 s over five runs, alternating with five more as built at 2.6 s; the console errors back to 387), and Full Body keeps 208 MB after 12 edits instead of 159. An undo further back still rebuilds. The six Undo tests pass. Which to push, Andrew's compaction as it is or with this, is Matthew's call.
• The data and tools: 014/data/update-2026-10-14/ (heap_asbuilt_8311, heap_undo2_always, heap_undo2_again, heap_compact, the undo checks, and holder/ with the dominator reports and the scripts), in the series' working copy, not yet pushed. The snapshots themselves (4 GB) are on this laptop only.

Where things are

• ~/hazel-livelits-modular is back on the PR branch, integration/livelits-modular-editors at 9fb2bdf067, rebuilt, and port 8341 serves it. The local branches: perf/2635-autosave-print (^ticket(84)'s fix), perf/2635-views-in-worker (plus the module move), experiment/2635-views-in-worker-all (plus experiments 1–3), test/2635-unproject-startup (^ticket(85)), perf/2635-parser-count (the parser's count), each on #2635's head.
• ~/hazel-2597-probe, a worktree of #2597: perf/2597-compact-undo (bc66ae5616, the undo compaction) and probe/2597-undo-cap-2 (the confirmation probe, not for pushing).
• Nothing is pushed: not ^ticket(84)'s fix, the module move, the experiment, or the undo compaction.

Next steps

1. Push ^ticket(84)'s fix to #2635 and cherry-pick it to #2597, on Matthew's word.
2. Revise ^report(8) with these numbers and graphs (^ticket(29)); the draft waits for review, and its answer depends on ^ticket(84)'s fix landing.
3. Push ^ticket(85)'s test fix and the parser's count fix to #2635 (and #2597), on Matthew's word.
4. ^ticket(18): decide the protocol (the ids of the livelits on screen in the request, a content-keyed cache) before more code.
5. Done (above): ^ticket(58)'s holder is #2597's undo stack.
6. Push bc66ae5616 (the undo compaction), with or without 3041fd0483 (the newest entries kept whole), to #2597, on Matthew's word; and decide whether dev should get Andrew's fix before #2469 merges, since dev keeps whole page models per edit too.
7. Bring ^report(14) up to date with the answer: drafted, a dated update with its data (local commits 0ddcef7, 6aabc40 and 8ba0119 in the series' working copy), waiting for review.