Crash log

Matthew’s work laptop (a Framework Laptop 13, 13th Gen Intel Core, BIOS 03.04, 20 CPUs). Started 2026-10-16 after the day’s third crash, from the system journal back to 2026-08-18. Kept by Claude with Matthew; a new crash gets a row at the top. Updated 2026-10-19 11:55 MDT.

What the evidence says

A reading that fits all of it, not proven: a laptop with a battery rides through a burst of power above what its adapter gives, and through a dip in the adapter, cable or USB-C power negotiation. This one has nothing to draw on, so either one stops it, at idle or under load. What would test it: a battery it can see (or, if one is fitted, its connector reseated), or a stronger USB-C adapter and cable; then see whether the crashes stop. Until then the rule stands: one heavy job at a time, and commit and push early.

In use since 2026-10-16 17:05 (Matthew’s call, after the 16:47 freeze): headless Chrome and the tools that drive it are pinned to 4 of the 20 threads (taskset -c 0-3 nice -n 10), to cap a burst of power. By 18:10 it had run the fps gate five times, 🧪, the still-picture, door, collision and pit checks, and the day’s 586-picture archive, about an hour of browser work, with no crash; the gate read the same as unpinned (10 of 10 at 59.3 to 60). Not proof that it helps: the 16:46 trial and the 16:40 gate ran unpinned without a crash too. It did not prevent the 20:47 freeze, about 90 seconds into a pinned headless-Firefox run (the same 4 threads, nice -n 10); the run before it read the CPU package at 58 to 72 °C. Nor the 20:53 one, about 45 seconds into the same run started again. All four crashes of the evening came while a headless browser ran.

From 2026-10-16 20:56, Matthew’s ask: no headless browser (Chromium or Firefox) runs on this laptop, pinned or not, and one heavy job at a time across all sessions.

Worth noting next time

Crashes (48 since 2026-08-18)

Newest first. Last line is the journal’s last entry (MDT). Up is how long that boot had run. Load is the 1-minute load average from sar at its last sample before the crash, where one survives (10-minute samples; 20 CPUs).

DateLast lineUpLoadWhat was running, and what was lost
2026-11-0309:03:42 MSTabout 2 h0.86 at 09:00Matthew: “computer crashed. you were doing too much before the crash” and “too much locally.” Under load, and stacked: one Claude session was working through eight of his yeses at once, across three worktrees. At 09:02:41 it ran a dune build of the test runner in ~/hazel-dev-fontscroll (niced, 2 jobs; done 09:03:22). At 09:03:40 it started ten alcotest groups under node on that build, in the background, and nine seconds later it cherry-picked in ~/hazel-663-deck. Full tests for #2635 were running on GitHub at the same time, not here. The transcript’s last whole line is 09:03:51, then NULs; the next boot began at 09:05:02 with the clock right. So the crash came 40 to 80 seconds into the test run, about 20 seconds after a build: the same shape as 10-19 11:46. Lost: the cherry-pick, which had not been pushed: 2 empty objects, including the new commit, and its file scripts/dev_serve.py at 0 bytes. The empties and the 0-byte file were moved to ~/du-shots/crash-2026-11-03-0903/, and the cherry-pick was redone from 8ee77cecbe (now 243874eecd). Also lost: 3 empty .all-deps files under _build (dune redoes them) and the test run’s output in /tmp. The uncommitted port in Evaluator.re came through whole. The 07:04 end of the boot before was a clean shut-down, not a crash.
2026-10-2307:41:5512 h 23 min0.58 at 07:40Matthew: “computer crashed while I was away.” Near idle: 99% of the 07:40 sample idle. One Claude session was working while Matthew was away for two hours (tickets 566 and 571: the game's three-way script boxes, and Fumola's semantics fixes). Its builds and tests were all on GitHub's computers; on the laptop it had just written one file, a second before the journal's last line (a Python rewrite of 25-scripting.js, 07:41:54), and was otherwise only polling GitHub with gh run watch. The journal's last lines are cron's publish at 07:41:01 and its session closing at 07:41:55. Back at 11:18 when Matthew returned, the clock right. Lost: nothing. The file written a second before the end read back whole (no NULs, every edit there) and was committed. 9 empty git objects in hazel-work-notes, cron's 07:41 dashboard publish (already pushed): moved to ~/du-shots/crash-2026-10-23-0741/ and fetched again with git fetch --refetch origin; git fsck clean. No other empty objects, no NULs in the reflogs, no other zero-length files written from 07:35 to 07:43.
2026-10-2219:16:162 h 28 min1.96 at 19:10Matthew: “crash. resume please.” Under load: a Rust build and test run of Fumola on this laptop. One Claude session was working on Fumola (ticket 549, a deep answer that crashed its WebAssembly instance). It pushed its fix to the deep-answers branch at 19:12. Fumola’s CI runs only on main and on pull requests, so at 19:13:37 it started CI’s test steps here in the background (nice -n 10 cargo test -j 4 --workspace in ~/fumola-549, a debug build of the whole workspace), and went idle at 19:14 waiting on it. The parser generator ran from 19:14:58 and compiling began at 19:16:30. The journal’s last line is rtkit’s, at 19:16:16, but the build kept writing files until 19:16:42, 26 seconds later: as on 10-20, the machine was still running after the journal’s last line. Back up at 19:18:20 with the clock right. Lost: build outputs only. 950 of the 1,710 files the build wrote from 19:16:30 on were 0 bytes. They were deleted, with rustc’s incremental cache. The branch’s two commits were already pushed. No empty git objects, no NULs in the reflogs, and no other zero-length files written from 19:10 to 19:18.
2026-10-2206:20:0238 h 53 min4.31 at 06:20Matthew: “woke up just now to frozen computer.” Overnight, while he slept. One Claude session had worked on Dress Up (ticket 531, game rules in Hazel, and the reports) and went idle at 03:19, leaving only a check of three pull requests’ comments on GitHub every 5 minutes. Its last local work was a helper’s lever fix, 01:10 to 01:14. No build, test run or headless browser ran here overnight; the game’s checks ran on GitHub’s computers. But the laptop was not idle: from about 00:55 to the crash the load sat at 3.2 to 5.6 (about 4 of the 20 threads busy: 20% user CPU, 78% idle), against 1.0 to 1.6 earlier that night. It rose minutes after Firefox opened two links (00:49:44 and 00:54:47), the reports pages Matthew sent in his last messages at 00:52 and 00:55, and it held for 5 h 25 min. No Claude job lines up with it, so a page left open in a browser is the likely source (not proven). The journal’s last lines are cron’s, at 06:20:01 and 06:20:02 (publish.py and sysstat). The clock kept time. Nothing lost: no empty git objects, no zero-length files written from 06:15 on. Back up at 06:49.
2026-10-2015:03:4917 h 5 min2.82 at 15:00Matthew: “computer crash -- you know what to do!” At idle for our work: the 15:00 sample was 89% idle. The one Claude session of the day had worked on Hazel (ticket 411: the hash and parser checks for #2661, #2673 and #2610) and went idle at 14:17, after pushing them. Their tests were running on GitHub’s computers, not here. No build, test run or headless browser had run here since #2610’s split sweep ended at 14:16. The journal’s last line is Slack’s, at 15:03:49 (the last cron ran at 15:03:01). A new clue: Matthew’s Firefox, up since 07:12, crashed at 15:04:08, 19 seconds after the journal’s last line. Its main thread crashed on a null pointer inside PipeWire, the desktop’s audio and video service. Its 2.3 MB crash report was written to disk and survived, so the machine was still running and still writing to disk for 19 seconds after the journal’s last line. The next boot read the clock at 15:04:34, so the machine stopped within about 25 seconds of the Firefox crash. That timing fits a freeze, or a device dropping off before a power cut. It does not fit an instant power cut at 15:03:49. Not proven. The clock was right at boot (15:04:38). Then a clean power-off from the desktop’s menu at 15:25:39 (not a crash: the journal ends with systemd-shutdown), back at 15:26:17 with the clock right. Lost: nothing. No repository had empty objects or NUL-padded reflogs. The only empty files written in the window were app journals (Slack, Signal), which are normal. The three repositories written that afternoon (hazel-perf-report, hazel-work-notes, matthewhammer.github.io) read back whole.
2026-10-1921:57:377 h 15 min1.40 at 21:50Matthew: “computer froze. please resume what we were doing.” The fourth crash of the day, 7 h 15 min after the 14:42 one. Under load: a headless Chromium on this laptop, against the rule of 2026-10-16 20:56 (no headless browser here, pinned or not). One Claude session was working on Dress Up while Matthew was away and in and out (ticket 439, people and templates, after tickets 438 and 457, the horse), and through the evening it had run the game’s checks and picture runs in a snap Chromium (--headless=new --remote-debugging-port=9555, not pinned, not niced), one at a time. At the crash it was running the checks before a release, one after another: people439, horse457 and saveload had finished (saveload wrote its saved game at 21:56:50) and surprise225 was starting. Cron’s publish.py started at 21:57:01; the journal’s last line is its session closing at 21:57:37. Damage: 6 empty git objects in hazel-work-notes, including main’s tip (already pushed), so the publish after the crash failed (“bad object HEAD”): set aside to ~/du-shots/crash-2026-10-19-2157/ and fetched again with git fetch --refetch origin; the data that publish had staged was committed by hand (1acd9afc). The game’s unreleased work (people, ticket 439; the centered head, ticket 458) was whole. Every other repo clean, no NULs in the reflogs touched that evening. Matthew, after: “recall that we try to run heavy loads on GH, and not locally. Not sure what happened to cause the computer crash, but maybe it was load related?” Since then: checks and picture runs go to GitHub’s runners only (the tag’s speed run); locally, only the script and load checks.
2026-10-1914:42:012 h 53 min0.88 at 14:40Matthew: “computer crash --- yet again”. The third crash of the day, 2 h 56 min after the 11:46 one. Under load: a dune build of the test runner, about 20 seconds in. One Claude session was working on Hazel (ticket 418, the term hash of #2673) in hazel-reuse-hash, while Matthew was away. At 14:41:30 it built and ran one alcotest group (ValueHash, done 14:41:44); at 14:41:52 it edited ValueHash.re and started dune build test/haz3ltest.bc.js (not under nice), which wrote its last files at 14:42:14. The desktop app’s browser pane was last used at 14:09, and the tests of #2610 and #2635 were running on GitHub’s computers. The journal’s last line is the 14:42:01 cron, the dashboard’s publish.py. Back up at 14:42:38, with the hardware clock right. Lost: only build outputs. All 2,027 files that build wrote were left at 0 bytes; dune had not yet saved its database for them, so they were deleted for the next build to redo. The earlier build’s outputs, the 90 MB test runner among them, read back whole, and so did the session’s two edited source files. No repository had empty objects, NUL-padded reflogs or zero-length files, the dashboard’s cron published again on its own at 14:43, and the session’s transcript ends cleanly.
2026-10-1911:46:011 h 19 min2.83 at 11:40Matthew: “computer crash, again -- a lot today”. Under load: a test run, right after a build. One Claude session was working on Hazel (ticket 405, making #2635’s CI green), in hazel-livelits-modular. At 11:44:54 it started one job at a time: a dune build of the app and the test runner (finished 11:45:30, after js_of_ocaml linked main.bc.js, cli.bc.js and haz3ltest.bc.js), then at 11:45:34 one alcotest group under node (DefStaticsParity, in its third case, the slide corpus). The journal’s last line is the 11:46:01 cron. Back up at 11:48:47, and the hardware clock read 2023-01-01 at boot, the first crash since 10-15 15:33 to lose it (the 10:26 crash that morning kept it). Lost: nothing. The build’s outputs read back whole (no NULs in its text outputs), and the test run’s own output file was left at 0 bytes. No repository had empty objects, empty refs or NUL-padded reflogs, and the session’s transcript ends cleanly.
2026-10-1910:26:0121 h 9 min2.10 at 10:20Matthew: “computer crash”. Under load: a dune build was linking. One Claude session was working on Hazel (ticket 394, the cursor inspector’s full view), with the desktop app’s own browser pane (not a headless browser) open on two local Hazel builds, localhost:8361 (#2698) and localhost:8341 (#2635). At 10:26:18 it started a chain of one job at a time: a commit and a merge in hazel-dev-inspector, a dune build there (done 10:26:33), a cherry-pick onto #2635’s worktree (10:26:33), then a dune build of #2635, whose log was last written at 10:26:43 with js_of_ocaml linking main.bc.js and cli.bc.js. The session’s transcript was last written at 10:26:52 and ends in NULs; the journal’s last line is the 10:26 cron. Back up at 10:27:27, with the hardware clock right. Lost, and redone: the cherry-pick’s 7 objects in ~/hazel/.git were left empty (its commit 9dd265796b, the tip of #2635’s local branch, with its trees and blob; not pushed, so not on GitHub), and #2635’s working copy of CursorInspector.re was left at 0 bytes. The commit it copied, 31ad35d388, was whole, so the branch was set back to its parent and the cherry-pick redone (b44bb9c50f). The empty objects went to ~/du-shots/crash-2026-10-19/. Nothing else: the other 192 objects written that day read back, and no other repository had empty objects, NUL-padded reflogs or zero-length files.
2026-10-1813:16:012 h 34 min— (sar unreadable)Matthew: “computer crash.” The third crash of the day, 2 h 36 min after the 10:40 one. No load figure: today’s sar file stops at the half-written sample from the 10:40 crash, and sar cannot read past it, so no sample after 10:30 survives. One Claude session was working on Dress Up (ticket 306, the tent’s flashlight beam). Until 13:16:26 it drove the desktop app’s own browser pane (not a headless browser) on the game at localhost:8133, with zoomed screenshots and small scripts. At 13:16:28 it released Dress Up 0.0.219: the script check and the load check (node, about a second), a commit at 13:16:29, then fetch, rebase and push of main and its tag (GitHub queued the tag’s speed run at 13:16:35). After that it ran its light wait-for-live loop. Its transcript’s last line is at 13:16:32. Cron’s 13:16 publish.py committed and pushed a dashboard snapshot to hazel-work-notes at 13:16:38, so the machine was still running then; the journal’s last line is the 13:16:01 cron. Back up at 13:17:21, with the hardware clock right. Nothing lost, but again (as on 10-17) pushed commits left empty local git objects: 8 in kids-choice-games (0.0.219’s commit, written 13:16:29) and 11 in hazel-work-notes (the 13:16 snapshot commit, 13:16:38). There was also a zero-length tag ref, refs/tags/dress-up-0.0.219, NULs in four reflogs, and a zero-length copy of the 13:16 snapshot in the working tree. Every lost object was already on GitHub. The empty objects went to ~/du-shots/crash-2026-10-18-1316/, the objects came back by bare clone, and git fsck is clean on both. Found during this check but not caused by the crash: 0.0.219 went out with an empty version.json, because its release script opened the file for writing before reading it.
2026-10-1810:40:012 h 28 min1.59 at 10:30Matthew: “computer crash”. The second crash of the day, 2 h 29 min after the 08:11 one. Lightly loaded (90% idle in sar’s 10:30 sample; the 10:40 sample was being written when the machine died and did not survive). One Claude session was working on Dress Up for M (ticket 282, the grass hut cluster). Until 10:38:57 it drove the desktop app’s own browser pane (not a headless browser) on the game at localhost:8133, and the game’s tab stayed open there. At 10:39:19 it ran the load check (node, about a second), and Matthew took a screenshot and opened it in the image viewer. At 10:39:25 to 10:39:28 it released Dress Up 0.0.207 and pushed main and its tag (cfb1258), then waited for the release to go live with a light loop (one HTTPS request every 10 seconds); its transcript’s last line is at 10:39:32. The game’s speed run for 0.0.206 and the tests of branch bgfs-279 were running on GitHub’s computers, not the laptop. The journal’s last line is the 10:40 cron. Back up at 10:41:03, with the hardware clock right. Nothing lost: 0.0.207 is live, every git object pushed in the last minute is whole (no empty objects in any of the 113 repositories), no zero-length files, and the transcript ends without NULs (unlike 08:11).
2026-10-1808:10:0116 h 27 min0.16 at 08:10Nearly idle (99% idle in sar’s last sample). Matthew had just asked (08:10:21) for the picture archive to be redesigned. The session’s only command was a 2-second read: find and du over the archive (1.1 GB of loose git objects) and git count-objects, finished at 08:10:52. The session was waiting on the model when the machine died: its transcript was still written at 08:11:09 and ends in 300 NULs. The journal’s last line is the 08:10 cron. Back up at 08:11:33, with the hardware clock right. Nothing lost: no empty git objects in any repository, and the archive repositories clean.
2026-10-1715:42:162 h 57 min3.00 at 15:40Matthew: “computer crashed and rebooted”. Nearly idle (87% idle in sar’s 15:40 sample): no build, and no browser on the laptop (the journal shows none from 15:20 on). A session had just released Dress Up 0.0.159 (15:41) and pushed two check branches, checks-159 and checks-225, whose runs went on GitHub’s computers. An image viewer opened at 15:42:13. The dashboard’s cron had committed and pushed its 15:42 snapshot. The game’s files were intact. A new kind of damage: git objects written in the last minute were left empty on disk, though already pushed. There were 11 in hazel-work-notes, among them the tip of main, so the next cron commit failed; its reflogs were padded with NULs and the 15:42 snapshot was left at 0 bytes. There were 6 in the homepage’s repository, among them the dashboard’s 15:42 commit. Both were restored from GitHub at 15:48. The hardware clock kept the right time across this crash (it had read 2023-01-01 after the clean power-off at 12:41 that started this boot).
2026-10-1620:53:014 min—, no sample survivedMatthew: “froze again”. The same pinned headless-Firefox run for ticket 167, started again at 20:52:13 by the session that resumed after the 20:47 freeze (taskset -c 0-3 nice -n 10, geckodriver up at 20:52:17). Its log, written line by line to disk this time, kept one line at 20:52:40: the CPU package at 63 °C, 3.3 to 3.7 GHz, the game at 59.7 frames a second. The journal stops in the 20:53 cron run, about 45 seconds after the browser started. This boot began 4 minutes after the last crash, with a thermal zone (TZ00) reading 76 °C at boot (71 °C at the next). Nothing lost: every file written in its last minutes is whole (no empty files, no NULs). The clock was right at the next boot.
2026-10-1620:47:013 h 54 min4.10 at 20:40Matthew: “computer froze”. Headless Firefox (geckodriver), launched 20:45:26 for ticket 167: Dress Up’s pond after many changes, against a control. Pinned (taskset -c 0-3 nice -n 10), the first crash since pinning began at 17:05. The run just before it (20:41 to 20:45, also pinned) logged the CPU package at 58 to 72 °C and 1.5 to 3.5 GHz on the pinned threads. The journal stops in that minute’s cron run, as at 08:01. Nothing lost: no empty files, the run’s output was in /tmp and only that is gone. The clock was right at the next boot.
2026-10-1616:47:0112 min—, no sample survivedMatthew: “froze again”, and then “it seems like the audit froze my machine”. Headless Chromium was running the ticket 160 audit (42 cases × 3 runs, begun 16:47:00). A session transcript was still being written at 16:47:08, so the end came after that, during the audit. Nothing lost: Dress Up 0.0.86 had been pushed at 16:42. The same machine had run the audit’s two-place trial (16:46) and the 2.5-minute fps gate (16:40) without trouble; the full audit, 42 places one after another, did not get far.
2026-10-1616:33:175 h 55 min1.20 at 16:30Matthew: “computer froze”. Headless Chromium, launched 16:32:54 to try Dress Up 0.0.86’s 12 speed passes. The edit was on disk, intact; tested and released after. This boot had itself started with the clock reset (below).
2026-10-1608:01:0116 h 27 min2.19 at 08:00Idle, 5 minutes after Dress Up 0.0.61 shipped; the journal stops in that minute’s cron run. Nothing lost.
2026-10-1515:33:0330 h 11 min3.92 at 15:30A minute into ticket 112. The game’s edits survived; the music archive’s vibes.mp3 was left at 0 bytes, though already listed. The next boot read the clock as 2023-01-01.
2026-10-1409:22:0172 h 26 min0.16 at 09:20No record of what was running.
2026-10-1107:26:4043 h 41 min2.34 at 07:20Matthew: “computer got unplugged”. The last line is a geckodriver session ending; both branches were pushed, nothing lost. Back up at 08:55.
2026-10-0411:09:312 h 21 min—No record.
2026-10-0408:46:2790 h 58 min—No record.
2026-09-3013:47:0917 h 07 min—No record.
2026-09-2920:35:013 h 16 min—Only uncommitted ^fumola_wip edits in ~/hazel-remote-splices in flight (saved 20:38 after, intact).
2026-09-2916:05:4424 h 47 min~1.3Nearly idle: one session doing git merges, no build.
2026-09-2815:17:062 h 27 min—No record.
2026-09-2812:48:0147 min—Two Claude sessions at once: one building a scratch worktree, the other a dune build plus tests in another worktree.
2026-09-2811:57:5642 h 17 min—No record.
2026-09-2617:38:215 min~1.4Idle: 5 minutes after logging in, no builds.
2026-09-2617:25:0125 h 56 min~1.4Idle: 93% idle on 20 CPUs, no builds.
2026-09-2515:27:522 h 50 min—No record.
2026-09-2512:36:294 h 58 min—No record.
2026-09-2216:57:1617 min—The same stacked work as at 16:38.
2026-09-2216:38:36123 h 45 min—Stacked work: a dune build, a 20-minute haz3ltest run and repeated headless-Firefox WebDriver sessions, often overlapping. About 15 minutes to re-establish state each time.
2026-09-1712:51:233 h 16 min—No record.
2026-09-1709:33:2145 h 21 min—No record.
2026-09-1400:08:557 h 04 min—No record.
2026-09-1317:03:166 h 13 min—No record.
2026-09-1310:48:452 h 04 min—No record.
2026-09-1308:42:021 h 56 min—No record.
2026-09-1306:43:5521 h 18 min—No record.
2026-09-0421:25:01106 h 18 min—No record.
2026-08-2815:05:314 h 05 min—The last lines are gnome-shell falling behind on the Power Button’s events: likely a hang ended by holding the button.
2026-08-1813:47:477 min—No record.
2026-08-1813:40:125 min—No record.
2026-08-1813:33:285 min—No record.
2026-08-1813:24:1343 min—No record.

Not counted above: 6 boots whose journal stopped while the desktop session was closing, a log-out or shut-down begun and never finished (10-09 11:44, 10-06 07:21, 09-15 12:11, 09-12 09:25, 08-31 11:06, 08-18 16:16), and 6 clean shutdowns (10-17 12:41, 10-16 10:34, 09-25 07:38, 08-28 10:59, 08-25 17:43, 08-22 12:34), and 1 plug pulled by hand (10-22 16:47, Matthew: “accidentally hit power plug --- not a crash -- my fault”; idle, nothing lost).

Adding a crash

After a crash, and before starting anything heavy:

journalctl --list-boots                          # the boot before this one: its LAST ENTRY is the crash
journalctl -b <boot id> -n 30 -o short-iso        # a clean end has systemd-shutdown or "Journal stopped"
journalctl -b 0 -k -g 'RTC time|BAT1'            # the clock at this boot, and the battery
LC_ALL=C sar -q -f /var/log/sysstat/saDD -s HH:MM:00 -e HH:MM:00   # load before it