Building a system that notices its own failures.
AXONOS is the stack I run on my own desktop: the services behind my apps, local language models, backups, and several AI coding agents working in parallel. This is one week of it, written up as it happened: four failures, how each one was noticed, what was fixed, and what is still open.
- Period
- 8–15 October 2026
- Scale
- 60 user services, 38 timers
- Role
- Solo, working with AI agents
01 · Context
Why bother
On 11 October I wrote down a hard truth about the stack: it had sensors, a watchdog and backups, but almost every problem was still found by me or by an AI agent by accident. The watchdog ran every five minutes and still missed a disk at 94% and a launcher opening the wrong address, because it only checked whether services were alive.
So I changed the rule. Checks are written from failures that actually happened, not from theory. Anything reversible and well understood the system may fix by itself; anything touching data, money, DNS or a public product it proposes with evidence and waits for me.
The four incidents below are what that looked like in practice during one week. Three are closed. One is not, and I say so.
02 · Four incidents
What broke, and who noticed
Thermidor · systemd service
A service pointing at a binary that was no longer there
Fixed before it failed
- What happened
- The service that runs Thermidor, my calorie diary app, started it through pnpm at a fixed path. pnpm no longer lived at that path. The app was up, so nothing looked wrong, but the next restart would have failed.
- How it was noticed
- The same missing path had already taken another service down: a review of the machine on 13 October traced a systemd 203/EXEC failure (the executable could not be run) to it. On 15 October the Thermidor unit turned out to point at the same path.
- Fix
- The unit now points at the path pnpm actually uses. The watchdog was changed the same day: when a service’s executable is missing it no longer restarts it blindly, it reports the missing file and backs off for 15 minutes.
Unit file diff · watchdog commit with 19 tests
Knowledge sync · nightly job with a local model
One malformed model answer failed the whole night
Partly fixed · still open
- What happened
- Every night a job turns my notes into a knowledge graph, using a language model running on my own GPU. When the model returned a section as plain text instead of a structured object, the code crashed on it and the whole run exited with an error. It ran fine from 8 to 11 October and failed four nights in a row, 12 to 15 October.
- How it was noticed
- Not by the system. The service sat in a failed state until the machine review on 13 October listed it. That is the honest part of this one.
- Fix
- Every section from the model is now checked, and only well-formed objects are kept. The run after the fix saved its first batch with no bad sections (87 nodes, 52 edges). The second batch then hit a request timeout on the local model and the run still exited with an error, though the existing graph was kept intact. Turning a timeout into a skipped section instead of a failed night is the next step.
Service journal 8–15 Oct · post-fix run log
Oxw Nou · checkout and order emails
Cross-review between different models caught duplicates before launch
Fixed · orders still closed
- What happened
- Oxw Nou is my family’s candle shop. Its order and payment flow was written by one AI model and reviewed by a different one, with repro scripts instead of opinions. The reviews found several ways a customer could end up with two payment sessions, two orders or two emails.
- Findings
- A retry five minutes later created a second Stripe Checkout session for the same order. Two concurrent email flushes sent the same queued email twice. And the fix for that one introduced the next: an email claim expired after 2 minutes while the mail library’s socket timeout was 10, so a slow send could be reclaimed and sent again. The reviewer reproduced it with a deliberately slow fake mailer.
- Fix
- A stable idempotency key per order, amount and attempt, and the open Checkout session is reused (2 sessions → 1). An email is claimed before it is sent, SMTP timeouts are 30 seconds, and a send is marked done only if its claim still matches. Identical order requests with the same request id create one order instead of two. A second charge is flagged for a manual refund. Delivery is still at-least-once; email has no exactly-once, and the code documents it.
Two review verdicts with repros · 5 commits
Backups · restic, local and off-site
Proving the backups restore, not just that they run
Running weekly
- What happened
- Backups ran every night and reported success. Nothing checked, on a schedule, that they could actually be restored.
- Fix
- A restore drill takes the newest daily snapshot, restores it into a temporary folder, and checks it against the live data: SQLite integrity checks and table lists, JSON that parses, file sizes and bytes. Then it deletes the folder. The first real run restored 87 files (4.1 MB) in 6.7 seconds locally; the off-site copy took 23 seconds. 18 tests run it against a fake restic with a corrupt database, a bad JSON line and a missing file.
- What it found
- The live shop database had three tables newer than that night’s snapshot: the email queue, the payment ledger and the Stripe event log, all from the work above. They would only be in the next backup. The health check now reports backups as healthy only if a drill has passed in the last 8 days.
2 commits · drill report · weekly timer
03 · The sensors were wrong too
Two false alarms
“The GPU keeps hanging”
A report on 14 October said the graphics stack kept wedging. The evidence pack the next day counted exactly two wedge events, both on 8 October, both recovered by the driver. The “repeats” in the log were my own watchdog’s check, logged every 25 seconds.
“The firewall does not answer”
A query to the firewall timed out and looked like a hung service. It was the authorization layer (polkit) refusing a non-root user; asked as root, the firewall was running.
04 · What I learned
Most failures were drift, not crashes
- Nothing here was dramatic. A path that was right when the service was written and wrong later. A backup that was correct for yesterday’s schema. Checks have to look for things that stop being true, not only for things that stop running.
- Model output is input. A local model that answers correctly a thousand times will one night return the wrong shape, and one bad item must not take the whole batch with it.
- A different reviewer finds different bugs, and the second review is worth as much as the first: the fix for one duplicate created the next one.
- Verify the sensor before acting on it. Two of the alarms this week were about the monitoring, not about the machine.
05 · What I would do differently
Next time, from the start
- Write the restore drill before trusting the backups, not after.
- Treat every external answer, model or API, as untrusted from the first line: validate it at the boundary and define what a partial failure means before the first nightly run.
- Check that every executable a service unit points to exists, the same way the system already checks launchers and autostart entries.
- Ask for the cross-review before the code feels finished. The first duplicate was found by a completion check, which is cheaper than finding it in production.