Node repair — diagnosing and fixing a broken installation

The operational runbook for the one failure that looks like a hosting problem and never is:

action 'X' is not a member of class Y. Written for the next engineer, the next agent, and the

support conversation at 11pm. Status: everything here is built and field-proven — the diagnosis

reproduced byte-identical on an isolated node, the engine repair confirmed on a real remote Mac we

never touched, the DSL repair and the audit exercised end to end. The pitfall register in

[../AGENTS.md](../AGENTS.md) carries the compressed version; this is the long form with the commands.

The one idea

A node's class cache is a mosaic, not a snapshot. Every class is hydrated lazily the first time

something touches it, and nothing ever re-questions a class directory that already exists. So one

installation can be simultaneously *current* and *months stale*, in different classes, and nothing in

the system notices — until a new class calls into an old one and the request dies on the seam.

The seam is always the same sentence:

That is never a bad publish and never a network fault. It is one node's local copy disagreeing with

itself. Check the published index before suspecting the store — <base>/zero.9.stores/_versions,

<class>/_manifest, <class>/_files — it will list the "missing" action every time.

Why it happens — four mechanisms, none of them a bug on its own

Mechanism 3 is the seam generator, and it explains why both observed failures had _core as the new

half: _core/boot → server/_entry, _core/_gate_deny → home/_has_pin.

A fifth mechanism decides whether a cold-hydrated class can even be *audited*: ensure_local writes

no _version marker. A class pulled on demand is therefore of unknown freshness until something

pins it. That population is what the audit below calls store_unpinned.

Why zero web.serve cannot fix itself

There *is* a live self-heal, and it is genuinely good — it just cannot reach this failure.

The engine can heal a class it is already serving. It cannot heal a class it needs in order to start

serving. A 500 on a live node usually heals itself within a TTL; a class that breaks boot kills the

process before its own repair mechanism exists.

Step 0 — the layer that *does* reach it: ExecStartPre

Added 2026-09-21, after it cost thirteen domains. Everything above is the engine trying to heal

itself from inside, and the table is a list of reasons that cannot work for a boot-critical class.

The layer that can is the one *outside* the process — the supervisor that starts it.

zero-hydrate.sh was never missing. It has shipped in deploy/cloud-init.yaml since the first

provisioned box: it walks the boot-critical classes, compares each against its published _manifest,

refetches anything short, retries three times, and writes _version last so an interrupted run

leaves a class visibly unversioned rather than silently wrong. It depends on nothing but curl and a

manifest, *precisely* so it works when the store is too short to trust.

It ran once, in runcmd, at birth. Nothing ever ran it again. So a class that went partial on day

two was permanent: the engine died before it could serve, systemd did exactly what Restart=always

promises and put it back into the identical broken store three seconds later, and the repair sat on

the same disk, unread, for as long as anyone left it there.

Three details in that unit line carry the whole design:

• boot, not the full sweep. The full sweep verifies every class on disk — a few hundred small

GETs on a mature box. Paying that inside a three-second restart loop would hammer the CDN and turn

one crash into a stampede. boot walks only the classes without which the engine cannot reach the

point of serving, which is exactly the population that can break a boot. Bare sh zero-hydrate.sh

keeps the full sweep, so runcmd and every hand-run are unchanged.

• The leading -. Without it a failed ExecStartPre aborts the start, so an unreachable CDN

would turn a transient blip into a hard outage — strictly worse than the fault being repaired.

Best effort: repair what it can reach, start either way.

• Not a Zero verb, and that is not a lapse. Every other repair in this system is a zero

action and should be. This one cannot be: stores.pullall and node.facts are DSL, and loading

DSL is exactly what is broken. zero update is a binary subcommand and would work, but it also

swaps the binary underneath its own caller, which is not a thing to do on every restart. The one

repair Zero cannot express as an action is the one that has to run when no action can run.

On a box provisioned before this date the line is not there. One rung installs it, and restarting

under it performs the repair:

It fetches the current zero-hydrate.sh from the store, refuses to arm a script that does not parse,

writes a zero.service.d/10-selfheal.conf drop-in (skipped on a box already born with the line),

daemon-reloads and restarts. Idempotent. Run it once per box; newly provisioned boxes have it

already.

Step 1 — diagnose: zero node.facts

One command, one GET. It folds the store's published _versions index (a single file, one

<class> <sha> line per published class) against the local _version markers.

Two deliberate refusals, because a wrong clean bill is worse than none:

• Store unreachable → every count reads n/a, never 0.

• Publisher / dev tree (.git in the store, or SPACES_KEY set — the same two guards the engine's

own store self-heal trusts) → store_mode=checkout, not judged. There the local files *are* the

published truth, while _version is a consumer artifact written only by thing._pull_one,

zremote::revalidate_class and zero update, and by no publish step at all. Measured on the

publisher's own box: 183 of 196 markers "differ" on a tree the release audit calls 0 drifted. The

content-hash audit for a publisher is creator.release (verify-release.mjs --post).

store_installed=0 or a node.facts with no store_* lines at all means that node's node/ class

predates the audit — repair first (step 2), then re-run.

Step 2 — repair: the ladder, cheapest first

zero update is the one-command answer on a current engine. Since v0.0.98 healStoreClasses()

runs on both update paths, so re-running it is a real repair instead of an early return. Before

that it sat at the tail of the binary upgrade and any node already on the newest binary got nothing.

zero stores.pullall is the better repair and needs no new binary:

• integrity-checked — thing._pull_one recomputes the publisher's content hash and commits

_version only on a match. A partial, corrupt or tampered pull leaves the marker stale, is logged

FAILED-integrity, and is retried next time — never served as good.

• convergent — _prune_stale deletes local files the fresh manifest no longer lists. Without it a

renamed or removed source file makes the hash *never* match; that stranded nodes for ~1.5 months on

one class rename.

• idempotent — a second run answers Everything is already up to date.

• it pins every class it touches, so after one run node.facts gives an exact verdict rather than

a large store_unpinned count.

• it survives the corpse — stores inherits thing, not server, so a dead server/ class never

blocks it. This is luck of the class tree, not design: a diagnostic or repair whose own chain is the

broken one dies with the node.

Browser equivalents on that node, owner-only: /stores/checkupdates (read-only, shows what drifted)

and /stores/pullall.

Targeted repair — a class that is *missing* rather than stale is left alone by both heals, and a

single bad class does not justify a full resync. Deleting one class directory forces the cold path,

which is atomic (tmp → rename) and aborts rather than leaving a partial:

Last resort, deterministic, no guards and no partial-success mode:

Never rm -rf ~/.zero. That is the node's ZeroID, keys, capabilities and device certs. The store

is *code*; ~/.zero and ~/zero.9.data are the machine's identity and its data, and nothing in this

runbook touches them.

Step 3 — verify

Then exercise the page that failed. Two traps when reading the result:

• A restart is required for brand-new actions. The engine parses actions once per process; changed

pages and data go live immediately, a newly added .action does not.

• A fragment never reaches the server. …?app=1#/dash arrives as ?app=1; the #/dash half is a

client-side route. And a signed-in-only rich view renders its public fallback for an anonymous

caller, which looks identical to "the fix didn't work" and is not.

Doing it remotely

Scope is the fleet and only the fleet, locked three independent ways: mesh._fleet_has refuses any

name that is not a kind= device in ~/.zero/registry; every op seals as *you* (root-signed

own-fleet), which a contact cannot forge; and mesh.manage refuses __remote, so nobody pivots an op

through your node. This is the same verb a person gets as buttons on mesh.device — one capability,

two users — not an operator backdoor.

So the reach question is only ever *whose fleet is that machine in*:

And one honest gap: a node that cannot boot is unreachable by every tool above, because they all

dispatch through boot. Its organ never comes up either, so the mesh cannot help. That half can only

live in the engine binary, which ships on its own track — until then it is a human at that keyboard

running zero update.

Field record

2026-07 → 2026-08. An owner's remote Mac: /healthclaims 500ing with _has_pin not in thing

(Jul 18, while serving — that one self-healed), then every zero web.serve dying with _entry not in

web (Aug 1, at boot — that one could not). Its server/ had been cached before server/_entry.action

existed; _core was current because _core always is. Reproduced byte-identical on an isolated node by

deleting one file from a cached class, healed, and re-served HTTP 200. Two zero update retries had

returned at is up to date without looking at the store, and the one upgrade that could have repaired

it was ^Z-suspended one line above the reconcile. Fixed at the source in v0.0.98; the owner then

ran a single zero update on that machine, it refreshed every class that had never been refreshed, and

the install came back. The audit in node.facts and this runbook exist so the next one takes one

command instead of two weeks.

Related

• [../AGENTS.md](../AGENTS.md) — the pitfall register (compressed), plus the store-is-code rule.

• [versioned-runtime.md](versioned-runtime.md) — how the substrate versions itself.

• [remote-access.md](remote-access.md) — the tunnel and login architecture behind mesh.

• [../PUBLISHING.md](../PUBLISHING.md) — the publish side: what a release actually verifies.

• /node/manual, /stores/manual — the live, self-documenting manuals for the two classes involved.