Multica daemon: disk hygiene (task-dir retention + monitoring)

Applies to: the crew daemon host joshuas-mbp-2 (ssh joshualockhart@192.168.86.30, runtime bf3c31e7-4a7e-452d-98c7-d0e88beda4dd) and its task workspaces root ~/multica_workspaces/<workspace-id>/<task-dir>.

On 2026-08-19 the host's root volume filled to 2.0 GiB free of 932 GiB, blocking every agent from checking out repos. Manual triage freed it back to ~359 GiB. It refilled to 115 MiB free by 2026-08-31 — see Recurrence 2026-08-31 for the four stacked failures that let that happen, including the fact that this very document was never merged. This doc is the durable fix: config for multica's own native GC (primary mechanism), a fail-closed backstop sweeper (secondary, justified below), a launchd disk/GC-health monitor, and the stale-lock repair procedure. All host artifacts here are repo-tracked at scripts/multica-host/.

Root cause

Multica has a native GC subsystem — it is not the case that nothing ever reclaims. It runs roughly every 2 hours and reclaims real disk (8–36 GB per cycle observed). The bug is narrower and easy to miss:

Native GC eligibility gates on the linked ISSUE's status, not on task completion. The eligibility log line reads:

gc: eligible for cleanup component=daemon dir=<8-hex> kind=issue issue=<uuid> status=done updated_at=...

.gc_meta.json in each task dir ({"kind","issue_id","workspace_id","completed_at"}) has no status field of its own — the daemon looks up the issue's status from the server at GC time. A task whose linked issue never reaches done pins its workdir on disk forever, with no TTL backstop, regardless of how long ago the task itself actually finished.

.gc_meta.json is a completion receipt, written by the daemon at task completion — not a provisioning record written when the task dir is created. Its absence means "this task never completed," not "a code path failed to write it." This distinction matters for the reaper's unknown bucket — see unknown bucket composition below.

Proof of the leak: task dirs were found on disk with completed_at ages of 82, 88, and 96 days at triage time — nothing was cleaning them up despite .gc_meta.json's completed_at field existing and being long past any reasonable window. Per-cycle log lines confirm the pattern is systemic, not one-off: cleaned=30..53 per cycle against skipped=450..548, with skipped growing monotonically cycle over cycle (450→456→485→503→545→548 across the observed run).

A secondary, unrelated failure recurs on the same host: gc: git maintenance failed ... error="signal: killed", caused by stale git lock files on the shared repo mirror. See Stale-lock repair below, and the related Multica gc/Maintenance Silent Failure doc for the deeper "every liveness signal stays green" failure mode this belongs to.

Recurrence 2026-08-31 (OBJ-3048 / OBJ-3055)

The host refilled to 115 MiB free of 932 GiB in 12 days — worse than the original incident, and full enough that even /tmp writes failed with No space left on device. That works out to a sustained burn of roughly 30 GiB/day, which is the number every threshold in this document is now sized against.

Four independent failures had to line up. Each one alone would have been caught by the others; the value of writing them down is that they are four different kinds of failure, and only the first was a simple oversight.

1. The fix was never merged. The ops/multica-daemon-disk-hygiene branch sat local-only — 2 ahead / 210 behind origin/master, no PR. Neither this document nor scripts/multica-host/ existed on master. Every cross- reference written during the first incident resolved only on an unmerged branch, which is indistinguishable from a resolved incident to anyone reading master.
2. The sweeper was never installed on the host. Only the monitor (~/.multica/multica-disk-guard.sh) had been copied over. The thing that observes was deployed; the thing that acts was not.
3. The monitor stopped silently and kept reporting OK.disk-guard.out.log ends cleanly at 2026-08-28T08:10:46Z overall=OK free=404GB, .err.log is 0 bytes, and launchctl list still showed it loaded with last exit 0. It simply stopped firing. So the status file reported "overall":"OK","free_gb":"404" for three days while the volume slid to zero. A monitor that stops running reports last-known-good forever — see the monitor section for the two mitigations added, and the honest statement of what they still cannot do.
4. The sweeper's directory guard no longer matched reality. Task dirs are now 12 hex characters; the guard matched 8. It would have skipped 99.2% of the disk even if it had been merged, installed, and scheduled — see the sweeper section.

There is also a fifth thing, which is not a failure so much as a wrong question: the retention gate asked "is this work pushed?" when it should have asked "can deleting this lose anything?". That cost ~170 GiB of wrongly-retained disk on its own, and is unpicked in Fail-closed stance.

Outcome of the 2026-08-31 pass: task dirs 1001 -> 302, free space 108 MiB -> 200 GiB, 178 dirs / 194 GiB reclaimed, zero dirty checkouts deleted. The shared mirror was then verified released: 309 registered worktrees, 0 prunable, 0 stale locks under refs/ or worktrees/, and master checked out by nobody.

Blast radius: multica issue search "No space left on device" returned 16 issues, 15 real — including the App Store review chain and the in-flight v4.47 milestone. Search that string first when triaging: the failure surfaces on every affected issue as a checkout timeout, so the ticket text almost never names disk.

Recurrence 2026-09-02 (OBJ-3186)

Not a disk-exhaustion recurrence — a corruption of the tooling that keeps this document's fix running. install-disk-guard.sh installed both agents with a plain cp onto their live ~/.multica/ path. cp opens the destination O_TRUNC and rewrites the same inode in place; bash reads a running script by byte offset and re-seeks at loop/function boundaries, so overwriting one mid-execution makes the running process resume inside different content and silently execute misaligned fragments.

This fired for real during this incident's own recovery pass: the installer was run while ~/.multica/reap-stale-task-workdirs.sh (pid 82318) was mid-flight under --ttl-days 2 --confirm. The file went 16619 → 19952 bytes underneath the running process. The in-flight run — the backstop sweeper's destructive rm -rf pass over ~1,200 task worktrees — had to be killed and restarted from scratch. Worst-case blast radius by construction: of the two scripts this installer manages, the reaper is the one that deletes things.

Fix (shipped): every installed file now arrives via cp-to-temp + mv in the same directory — a real rename(2), so a running process keeps reading its original, unmodified inode until it exits — plus a pre-flight guard, hoisted above both installs, that refuses to run while either installed target is currently executing. The guard names the offending pid(s) and path(s) and aborts before any file is touched; FORCE_INSTALL=1 proceeds anyway, safe because the replace beneath it is atomic. Detection uses ps -eo pid=,command= | grep -F <path> | grep -v grep, deliberately not pgrep -f — pgrep -f self-matches here, since the target path also appears in pgrep's own command line. Two related defects on the same code path closed in the same pass: plutil -lint and PlistBuddy now run against the temp file, before the mv, so a lint failure leaves the previously-installed plist byte-identical instead of corrupt (previously it validated after the live plist was already overwritten). Full install contract: scripts/multica-host/README.md.

General rule: never cp (or any O_TRUNC write) onto a path that a long-running process might have open for reading — a same-directory temp-file-then-mv is the only replace that doesn't risk corrupting an in-flight reader.

Retention policy (decided)

2 days (TTL_DAYS, OBJ-3162). A task dir is eligible for reclamation TTL_DAYS after its .gc_meta.jsoncompleted_at, regardless of whether its linked issue is still open. This is the whole point of the fix — closing the gap the issue-status gate leaves open. The original 7-day default implied a ~600 GiB steady state on this host's ~150–220 task dirs/day turnover — larger than the disk itself, which made the sweeper reclaim nothing in practice; 2 days fits the host while still leaving room for a human to notice a bad run before its dirs age out. (Don't confuse this with MULTICA_GC_TTL, the native daemon's own 7-day retention knob documented below — a separate, unrelated setting.)

Fail-closed stance: what deletion can actually lose

Corrected 2026-08-31. The original version of this section gated reclamation on whether a branch was pushed. That was the wrong question, and it retained ~170 GiB of genuinely reclaimable disk while the host filled to 115 MiB free. The correct question is narrower: does deleting this directory lose anything that does not exist elsewhere?

Every task checkout is a linked git worktree of one shared bare mirror:

~/multica_workspaces/<ws-uuid>/<task-hex>/workdir/enkidu/.git   <- a FILE, not a dir
    contents: gitdir: ~/multica_workspaces/.repos/<ws-uuid>/github.com+objectuve-softworks+enkidu.git/worktrees/enkiduNNN

Three consequences follow, and they are what the gates are built on:

1. Commits live in the mirror's shared object store, not in the task dir.
2. refs/heads/<branch> lives in the mirror's common ref store, not in the worktree's private git-dir. A linked worktree does not own its branch ref; it borrows the mirror's.
3. Therefore rm -rf <task-hex>/ discards only uncommitted working-tree changes. The branch survives and is re-checkoutable with git worktree add.

See Why --reference clones are not the lever below for why this rules out the obvious-looking --reference-clone fix.

So the real data-loss guard is git status --porcelain, plus a proof that the commits are still pinned by a ref. Retention categories, all fail-closed — anything that cannot be positively verified safe is retained, never deleted:

Retained as	Meaning
in_flight	.task_lock touched within IN_FLIGHT_MINUTES (default 120). Checked first, before the TTL gate, because a re-picked-up task can carry a stale completed_at.
too_young	completed_at is newer than TTL_DAYS.
dirty	git status --porcelain non-empty — real uncommitted work.
unreadable	git itself failed (non-zero exit). See the fail-open bug below.
detached	Detached HEAD: no ref pins those commits, so deletion is not provably lossless.
unreffed	Branch ref absent from the mirror, or present but not equal to the worktree HEAD.
no_mirror	The mirror is not where the script expects. The script aborts outright rather than falling through to deleting.
unknown	No parseable .gc_meta.json completed_at, and a checkout exists. Ages in on its own mtime once past UNKNOWN_BACKSTOP_DAYS (default derived from TTL_DAYS — 2 × TTL_DAYS, capped at 7, currently 4 with TTL_DAYS=2; --unknown-backstop-days N overrides explicitly, uncapped; OBJ-3286 — was a stale hardcoded 14, 2× a TTL_DAYS of 7 that OBJ-3162 had already cut to 2 — logged as BKSTP) — until then, retained the same as any other category. A receipt-less dir with no checkout at all skips this gate entirely and is reclaimable directly on TTL_DAYS + mtime — nothing a deletion could lose — logged as RECLAIM shell (OBJ-3286). An aged-in unknown dir is still subject to every other gate above (in-flight, dirty, unreadable, detached, unreffed, no_mirror) unchanged. A large unknown count is the expected steady state on a high-churn host, not evidence of a broken writer — see below.

unknown bucket composition: not a broken writer

A reader seeing unknown=700+ on this host should not read that as alarming on its own — measured composition (2026-09-02, OBJ-3283, f56c86e0-…a655aa workspace):

	dirs	disk
Total task dirs scanned	1754	—
With .gc_meta.json	1043	—
Without (unknown)	711	5105 MiB
↳ stubs: .task_lock + .task_owner only, no workdir/ at all	707	2.8 MB total (~4 KB each)
↳ fully provisioned, no receipt	8	~5.1 GiB
↳↳ live in-flight (mtime within 15 min)	6	~3.5 GiB
↳↳ genuine orphans (never completed)	2	1.77 GB

So: ~99% of the bucket is empty stubs contributing 2.8 MB, and most of the remaining bulk is live in-flight tasks that simply haven't finished yet — GATE 0's in-flight-lock check (checked first, see the table above) already protects those. The permanent, unreclaimable leak in this snapshot was 2 dirs / 1.77 GB over ~2 days — real, but roughly 3% of the documented ~30 GiB/day baseline, not the dominant risk. The daemon owns the .gc_meta.json writer and it is not in this repo, so the reaper must stay fail-safe about missing receipts rather than assume a fix can make provisioning write metadata earlier.

The defect this measurement found (OBJ-3286, fixed): the 707 empty-shell stubs above were provably zero-risk (no checkout at all — see the fail-closed stance), but the empty-shell reclaim check used to sit after the unknown backstop's continue, so none of them could reach it — they sat retained for the full UNKNOWN_BACKSTOP_DAYS window regardless. OBJ-3286 also recalibrated UNKNOWN_BACKSTOP_DAYS (was a stale 14, derived from a TTL_DAYS of 7 that OBJ-3162 already cut to 2) and added an oldest-unknown-age figure to the summary line, since unknown_aged_in=0 on its own can't distinguish "nothing qualified" from "this code path is dead." Empty shells now reclaim on age alone (logged RECLAIM shell), and the unknown count drops to roughly the fully-provisioned-with-no-receipt slice.

Lock-only stubs traced to daemon wakeup, not a write gap (OBJ-3285)

A follow-up investigation traced individual lock-only stubs through the daemon's own logs to answer the question the bucket-composition measurement above left open: why do these dirs never get a .gc_meta.json?

Re-measured 2026-09-03, same host: 1,910 total task dirs, 720 without .gc_meta.json — 716 of those are lock-only stubs (.task_lock, 0 bytes, + .task_owner, 36 bytes — the task UUID — and nothing else: no workdir/, no reply.md), ~40% of all task dirs, ~4 KiB each, 2.8 MiB total for the entire population. The remaining 4 no-meta dirs all had a workdir/ and were live in-flight runs at measurement time (oldest started that same day), not orphans.

Lock-only task dirs are expected daemon behavior, not a leak. Four lock-only dirs were traced individually by task UUID through the daemon logs; the only log line for each was task wakeup received — no claim, no start, no completion. The daemon creates .task_lock + .task_owner at wakeup as a local claim guard, and the task is often never claimed or started after that. This is expected behavior of the closed-source Homebrew multica daemon binary, not a defect in the reaper or anything this repo controls.

No unintended write gap. .gc_meta.json is a completion receipt, written by the daemon only at task completion (see Root cause above) — every in-flight or never-started task is a no-meta dir by definition, so "no receipt" is not evidence of a leak. The daemon binary's only skip path for the write is the string execenv: skipping .gc_meta.json write: kind is empty, and it appears zero times across every retained daemon log — there is no unintended write gap, just tasks that woke up and never ran.

Both behaviors are daemon-internal. Neither lock-only-shell creation nor receipt-write timing is something this repo's tooling controls — the binary is closed-source. The empty-shell reclaim path (RECLAIM shell, OBJ-3286) is the only lever the crew holds over the resulting disk footprint, and at 2.8 MiB total the footprint was never the real concern — visibility into the unknown bucket was.

Reading unknown= correctly: it means "age could not be established for this dir," not "this dir is a leak." It's only a meaningful triage signal once lock-only stubs are classified and reclaimed separately from genuine unknowns — exactly what OBJ-3286's empty-shell-first reordering (above) does.

See OBJ-3285 for the full investigation (measurement, per-task-UUID log traces, and the questions it closed) and OBJ-3286 for the reaper-side fix this finding validates.

Why the old "is it pushed?" gate was wrong

A live dry run with an ls-remote origin gate found only 15 dirs / 17 GiB eligible and retained 157 dirs (~170 GiB) as "unpushed/detached". Those branches read as unpushed mostly because they were squash-merged and then deleted from origin — which is indistinguishable from genuinely unpushed work by origin-comparison alone (merge-base --is-ancestor says NO for a squash-merged branch even when its content is fully landed).

The mirror-ref gate replaced it. Against the live host it found refs/heads/<branch> in the mirror equal to the worktree HEAD for 175 of 175 candidates (unreffed=0) — the premise holds population-wide, not just in a sample. Dropping ls-remote also removed ~170 GitHub network round-trips, which were the slow part of the scan.

One honest caveat about that unreffed=0: because a linked worktree's HEAD follows the mirror's own ref, the sha-mismatch arm of that gate is structurally unreachable for a healthy linked worktree. It fires on ref absence, on a standalone (non-worktree) repo, or on ref-store corruption. It is retained because it is fail-closed and costs nothing, not because it is expected to trip.

cmd | wc -l made the dirty gate fail OPEN

A pipeline's exit status is the last command's, so:

dc=$(git status --porcelain 2>/dev/null | wc -l) || dc=1   # WRONG

returns 0 when git itself failed — rendering an unreadable tree as "clean". On a full disk git cannot write index.lock, so the one gate protecting uncommitted work failed open under exactly the condition it was running in. The fix captures git's own status and retains on failure:

porcelain=$(git -C "$checkout" status --porcelain 2>/dev/null); git_rc=$?
[ "$git_rc" -ne 0 ] && { retain unreadable; }

Verified against a fixture worktree with an unresolvable gitdir: pointer: old gate => delete, new gate => retain; clean and dirty cases unchanged. This also makes an ssh-interrupted run safe to re-run, since a half-deleted dir reads as unreadable. The identical pipeline bug existed in the monitor's df read and was fixed the same way.

Stash is mirror-wide, not per-worktree (2026-08-19)

The dirty check originally also gated on git stash list being non-empty. refs/stash is a common ref, so a single historical stash anywhere on the mirror flagged every worktree of that mirror as dirty, permanently (the original dry run showed skip_dirty=6, reclaimable=0 for the whole host; four unrelated task dirs all reported an identical stash_count=7). It was never a real preservation concern either — rm -rf of a task dir never touches the shared mirror where the stash actually lives. Stash count is still read and printed per-dir, but as an advisory note, not a gate.

Why --reference clones are not the lever

Every task checkout is already a linked git worktree of one shared bare mirror (see the three consequences above), not a standalone clone. A --reference clone exists to let N clones share one object store instead of each duplicating it — but there is no per-task object-store duplication here to eliminate: the mirror's object store is already shared, by construction, across every worktree. --reference deduplicates objects; objects are not where the ~1.1 GB per-task disk cost comes from. That cost is the materialized working tree — the checked-out files on disk — which a --reference clone does nothing to shrink, since it would still check out and materialize the identical tree. Retention/reclamation (this doc's subject) is the only lever on per-task disk cost; --reference is not a second one. Recording this so it isn't re-proposed at the next incident.

The real lever on the materialized-working-tree cost itself — sparse checkout or a partial clone, so a task only materializes the files it actually touches — is a separate, unstarted idea, filed to backlog as OBJ-3081.

The fix

1. Native GC config (primary mechanism)

Config lives at scripts/multica-host/gc-env-additions.plist.xml (reference/diff, human-readable, not applied directly) and is applied via scripts/multica-host/apply-daemon-gc-env.sh, which idempotently merges these into the live ~/Library/LaunchAgents/com.longwoodlabs.multica-daemon.plist's existing EnvironmentVariables dict (preserving CLOUDSDK_CORE_PROJECT, GOOGLE_APPLICATION_CREDENTIALS, PATH untouched):

Var	Value	Why
MULTICA_GC_ENABLED	true	Already true by observed behavior; pinned explicitly as config-as-code rather than an implicit binary default.
MULTICA_GC_INTERVAL	2h	Matches observed cadence; pinned so a future multica upgrade can't silently change it.
MULTICA_GC_TTL	168h (7 days)	The policy knob. See the "unverified" callout below.
MULTICA_GC_ORPHAN_TTL	48h	Orphaned dirs (no resolvable issue at all) have zero status-gate backstop; short TTL with a buffer for transient lookup errors.
MULTICA_GC_REPO_TTL	720h (30 days)	Repo mirrors are shared per-repo (not per-task) and expensive to rebuild; longer TTL avoids repeated full re-clones for infrequently-touched repos.
MULTICA_GC_ARTIFACT_TTL	168h (7 days)	Matched to the same policy window as the main TTL for one consistent retention story.

Every *_TTL/*_INTERVAL var uses Go duration syntax (168h, not 7d — Go durations don't support day units directly). These are env-var only: multica daemon start --help exposes 13 flags, none GC-related; the env-var names above were extracted via strings /usr/local/bin/multica.

RESOLVED 2026-09-01 — MULTICA_GC_TTL does not override the status=done gate. This was previously flagged here as unverified, with a "check again after ~2 weeks" caveat. No wait was needed: the daemon log answers it directly. The GC env has been live since 2026-08-19T17:17:48 (all six MULTICA_GC_* keys present in both the plist and the running process), and every cycle since reads:

10:26:14 gc: cycle complete cleaned=23 orphaned=0  skipped=1074 bytes_reclaimed=9965673
12:27:16 gc: cycle complete cleaned=10 orphaned=22 skipped=1085 bytes_reclaimed=2530319
14:27:10 gc: cycle complete cleaned=5  orphaned=30 skipped=1067 bytes_reclaimed=1279782
16:26:08 gc: cycle complete cleaned=5  orphaned=60 skipped=1017 bytes_reclaimed=1628523

Native GC inspects ~1,020–1,085 dirs every cycle and skips essentially all of them, reclaiming 1.3–10 MB against a host burn rate of roughly 30 GiB/day. Dirs far older than the 168h TTL are being skipped, so the TTL is not the binding constraint — the status=done eligibility gate is, exactly as the issue-status finding at the top of this doc describes.

Consequence: reap-stale-task-workdirs.sh is the primary defense, not a backstop. It needs a scheduler; native GC will not close this leak on its own.

The remaining unverified item, flagged during review — MULTICA_GC_REPO_TTL (30 days) reaps the whole shared mirror, not a task dir. One mirror backs every worktree for a repo (89 worktrees observed on this repo's single mirror). If the daemon's "last used" signal for repo-mirror GC is only touched at worktree creation (not on every git operation an in-flight task performs), a single long-running or stalled task on an otherwise-idle repo could have its mirror reaped by native GC out from under it mid-task, corrupting that worktree (commondir would point at a deleted path) — distinct from and more severe than a reclaimed task dir, since it can break a task that's still running, not just one that already finished. Observed .multica_last_used in the mirror root (touched same-day as normal fetch activity) is consistent with per-operation freshness but does not prove it — this is genuinely unverified, same as the MULTICA_GC_TTL/issue-gate question above. Recommend confirming via daemon.log (gc: eligible for cleanup ... kind=repo) correlated against multica agent tasks before trusting 30 days blind on a repo with any long-lived or stalled task.

Applying it — read this before running anything:

• apply-daemon-gc-env.sh only edits the plist file and prints an advisory idle heuristic. It never runs launchctl unload/load/kickstart. A plist edit is inert until the daemon is reloaded, so running the script cannot interrupt an in-flight agent task by itself.
• An agent task may be running on this host at any given time. Before reloading, confirm no task is in flight (multica agent tasks, or the workspace dashboard) — a reload restarts the daemon process and kills whatever it's mid-run on.
• Once confirmed idle, a human runs:

  launchctl unload ~/Library/LaunchAgents/com.longwoodlabs.multica-daemon.plist
  launchctl load ~/Library/LaunchAgents/com.longwoodlabs.multica-daemon.plist

• Rollback: apply-daemon-gc-env.sh writes a timestamped <plist>.bak.<UTC-timestamp> backup before editing. Restore it and repeat the unload/load above.

2. Fail-closed backstop sweeper

scripts/multica-host/reap-stale-task-workdirs.sh — explicitly secondary to the native GC config above, justified only by the direct evidence in Root cause: dirs 82–96 days old surviving on disk despite native GC running normally. This is not a "reinvent GC" decision — it's a narrow backstop for the one gap the native mechanism has demonstrably left open, scoped tightly to reduce its own blast radius:

• Directory matching requires an exact 8- or 12-lowercase-hex-character name directly under a UUID-shaped workspace-id directory, and nothing else. It cannot wander into .repos, .multica, .skill-cache, or the standalone enkidu checkout that also lives at the workspace root.
• Only ever inspects <taskdir>/workdir/enkidu — scoped to this repo, not a general-purpose reaper.
• Dry-run by default. Deletion requires an explicit --confirm; there is no way to delete anything by omission.
• Refuses to run at all if the shared mirror is not where it expects (no_mirror) — the survivability gate is the only thing standing between this script and real data loss, so if that check cannot run, nothing runs.
• Every skip path is fail-closed — see the retention table above.

Mirror worktree-admin prune (behind --confirm, after the deletion loop).rm -rf on a task dir only removes its working tree — it never touches the shared bare mirror, so the mirror's worktrees/<name> admin entry for that checkout is left behind and accumulates forever (confirmed on a live host: git worktree prune dropped 614→415 registered worktrees, ~199 stale, in one pass), slowing every git worktree add/list/gc call against that mirror. So after reclaiming task dirs, strictly behind --confirm and never during a dry run, the script collects each reclaimed dir's shared mirror, dedupes, and runs git worktree prune --expire=1.hour.ago once per distinct mirror — never a bare git worktree prune (that expires everything immediately by default and could drop the admin entry of a worktree another task is mid git worktree add on elsewhere on the same host). A prune failure logs a WARN and does not fail the run; it retries clean on the next scheduled sweep. Known limitation: a task's own admin entry is never prunable in the same run that reclaimed it (GATE 1's git status --porcelain check rewrites the admin dir's mtime, so it always reads as "just touched" in that pass) — it's swept on the next run against the same mirror instead, once the entry clears the 1-hour expiry window.

The 8-hex glob was a silent 99.2% miss (found 2026-08-31). Multica task dirs are now 12 hex chars (0001205e2f96); the on-host histogram is 8 chars -> 8 dirs, 12 chars -> 993. The original script globbed "$ws"/???????? and gated on an 8-hex case, so it would have skipped 99.2% of the disk even if it had been merged, installed, and scheduled. A read-only analyzer inheriting that same guard reported total_task_dirs=8; the identical scan with a 12-hex arm reported 1001. An implausibly small count from a shape-matched scan is the tell. The glob is now "$ws"/* with the length check done by the case alone, so a future length change fails visibly (count drops to zero) rather than silently.

Usage:

# Dry run (default) — reports only, deletes nothing:
scripts/multica-host/reap-stale-task-workdirs.sh

# Zero-risk headroom pass first: empty shells only (task dirs with no
# checkout at all, so there is no working tree to lose):
scripts/multica-host/reap-stale-task-workdirs.sh --shells-only --confirm

# Full pass:
scripts/multica-host/reap-stale-task-workdirs.sh --confirm

# Non-default TTL:
scripts/multica-host/reap-stale-task-workdirs.sh --ttl-days 2 --confirm

Run --shells-only first on a host that is already full. Empty shells have no checkout, so deleting them is risk-free — and it buys back the headroom the real pass needs. Below roughly 100 MiB free, git cannot write index.lock, so the dirty gate cannot even be evaluated; see the monitor's EMERGENCY floor.

A reclaim script cannot log to the host it is reclaiming. At 108 MiB free, the host's fish shell refused the redirect (open: No space left on device) and therefore never ran the command at all — a nohup ... > ~/reclaim.log & handoff silently did nothing, which is indistinguishable from a run that finished instantly. Pipe to a localtee instead. The remote shell on this host is fish, so an ssh payload must be fed as bash:

ssh <host> bash -s -- --confirm < scripts/multica-host/reap-stale-task-workdirs.sh | tee local.log

Real dry-run output against the live host (2026-08-31, ttl_days=2, before the confirm pass):

Summary: scanned=1001 shells=516 reclaimable=175
Retained: in_flight=10 too_young=284 dirty=11 unreadable=0 detached=5 unreffed=0 no_mirror=0 unknown=0
Empty shells: 516 (~4380 MiB)
Task dirs:    175 (~191 GiB)

And the confirm pass outcome: task dirs 1001 -> 302, / free space 108 MiB -> 200 GiB, 178 dirs / 194 GiB reclaimed.

This now runs as a recurring launchd job — see Reaper schedule below for the plist, cadence, and staged dry-run→--confirm promotion process. The script itself is unchanged by that: it still copies to a stable host path first via disk-guard/install-disk-guard.sh rather than running in place from a repo checkout, since a checkout under a multica task workdir is itself GC-eligible and would eventually point a LaunchAgent at a deleted script.

3. launchd disk + GC-health monitor

A separate LaunchAgent, com.longwoodlabs.multica-disk-guard (distinct Label from com.longwoodlabs.multica-daemon — loading/unloading it never touches the daemon or any in-flight task):

• disk-guard/com.longwoodlabs.multica-disk-guard.plist — StartCalendarInterval firing at :00/:15/:30/:45 past the hour (the same 15-min cadence as the old StartInterval: 900, but on launchd's calendar-fire primitive, not its interval-poll one). See Root cause of the 2026-08-28 stop below for why: a StartInterval timer stopped firing silently for 87.7 hours and never recovered on its own, while StartCalendarInterval is the documented mechanism that re-fires for a missed window. KeepAlive is deliberately absent — this is a poll, not a service; a bad exit just waits for the next tick rather than restart-looping.
• disk-guard/check-disk-and-gc-health.sh — the check body.
• disk-guard/install-disk-guard.sh — copies the script to the stable path ~/.multica/multica-disk-guard.sh (not run in place from a repo checkout, for the same reason as the sweeper above), installs the plist, and registers it via explicit-domain launchctl bootout/bootstrap (OBJ-3283 — see Explicit-domain registration below), then verifies it actually ran, not just registered, via launchctl kickstart -k plus a bounded wait for a fresh log line (OBJ-3282 — see Post-load verification below). Safe to run immediately — different Label, no daemon interaction.

Checks, every 15 min:

Check	Thresholds	Rationale
Free disk (df -k /, evaluated in MiB)	WARN < 150 GiB, CRIT < 75 GiB, EMERGENCY < 10 GiB	Sized against the observed refill rate of ~30 GiB/day on this host, so the tiers map to time: WARN ≈ 5 days of runway, CRIT ≈ 2.5 days, EMERGENCY ≈ hours.
GC cycle recency (grep 'gc: cycle complete' ~/.multica/daemon.log)	STALE if last cycle > 6h old	3× the 2h MULTICA_GC_INTERVAL, tolerating one missed cycle before alarming, to avoid flapping on a single slow cycle.
Monitor self-age (mtime of the previous disk-guard-status.json)	RESUMED_AFTER_GAP if > 3× the check interval (CHECK_INTERVAL_SECONDS, 45 min)	The 2026-08-31 incident's third root cause: the agent stopped firing and the status file froze at last-known-good for three days.
Reaper health (launchctl print gui/$UID/com.longwoodlabs.multica-reaper)	NOT_REGISTERED if not loaded at all; NEVER_RAN if registered but runs=0; STALE if its log has gone quiet > REAPER_STALE_HOURS (72h, 3× its once-daily cadence)	Independent of the gc check above — see gc.status covers native in-daemon GC only, not the reaper below. NEVER_RAN is the exact OBJ-3282 incident signature: bootstrap returned 0 and the job registered cleanly, but RunAtLoad never fired it.

The two subsections below (explicit-domain registration, post-load verification) and the reaper field described further down are OBJ-3282, PR #2855 — merged to master 2026-09-03.

Explicit-domain registration (OBJ-3283)

install-disk-guard.sh registers all three LaunchAgents it manages (disk-guard monitor, its liveness watchdog, the reaper) via explicit-domain launchctl bootout gui/$(id -u)/<label> (idempotent — tolerates "not loaded") followed by launchctl bootstrap gui/$(id -u) <plist>, then asserts launchctl print gui/$(id -u)/<label> finds the job there before declaring success. This replaced the legacy launchctl unload/load API, which is the confirmed root cause of OBJ-3258 and OBJ-3276: that API registers a job into the caller's session domain rather than an explicit one, so a job's continued registration depended on the session that happened to run the installer, not on the job itself. A third LaunchAgent (com.longwoodlabs.multica-daemon), installed by a different path, survived both incidents untouched — the failing set exactly matched this installer's target set. launchctl list cannot make the same assertion print does: it is domain-blind, reporting a job as "registered" with no way to tell which domain, which is precisely why OBJ-3258's live remediation read the disk-guard job as healthy right up until it silently vanished, with zero log trail. See the general gotcha entry: launchctl load/unload/list are domain-blind.

Post-load verification (OBJ-3258, OBJ-3282)

Explicit-domain registration alone is still not proof a job actually ran. Confirmed twice on this host: OBJ-3258 (disk-guard registered in launchctl list but had silently vanished with zero log trail) and, during the reaper-hardening remediation, launchctl bootstrap returning 0 and the job registering cleanly while RunAtLoad never fired it — state = not running, runs = 0 immediately after load, until an explicit launchctl kickstart -k was issued (see the RunAtLoad host quirk below for the full data). So every install in install-disk-guard.sh now runs launchctl kickstart -k gui/$(id -u)/<label> and polls the job's own log file for a fresh line within KICKSTART_VERIFY_TIMEOUT_SECONDS (10s default) — not just registration. A job that never produces a fresh line fails the install loudly (exit 1) with a FATAL: message naming the label and log path, instead of reporting success on registration alone. The kickstart -k call itself is guarded with || true: the polling loop and its FATAL diagnostic are the actual verification either way, so a kickstart that itself errors still falls through to that diagnostic rather than aborting the script with no explanation (Roy's review on PR #2855 flagged this guard when an intermediate revision dropped it).

This supersedes an earlier, disk-guard-only fix (OBJ-3273) that polled disk-guard-status.json's checked_at for a value newer than pre-install — the uniform log-file check above is at least as strong evidence the job executed, and applies to all three managed agents, not just the one with a machine-readable status file.

On the reaper specifically, this verification is not inert. kickstart -k here doesn't just prove the job fired — it runs whatever ProgramArguments are already in the installed plist. If REAP_CONFIRM=1 promoted that plist to --confirm in this same install, the kickstart is a live, full delete sweep. See the REAP_CONFIRM=1 callout for the measured blast radius (OBJ-3331).

gc.status covers native in-daemon GC only, not the reaper (found 2026-09-02, OBJ-3276)

The gc field name in disk-guard-status.json is a trap for anyone triaging a disk incident. It greps daemon.log for "gc: cycle complete" (disk-guard/check-disk-and-gc-health.sh:194-235, installed on-host as ~/.multica/multica-disk-guard.sh) — that is a health check for native in-daemon GC only (the mechanism in Native GC config above). It says nothing about the reaper (reap-stale-task-workdirs.sh), which is a separate mechanism scheduled by its own separate LaunchAgent (com.longwoodlabs.multica-reaper — see Reaper schedule).

Confirmed live on joshuas-mbp-2, 2026-09-02: the reaper's LaunchAgent was silently deregistered from launchd for 22+ hours while gc.status reported OK, last_cycle_age_hours=0 the entire time — native GC kept cycling normally (6 cycles, ~5GiB reclaimed) because it genuinely was healthy, but a reader skimming "gc": {"status": "OK"} on a CRIT disk event would reasonably (and wrongly) conclude the reaper backstop was already covered. Root cause detail: OBJ-3276 (parent OBJ-3258).

Fixed (OBJ-3282, PR #2855): disk-guard-status.json now carries an independent reaper field — gc.status=OK was never evidence the reaper ran, is running, or is even loaded, and this field checks launchctl print gui/$UID/com.longwoodlabs.multica-reaper directly rather than inferring health from the unrelated gc field, mirroring this monitor's existing self-check pattern:

"reaper": {
  "status": "OK",
  "state": "running",
  "runs": 42,
  "last_log_age_hours": 1,
  "stale_after_hours": 72,
  "note": "Independent of gc — a different mechanism (com.longwoodlabs.multica-reaper LaunchAgent, not the native in-daemon GC). See OBJ-3282: gc.status=OK does not mean the reaper ran."
}

status is one of NOT_REGISTERED (not loaded in the domain at all), NEVER_RAN (registered but runs=0 — the exact OBJ-3282 incident signature: bootstrap returned 0 and the job registered cleanly, but RunAtLoad never fired it), STALE (log silent longer than REAPER_STALE_HOURS), UNKNOWN (launchctl absent, e.g. non-macOS, or neither state nor runs parsed), or OK. NOT_REGISTERED and NEVER_RAN roll into overall at the same CRIT/STALE rank as the disk and GC checks — see Rollup: highest severity wins below. Manual verification (launchctl print + reap.out.log) still works and is documented at Reaper schedule below, but reading this field is now the fast path.

Why the disk floors are where they are

df -g destroyed the resolution that mattered most. The original check read df -g, which truncates to whole GiB. At the 2026-08-31 incident's 115 MiB free, df -g reports free=0 — identical to 900 MiB free, and identical to a genuinely full disk. The check now reads df -k and evaluates in MiB. free_gb is still emitted for humans and back-compat, but free_mib is the authoritative field.

EMERGENCY is deliberately not set near zero. Below roughly 100 MiB the host cannot write an index.lock, a log file, or even a shell redirect — which means the remediation tooling itself stops working. That is not a hypothetical: during this incident a nohup ... > ~/reclaim.log & handoff silently did nothing because fish could not open the redirect target, and a /tmp/reap_candidates.txt write failed with No space left on device. A floor that only alarms once the disk is actually full alarms too late to be actionable. 10 GiB leaves room for the reaper to run, log, and let git take a lock.

df ... | tail -1 swallowed df's exit status — the same fail-open pipeline shape as the sweeper's dirty gate. Fixed identically: capture df's own $?, then tail the captured output. A failed df now reports UNKNOWN, not a false OK.

grep without -a silently disabled the GC alarm

Caught 2026-08-31, on the live host, hours after this guard was first installed there. The GC check read:

last_cycle_line=$(grep "gc: cycle complete" "$DAEMON_LOG" | tail -1)

~/.multica/daemon.log contains 503 lines with non-printable bytes, so grep classifies the file as binary and emits Binary file ... matchesinstead of the matching line. The awk '{print $1}' that follows then extracts the literal string Binary as the timestamp, and the sanity test [[ "$last_ts" > "$now_ts" ]] is true — letters sort after digits — so control falls straight into gc_status="UNKNOWN".

Net effect: the GC staleness alarm could never fire, on the one host it was written for. overall sat at UNKNOWN (correctly, via the rank rollup below — the rollup was not at fault), and a genuinely stalled GC would have looked identical to a healthy one. Fixed by grep -a. Confirmed on the host:

without -a:  last_cycle_line = [Binary file /Users/…/daemon.log matches]
             last_ts         = [Binary]
with    -a:  last_cycle_line = [16:26:08.157 INF gc: cycle complete … skipped=1017 …]
             last_ts         = [16:26:08]   now = [19:42:25]

The general shape worth remembering: a check that degrades to UNKNOWN on malformed input fails quietly by construction. It never reds, so nothing prompts anyone to look at it. UNKNOWN on a signal that should be routinely determinable is itself the alarm — treat a persistent UNKNOWN as a bug in the check, not as an absence of data.

Monitor self-age: what it does and does not fix

On 2026-08-28 this agent simply stopped firing. launchctl list still showed it loaded with last exit 0, .err.log was 0 bytes, and .out.log ended cleanly at overall=OK free=404GB — so disk-guard-status.json kept reporting "overall":"OK","free_gb":"404" for three days while the volume slid to 115 MiB free. A monitor that stops running reports last-known-good forever. Two mitigations, and it matters to be precise about each:

1. Every status file carries its own expiry. interval_seconds, stale_after_seconds, and next_check_due_by are written into the JSON, so any reader — a human, a jq one-liner, an external watchdog — can decide the file is stale without knowing this agent's cadence.
2. A resumed monitor reports its own gap. Each run stats the previous status file's mtime and, if more than stale_after_seconds has elapsed, reports monitor.status = RESUMED_AFTER_GAP and escalates overall to at least WARN.

Neither detects a currently-dead monitor. A process that is not running cannot alarm about not running. Mitigation 2 only fires once the agent comes back — which, in this incident, would have been three days late. Closing that gap requires something outside this process reading next_check_due_by and alarming when it passes. This script makes that external check trivial to write; it is not a substitute for one.

OBJ-3084 wired that external check into the existing Dave — gc maintenance watchdog autopilot (b8558275-1db1-4655-b484-bd086ce7c83c, resumed from paused to active 2026-08-31), as a second, independent check alongside its pre-existing git-maintenance-failure detection (see Multica gc/Maintenance Silent Failure). On each run it now also:

• Reads ~/.multica/disk-guard-status.json on joshuas-mbp-2.
• If monitor.next_check_due_by is present, compares it to now — past due fails the check.
• Fails closed on the transition window, not just on staleness. If the monitor block or next_check_due_by key is absent entirely — the state the live host was actually in until the reaper-scheduling task deployed the updated monitor script — that reads as an alarm too, not as "looks fine." A missing signal is never treated as an all-clear.
• Files a separate [disk-guard-watchdog]-titled issue on failure, deduped independently from the pre-existing [gc-watchdog] git-maintenance issues — the two checks share one autopilot but never merge their alerts.

This closes the gap regardless of which launchd-level mechanism turns out to have caused the 2026-08-28 stop (see the next section) — staleness-based alarming only needs to know the file stopped updating, not why.

Root cause of the 2026-08-28 stop: investigated, not determined (OBJ-3083)

The leading hypothesis for why the agent stopped firing on 2026-08-28 (see above) was a macOS sleep/wake transition dropping the StartInterval timer registration — StartCalendarInterval is the documented mechanism that reliably re-fires after a missed window, and a separate change migrates this plist to it on that basis. OBJ-3083 tested the hypothesis directly against this host's own logs and refuted it: pmset -g log's own end-of-session summary line, for the session containing the incident (boot 2026-08-26 06:22 → next reboot 2026-08-31 18:49), reports zero sleep/wake cycles — corroborated by continuous ~13–15 min assertion-log activity running straight through the exact stop window. There was no wake transition for a timer to drop across.

The true root cause could not be determined, because the one log source that could show it — the unified log — had already rotated past the incident window by the time anyone investigated (see the retention gotcha below). StartCalendarInterval remains reasonable hardening, but don't record it as the fix for this specific incident — it doesn't address whatever actually killed the timer.

Four host/OS-build-specific foot-guns this investigation (and a follow-up, OBJ-3258) surfaced, for whoever diagnoses a silent-launchd-failure next on this host (macOS 26.5.2):

1. log show --predicate 'subsystem == "com.apple.xpc.launchd"' is a dead predicate on this build — it returns 0 rows unconditionally, including during --last 6h on a live session seconds after launchd activity. launchd's log lines on this macOS build carry process == "launchd" with no subsystem field set. Use process == "launchd" instead — the wrong predicate silently looks like confirmation of "no anomaly" rather than a broken query.
2. Unified-log retention for process == "launchd" on this host is currently only ~22 hours — far shorter than the realistic lag between a silent failure and someone investigating it (87.7h to discovery here, via the unrelated OBJ-3055 disk-exhaustion symptom, plus queueing time before the investigation ran). A runbook step that says "check log show for launchd" only works if run within hours of the incident, not days.
3. launchctl unload on an already-absent service can return an anomalous errno instead of "Could not find service" (OBJ-3258, live remediation of the disk-guard job on 2026-09-02). The expected/normal response to unloading a service that isn't loaded is Could not find service — that's what the original diagnostic snapshot for this incident saw. During the fix, the same idempotent launchctl unload (mirroring the installer's own unload-before-load step) instead returned Unload failed: 5: Input/output error. Treat this as seen, not necessarily alarming on its own — one data point, not proof of a broader degraded state — but don't mistake it for a new failure mode if it recurs.
4. RunAtLoad is not reliably immediate on this host (OBJ-3258, same remediation). A fresh launchctl load can leave the job correctly registered (launchctl print shows runatload | inferred program) while runs stays 0 for several minutes, with no execution until the next natural StartCalendarInterval tick actually fires it. Anyone verifying a reload should confirm execution via the job's own output (e.g. disk-guard-status.json's checked_at, or a fresh .out.log line) at the next real tick — not assume runs > 0 immediately after load reports success.

This residual risk — root cause open, forensic trail perishable — is why the Dave — gc maintenance watchdog autopilot now captures a forensic snapshot (Step 8) the moment its disk-guard-staleness check (Step 7) fails: a launchctl print + log show --predicate 'process == "launchd"' --last 6h, pasted in full into the alert issue, instead of leaving that evidence for a human to gather after it's already aged out. That closes the gap regardless of which launchd-level mechanism eventually turns out to be at fault — staleness-based alarming only needs to know the file stopped updating, not why.

Rollup: highest severity wins

The original rollup upgraded overall only if it was still OK, so an early UNKNOWN could mask a later WARN. That was invisible while gc was the only other signal (it never emits WARN), but the new monitor check does emit WARN and is evaluated third. The rollup is now rank-based — EMERGENCY(4) > CRIT/STALE(3) > WARN/RESUMED_AFTER_GAP(2) > UNKNOWN(1) > OK/FIRST_RUN(0) — and the highest rank across all three signals wins. Verified: a failing df plus a 3-day monitor gap now yields overall=WARN, not UNKNOWN.

Failure stance: fail-open on the monitor's own errors (no -e, KeepAlive absent — a broken check waits for the next tick rather than crash-looping or alarm-flooding), but fail-closed on the signal itself: if disk, GC, or monitor state can't be determined, it reports UNKNOWN loudly, never a false OK.

Gotcha found and fixed during verification (2026-08-19): daemon.log timestamps are local host time (CT on this host), not UTC — confirmed by comparing a live log line (16:36:26) against the host's own date output (17:10:58 CDT, ~34 min apart, consistent with the 2h cadence). An initial version of this check compared the log's local timestamp against date -u and read a genuinely-0h-old cycle as 5 hours old. Fixed to compare local time on both sides.

Signal surfaces (all repo-documented, all "findable"):

• ~/.multica/disk-guard-status.json — machine-readable last-check result.
• ~/.multica/disk-guard.out.log / .err.log — matches the real daemon's own ~/.multica/launchd.{out,err}.log convention.
• logger -t com.longwoodlabs.multica-disk-guard ... on any non-OK result — visible in Console.app / log show --predicate 'process == "logger"'. EMERGENCY and RESUMED_AFTER_GAP append an explanatory clause naming what to do and what was not being reported during the gap.
• A best-effort osascript macOS user notification (requires a logged-in GUI session; failure here is silently swallowed — the log/status-file signals are authoritative, this is a convenience layer on top).

Status file shape (2026-08-31 format for checked_at/disk/gc/monitor; reaper added OBJ-3282, not present in the original 2026-08-31 capture). JSON nulls are bare, not the string "null" — the previous "${x:-null}" heredoc form emitted a four-character string that jq -r prints as null with exit 0, which is indistinguishable from a real value to a string compare:

{
  "checked_at": "2026-08-31T23:39:31Z",
  "overall": "WARN",
  "disk": {
    "status": "OK", "free_mib": 413696, "free_gb": 404,
    "warn_below_mib": 153600, "crit_below_mib": 76800, "emergency_below_mib": 10240
  },
  "gc": {"status": "OK", "last_cycle_age_hours": 0, "stale_after_hours": 6},
  "monitor": {
    "status": "RESUMED_AFTER_GAP",
    "interval_seconds": 900,
    "stale_after_seconds": 2700,
    "next_check_due_by": "2026-09-01T00:24:31Z",
    "previous_check_age_seconds": 259231,
    "note": "If now > next_check_due_by, this file is STALE and its values are last-known-good, not current. A dead monitor cannot alarm about itself; that check must be made by a reader."
  },
  "reaper": {
    "status": "OK", "state": "running", "runs": 42,
    "last_log_age_hours": 1, "stale_after_hours": 72,
    "note": "Independent of gc — a different mechanism (com.longwoodlabs.multica-reaper LaunchAgent, not the native in-daemon GC). See OBJ-3282: gc.status=OK does not mean the reaper ran."
  }
}

Tier boundaries are exclusive-below (-lt), verified by sweep:

Free	Reported
404 GiB	disk=OK free=404GB
150 GiB	disk=OK (at the boundary, not below it)
140 GiB	disk=WARN
75 GiB	disk=WARN (boundary)
50 GiB	disk=CRIT
10 GiB	disk=CRIT (boundary)
5 GiB	disk=EMERGENCY
108 MiB	disk=EMERGENCY free=108MiB

Install:

scripts/multica-host/disk-guard/install-disk-guard.sh

Reaper schedule

disk-guard/com.longwoodlabs.multica-reaper.plist schedules the backstop sweeper above as its own LaunchAgent, com.longwoodlabs.multica-reaper — a Label distinct from both com.longwoodlabs.multica-daemon and com.longwoodlabs.multica-disk-guard, so loading/unloading it touches neither. It fires once daily (StartCalendarInterval at 03:30, not the monitor's 15-minute cadence — a filesystem sweep doesn't need quarter-hour resolution) and, like the monitor plist, uses StartCalendarInterval rather than StartInterval for the same 87.7-hour-silent-stop reason (see Root cause of the 2026-08-28 stop). disk-guard/install-disk-guard.sh installs both the monitor and the reaper together — since OBJ-3186, it refuses to run while either installed target (including a live reaper --confirm pass) is currently executing; see Recurrence 2026-09-02.

The checked-in plist ships dry-run-only. Its ProgramArguments carry no --confirm flag, matching reap-stale-task-workdirs.sh's own safe default — a fresh host bootstrapping from this repo is never one launchctl load away from live deletion with no verification cycle in between. Promotion to live deletion is a deliberate, host-local decision, not a repo-tracked default — REAP_CONFIRM=1 edits the installed plist copy only, via PlistBuddy; the repo template is untouched, and re-running the installer without REAP_CONFIRM set reverts the installed copy back to dry-run:

# Install both LaunchAgents (safe any time — dry-run only until promoted):
scripts/multica-host/disk-guard/install-disk-guard.sh

# Read the dry-run log before trusting or promoting anything:
cat ~/.multica/reap.out.log

# Only once that count looks sane, promote the INSTALLED reaper copy to live
# deletion:
REAP_CONFIRM=1 scripts/multica-host/disk-guard/install-disk-guard.sh

That REAP_CONFIRM=1 install is itself a live delete sweep, not just a promotion — the post-load kickstart verification fires it immediately. install_agent bakes --confirm into the installed plist's ProgramArguments (install-disk-guard.sh:227-230) before bootstrapping and verifying the job (:247, :256); verify_post_load's launchctl kickstart -k (:185) then runs whatever is in that plist for real, to prove the job fired. So the same command that promotes a host to live deletion also triggers one full reap-stale-task-workdirs.sh --confirm pass — not --shells-only, and not a smoke check — before the operator runs anything else by hand. Confirmed on joshuas-mbp-2 2026-09-03 (OBJ-3331): a REAP_CONFIRM=1 reinstall's kickstart ran ~13 minutes, scanned 1,926 dirs, and reclaimed 120 (52 empty lock shells + 68 full clean/merged task dirs). On a host carrying a large backstop backlog this is a materially bigger blast radius than "did the job start" — read the dry-run log before setting REAP_CONFIRM=1 (see the count sanity-check below), since setting it is the live pass, not a step before one.

An implausibly small task-dir count is the tell for the 8-hex regression coming back — a standing operator instruction, not a one-time check. The 8-hex-glob regression that caused this whole incident chain produced total_task_dirs=8 while the identical scan with a 12-hex arm found 1001 — a silent 99.2% miss that still looked like a plausible, clean result. Before ever running --confirm by hand, setting REAP_CONFIRM=1, or trusting an unattended dry-run cycle, compare the reaper's reported scanned= count in ~/.multica/reap.out.log against a manual count of task dirs on the host. A count in the low-hundreds on a normally-loaded host is sane (the reaper-scheduling rollout treated anything near ~18 instead of the expected ~350 as the explicit gate for re-checking the glob, per this same regression shape); a count near the low tens means the glob or case-gate has regressed again, not that the host is suddenly clean. Validated at rollout: 650 reaper-reported vs. 653 manually counted (0.5% diff) — that's the shape a healthy scan looks like.

joshuas-mbp-2 was promoted to live --confirm on 2026-09-01 after a clean manual validation cycle (18 zero-risk empty shells reclaimed, matching the dry-run prediction exactly — dirty=0 unreadable=0 detached=0 unreffed=0 no_mirror=0). A human spot-check after the reaper's first unattended 03:30 fire is still worth carrying forward: that validation cycle only ever had empty shells available to reclaim, so a real full-checkout deletion under the live daily schedule hasn't been exercised yet.

Verifying reaper health manually

disk-guard-status.json's independent reaper field (OBJ-3282, see above) is the fast path once installed — jq '.reaper' ~/.multica/disk-guard-status.json. The manual check below is what that field is built on, and is still useful for a live triage, if the status file itself is suspect, or on a host that hasn't yet re-run the installer to pick up OBJ-3282:

launchctl print gui/$UID/com.longwoodlabs.multica-reaper   # "Could not find service" = not loaded at all
cat ~/.multica/reap.out.log                                 # last entry should be recent, and post-date any plist/script redeploy

A registered-but-never-run job (see the RunAtLoad quirk directly below) can pass the first check and still fail the second — check both. This is exactly the NEVER_RAN case the reaper.status field now catches directly, via runs=0 from launchctl print, without needing to run both commands by hand.

Host quirk: RunAtLoad is not reliably immediate on this host (confirmed twice more, 2026-09-02)

com.longwoodlabs.multica-reaper.plist and com.longwoodlabs.multica-disk-guard.plist both set RunAtLoad. It is not guaranteed to fire immediately on a manual reload on this host — three independent, directly-observed data points now:

• Disk-guard, 2026-08-31 (documented in the disk-guard plist's own StartCalendarInterval comment): a manual launchctl unload/load resumed the job within ~40s of the reload — RunAtLoad worked here.
• Disk-guard, 2026-09-02 (OBJ-3258): a later launchctl unload/load, recovering from the same LaunchAgent's silent deregistration, did not fire immediately — runs = 0, last exit code = (never exited) for 4+ minutes post-load. The job only actually executed at the next natural StartCalendarInterval tick.
• Reaper, 2026-09-02 (OBJ-3276): launchctl bootstrap gui/501 <plist> returned 0 and the job registered cleanly (launchctl print showed runatload | inferred program), but immediately after: state = not running, runs = 0. It only ran once launchctl kickstart -k gui/501/com.longwoodlabs.multica-reaper was issued explicitly.

So RunAtLoad sometimes fires promptly and sometimes doesn't, with no observed way yet to predict which — a coin-flip you cannot see the outcome of without checking. Operator rule: after loading or bootstrapping either LaunchAgent on this host, don't trust RunAtLoad (or registration in launchctl list/print alone) to mean the job actually ran. Always follow with an explicit launchctl kickstart -k gui/$UID/<label> and confirm a fresh line in the job's own log (disk-guard.out.log / reap.out.log) before considering the reload verified.

Automated for all three managed agents (OBJ-3282, PR #2855):install-disk-guard.sh now performs exactly this kickstart-then-confirm sequence itself on every run — see Post-load verification above — so running the installer already gives you this guarantee. The operator rule above still applies verbatim to anyone reloading a LaunchAgent by hand outside the installer (e.g. a manual launchctl bootstrap during live triage).

Stale-lock repair procedure

The other recurring failure on this host — gc: git maintenance failed ... error="signal: killed" — is caused by stale lock files on the shared repo mirror (~/multica_workspaces/.repos/<workspace-id>/github.com+objectuve-softworks+enkidu.git). Check both of these; a refs-only scan misses the worktree case:

repo=~/multica_workspaces/.repos/<workspace-id>/github.com+objectuve-softworks+enkidu.git

# 1. ref locks
find "$repo/refs" -name "*.lock"

# 2. per-worktree HEAD locks — NOT covered by the refs-only scan above
find "$repo/worktrees" -maxdepth 2 -name "HEAD.lock"

# also worth checking: a stale gc.pid
find "$repo" -maxdepth 1 -name "gc.pid"

If any are found and no git/multica process is actually running against that path (verify with ps aux | grep git before touching anything — removing a lock held by a live process corrupts the repo), remove the stale lock file(s) and re-run:

rm "$repo/refs/.../<name>.lock"        # per finding above
rm "$repo/worktrees/<wt>/HEAD.lock"    # per finding above
git -C "$repo" gc --prune=30.days

This was performed manually as part of the 2026-08-19 triage (no stale locks were present at verification time — the mirror was already clean). This procedure is the durable record of how to repeat it if it recurs; see Multica gc/Maintenance Silent Failure for why this failure is silent (no health-check signal) and the existing gc-maintenance-watchdog autopilot that escalates it after 3 consecutive failures — that autopilot is unaffected by anything in this doc.

Related

• scripts/multica-host/README.md — index of the repo-tracked host artifacts
• Multica gc/Maintenance Silent Failure — the silent-failure class this doc's stale-lock section belongs to
• Multica Runtime Identity (daemon.id hijack) — same host, a different "every liveness signal stays green" failure mode
• Personal Mac Doubles as Shared Crew Host — same No space left on device symptom, a different and not agent-fixable root cause: TCC-protected personal data (Photos/Mail/Messages/iCloud) sharing this volume with the agent fleet
• OBJ-3083 — refuted the sleep/wake StartInterval hypothesis for the 2026-08-28 stop and found the two log show/launchd foot-guns above
• OBJ-3084 — resumed and extended the Dave — gc maintenance watchdog autopilot with the disk-guard-staleness check described above
• OBJ-3085 — shipped the reaper LaunchAgent, the StartCalendarInterval migration, and the REAP_CONFIRM=1 staged-promotion flow
• OBJ-3081 (backlog) — sparse-checkout/partial-clone spike, the real lever on the materialized-working-tree disk cost --reference cannot touch
• OBJ-3186 — atomic install + pre-flight running-target guard for install-disk-guard.sh, closing the cp-truncation hazard described in Recurrence 2026-09-02
• OBJ-3258 — disk-guard-watchdog caught the LaunchAgent silently deregistered on joshuas-mbp-2; live remediation (reload) surfaced the two additional launchd foot-guns above (unload I/O-error errno, non-immediate RunAtLoad)
• OBJ-3258 / OBJ-3276 — the reaper's LaunchAgent silently deregistering (same failure signature as disk-guard's), the gc.status field-scope trap, and the RunAtLoad host quirk documented above
• OBJ-3282 (Codi, shipped, PR #2855) — hardened LaunchAgent install-verification to use kickstart -k (see Post-load verification) and added the independent reaper field to disk-guard-status.json (see gc.status covers native in-daemon GC only, not the reaper)
• OBJ-3283 (Orion) — systemic root-cause analysis for why two sibling LaunchAgents on this host both silently deregistered the same day (traced to the legacy launchctl load/unload API; fixed by OBJ-3282 Task 3, PR #2855 — see Explicit-domain registration). Also measured the unknown-bucket composition documented above — an earlier framing of this as unbounded task-dir growth was investigated and retracted; the real, smaller defect is the empty-shell/backstop ordering bug tracked as OBJ-3286
• OBJ-3286 (Codi, shipped) — reordered the reaper's empty-shell reclaim ahead of the unknown backstop and recalibrated UNKNOWN_BACKSTOP_DAYS; see unknown bucket composition above
• OBJ-3285 (Orion) — traced lock-only task-dir stubs to daemon wakeup behavior via per-task-UUID log tracing, confirming no unintended .gc_meta.json write gap; see Lock-only stubs traced to daemon wakeup above
• OBJ-3275 (sibling docs entry, same incident family) — anomalous Input/output error unloading an absent service, and the first RunAtLoad confirmation (on disk-guard)
• OBJ-3331 (Codi, deploy; Dori, docs) — deployed OBJ-3286/OBJ-3193 to joshuas-mbp-2 and documented that a REAP_CONFIRM=1 install's post-load kickstart -k runs the reaper with --confirm already active — a live, full delete sweep as a side effect of verification, not a smoke check; see Reaper schedule and Post-load verification

---

Last updated: 2026-09-03 (OBJ-3331: document that a REAP_CONFIRM=1 install's post-load kickstart -k runs the reaper with --confirm already baked into its plist — a live, full delete sweep triggered as a side effect of install verification, not a liveness smoke check — with the measured blast radius from the joshuas-mbp-2 deploy; OBJ-3285: document lock-only task-dir stubs as expected daemon wakeup behavior, not a leak, with the per-task-UUID log trace and the confirmed absence of a .gc_meta.json write gap; OBJ-3301: document OBJ-3282's shipped LaunchAgent hardening [PR #2855] — explicit-domain bootout/bootstrap/print registration [OBJ-3283 root cause] and launchctl kickstart -k + fresh-log-line post-load verification, both now in install-disk-guard.sh; add the reaper field's real shape to disk-guard-status.json's documented JSON and the checks table; update "verify manually" framing now that the field exists; cross-link the new docs/development/gotchas.md entry for domain-blind launchctl load/unload/list; OBJ-3286: correct the unknown category's stale "default 14" UNKNOWN_BACKSTOP_DAYS reference — now derived from TTL_DAYS — and document the checkout-less RECLAIM shell path that skips the backstop entirely; OBJ-3284: clarify gc.status field scope [native GC only, not the reaper], document the RunAtLoad host quirk confirmed again during reaper remediation, and document .gc_meta.json's write-time semantics + the measured unknown-bucket composition (Task C, OBJ-3283); OBJ-3258: document two launchd quirks surfaced during live disk-guard reload remediation; OBJ-3206: document the mirror worktree-admin prune behind --confirm; OBJ-3186: install-disk-guard.sh cp-truncation recurrence + atomic-install fix)