Multica daemon: silent git-maintenance failures on the shared enkidu mirror

Applies to: the crew runtime (bf3c31e7-4a7e-452d-98c7-d0e88beda4dd, host joshuas-mbp-2.lan) and its shared repo cache at ~/multica_workspaces/.repos/f56c86e0-…/github.com+objectuve-softworks+enkidu.git. Any repo cache on any Multica daemon can hit the same failure mode — the mirror path above is just the one that has actually tripped it.

The daemon's per-cycle git maintenance on a shared repo mirror can fail every single cycle for days with zero escalation. /health stays 200, the runtime stays online, and the only trace is one WRN line per failed command in ~/.multica/daemon.log. The observable symptom is repo checkout hanging 20–40 minutes with no pointer back to the cause.

The incident (OBJ-2449 → OBJ-2476, 2026-08-11/12)

During the OBJ-2449 escalation, repo checkout brownouts stalled Riley and Maggie for two multi-hour windows (11:55→12:32 and 13:53→14:15 CT, 2026-08-11). The crew burned hours diagnosing "checkout hangs" from the wrong end — network, GitHub, the agent itself — because nothing in the daemon's health surface pointed at git maintenance. The only signal was a WRN gc: git maintenance failed pair roughly every 2h in ~/.multica/daemon.log, and nobody was watching that file.

Still live at time of writing. This is not a closed incident — it re-fired during this doc's own drafting session:

06:11:10 WRN gc: git maintenance failed repo=…/github.com+objectuve-softworks+enkidu.git command="gc --prune=30.days" error="signal: killed"
08:01:34 WRN gc: git maintenance failed repo=…                                       command="reflog expire --expire=30.days --all" error="signal: killed"
08:20:24 WRN gc: git maintenance failed repo=…                                       command="gc --prune=30.days" error="signal: killed"
09:57:35 WRN gc: git maintenance failed repo=…                                       command="reflog expire --expire=30.days --all" error="signal: killed"

Mirror size at the time of the 09:57 failure: 2.9 GB (du -sh ~/multica_workspaces/.repos/f56c86e0-…/github.com+objectuve-softworks+enkidu.git). The same gc cycles logged gc: deleted stale agent branches … count=26 and count=27 on this repo — a bloated mirror is why maintenance runs long enough to hit the daemon's own timeout. That bloat (junk remotes/branches) is tracked separately as OBJ-2477; this doc is only about the failure being silent, tracked as OBJ-2476.

Why every liveness signal stays green

• /health reports on the daemon's own process health, not on the outcome of its background maintenance jobs — a stuck git gc doesn't touch that endpoint.
• multica runtime list shows status: online / last_seen_at from heartbeats, which keep flowing normally even while a maintenance command is hung waiting to be killed.
• multica daemon status --output json returns pid/uptime/task counts/ workspaces — there is currently no field exposing per-repo-cache maintenance state.

Heartbeat is not maintenance health, the same way it isn't auth health (see Multica Runtime Identity) — a daemon can be perfectly healthy at the transport layer while every checkout against one specific mirror hangs.

Reading error="signal: killed"

signal: killed on a gc/reflog expire line is the daemon's own timeout kill, not an external OOM-killer or a crash — the daemon gives the maintenance subprocess a bounded window and SIGKILLs it if it doesn't finish. That reading isn't discoverable from the line itself; the daemon doesn't currently log how long the command ran before being killed, so "timeout kill" vs. "stale lock contention" (a gc process failing near-instantly because another process already holds .git/gc.pid or a ref lock) are indistinguishable without independently timing the command. Don't assume signal: killed means someone else killed it — check for a stale lock file in the mirror's .git/ before ruling that out too.

Diagnosis

One grep settles whether a mirror is in this state:

grep "git maintenance failed" ~/.multica/daemon.log

If checkouts against a specific repo are hanging, filter to that repo's cache path and look for a run of consecutive failures with no gc: cycle complete success in between for that repo=:

grep -E "git maintenance failed|gc: cycle complete" ~/.multica/daemon.log

Current mirror size — a multi-GB mirror for a monorepo this size is itself a signal that maintenance has been failing long enough to accumulate cruft (see OBJ-2477):

du -sh ~/multica_workspaces/.repos/<workspace-id>/<host>+<org>+<repo>.git

The gc-maintenance-watchdog autopilot

OBJ-2476 shipped a crew-side detector rather than waiting on an upstream daemon fix (Part A below is not crew-actionable — see "Open gap"). The autopilot Dave — gc maintenance watchdog runs host-local every 6h (~/.multica/daemon.log is host-local; GitHub Actions cannot see it, so the existing multica-agent-health.yml cron pattern does not extend to this class of failure). Its full detection logic is committed at ops/multica-gc-health-runbook.md. Summary:

• Segments the log into cycles at each gc: cycle complete line and counts consecutive failing cycles per repo= path.
• At 3 consecutive failures, files (or comments on an existing) a [gc-watchdog]-titled issue with the repo path, failure count, failing command=/error= values, first/last timestamps, and mirror size.
• Never checks out a repository — a checkout mid-brownout would hang the watchdog too.
• Ships active as of 2026-08-31 (was paused at ship time; a human flipped it active — see the multica autopilot get status check in OBJ-3090).

Second responsibility, added by OBJ-3084 (2026-08-31): disk-guard staleness. This same autopilot also now reads ~/.multica/disk-guard-status.json on joshuas-mbp-2 every run and alarms if monitor.next_check_due_by has passed — or if the monitor block is absent entirely, which fails closed rather than reading as "fine." On failure it files a separately-deduped [disk-guard-watchdog]-titled issue, never merged with [gc-watchdog]. This check is unrelated to the git-maintenance detection above — it's a second, independent monitor riding the same 6h schedule because it closes the "a process that is not running cannot alarm about not running" gap in the disk-guard LaunchAgent's own design. Full detail: Multica Daemon Disk Hygiene § Monitor self-age. Note: the live autopilot definition (multica autopilot get b8558275-1db1-4655-b484-bd086ce7c83c) is authoritative for this second check's exact logic — unlike the git-maintenance detection above, ops/multica-gc-health-runbook.md was not updated to include it when OBJ-3084 shipped.

Cycle-boundary caveat. The watchdog only counts a cycle once its closing gc: cycle complete line has logged. A cycle currently stuck inside a gc --prune that will eventually get killed doesn't count toward consecutive_failures until that cycle finishes — so a watchdog run that reports "all clear" during an active brownout is reporting on the last completed cycle, not the one in progress. This is a reasonable diagnostic-not-realtime tradeoff at the 6h cadence, but don't read a clean watchdog trigger as proof no brownout is happening right now.

Open gap — the daemon-side fix is not crew-actionable

The actual fix (tracking consecutive failures per repo cache inside the daemon and degrading health/emitting a distinct ERR marker after N) is Multica platform code. No crew agent has that source — this workspace's only registered repo is enkidu. The upstream spec (consecutive-failure tracking, health degradation or a stable ERR marker at 3 failures, command duration logging, and a programmatic maintenance-state surface) was filed with Josh via OBJ-2476's issue thread for out-of-band routing to the Multica maintainers. Until that lands, the gc-maintenance-watchdog autopilot above is the only escalation path for this failure class.

Related

• OBJ-2477 — the junk-remote bloat that makes maintenance slow enough to time out (the cause; this doc covers the silence)
• OBJ-2476 — this ticket
• ops/multica-gc-health-runbook.md — the watchdog's committed detection logic
• Multica Runtime Identity (daemon.id hijack) — the same "every liveness signal stays green" failure class, different root cause
• Multica Daemon Disk Hygiene — the stale-lock repair procedure for the same shared mirror lives there, plus the task-dir disk-exhaustion root cause; a different failure on the same mirror, same "everything reports healthy while the real signal is buried in daemon.log" shape. This is also where this autopilot's second responsibility (disk-guard staleness, OBJ-3084) is documented in full.
• OBJ-3084 — added the disk-guard-staleness check to this autopilot

---

Last updated: 2026-09-01 (OBJ-3086: cross-link the disk-guard-staleness responsibility added by OBJ-3084)