Objectuve Docs

Appearance

Multica autopilot run status: "failed" is not reliable fault ground truth ​

Applies to: any process reading multica autopilot runs <id> or multica autopilot list to detect fleet-wide autopilot failures — currently the Dave — Autopilot run failure sweep autopilot (a43b4c1d-1fd4-404c-86e8-fec51ce3dbf9, §2f in the autopilots guide).

A run's recorded status/failure_reason is not a reliable, self-updating fault signal. Three distinct ways a naive reader gets the wrong answer, all found while scoping OBJ-3279:

1. issue blocked runs never update after the block clears ​

A run fails with failure_reason: "issue blocked" when it stalled waiting on a human to unblock the linked issue. The run row is a point-in-time record — it never gets rewritten once the stall clears, so a run that blocked, got unblocked, and finished normally days ago still shows status: "failed" forever. Checked directly against OBJ-3279's own historical dataset (32 autopilots, 420 retained runs, 2026-04-27 → 2026-09-03): of 48 failed runs, 13 carried failure_reason: "issue blocked", and a spot-check of three (OBJ-3148, OBJ-3146, OBJ-3060) found all three linked issues already done. A sweep trusting status alone reports roughly a quarter of its findings as live faults that were resolved days earlier.

The fix: the linked issue_id's current status is the real signal, not the run's own status field. Look it up via multica issue get <issue_id> --output json before deciding whether an issue blocked run is still live.

2. last_run_status hides fail-then-succeed ​

multica autopilot list --output json returns last_run_status, which reflects only the most recent run. An autopilot that fails and then succeeds on its very next fire shows last_run_status: completed — the intermediate failure is invisible. This is the exact incident that surfaced the whole gap: Dave — gc maintenance watchdog died on a session-limit failure at 2026-09-02 17:17 UTC, then completed normally at 23:17 the same day. Nothing read the failed run in the six hours between, because nothing read runs — only list's always-fresh last_run_status.

The fix: use list only to enumerate the fleet (it's the right source for every active autopilot's id). The actual fault detector has to be multica autopilot runs <id> --output json, read in full for the lookback window, never just the newest row.

3. Real faults arrive correlated, not per-autopilot ​

Infrastructure faults hit many autopilots in the same window with the same failure_reason, and a per-autopilot reading turns one incident into many issues. In the same OBJ-3279 dataset: 7 runs failed with Failed to authenticate. API Error: 401 — all from a single expired-credential event on 2026-06-30/07-01. 11 runs were skipped with agent runtime is offline at dispatch time — from two separate host-down events (2026-06-01, 2026-06-16). A sweep that files one issue per failing run turns those two incidents into 18 issues instead of 3.

The fix: group findings by (failure_reason, day) before filing — one incident is one line (or one issue), not N.

The mitigation ​

Dave — Autopilot run failure sweep (§2f in the autopilots guide) applies all three corrections: it reads runs per autopilot (never trusts last_run_status), resolves issue blocked runs against the linked issue's current status before tiering them, and groups same-day correlated failures into a single digest or dedicated issue instead of one per run. It runs every 6h with a 12h lookback, so a missed sweep never loses a failure — every run is seen by two consecutive sweeps.

Last updated: 2026-09-03 (OBJ-3279)

Loading…