Common Gotchas

This file was extracted from CLAUDE.md's former ## Common Gotchas section (2026-08-10) to reduce the token cost of the always-loaded project instructions. CLAUDE.md now carries a compact one-line index pointing here; this file is the canonical full record of every documented gotcha, verbatim. When you discover a new gotcha, add its full write-up here (as a new ### entry under the right domain group) and add a matching one-line index entry back in CLAUDE.md's ## Common Gotchas section.

CI & GitHub Actions

CI hard-gate steps: provision secrets before merging

• CI hard-gate steps: provision secrets before merging — When adding a Verify required secrets step to any GitHub Actions workflow, the secret must be provisioned in Repository Settings → Secrets and variables → Actions BEFORE merging the PR that adds the verify step. Provisioning after merge blocks 100% of PR CI until fixed. See docs/development/testing.md#ci-secret-provisioning for full details.

Secret-to-variable migration: delete secret only after merge

• Secret-to-variable migration: delete secret only after merge, not before — When promoting a GitHub Repository Secret to a Repository Variable, do NOT delete the old secret until the PR removing the secrets.<NAME> reference has been merged to master. Deleting the secret first causes secrets.<NAME> to resolve to "" on master, breaking any workflow that uses it — even workflows with unrelated content changes. This is the symmetric risk to the gotcha above. (Root cause: OBJ-751, where GCP_PROJECT_ID was deleted from Secrets before chore/gcp-project-id-secret-to-var merged, failing all 5 staging Docker builds and blocking production deploys for multiple releases.)

Duplicate env: keys in a reusable workflow break every caller

• Duplicate keys in a reusable workflow's env: block silently break every caller — A duplicate key (e.g. CLERK_PUBLISHABLE_KEY declared twice in the same env: mapping in .github/workflows/playwright-smoke.yml) makes GitHub Actions reject the workflow at parse time. When a caller workflow references it via uses: ./.github/workflows/foo.yml, the rejection cascades up: the caller fails with 0 jobs created, 0s duration, "This run likely failed because of a workflow file issue." — no logs, no annotations, no check runs. The gh run view name field shows the file path instead of the workflow name, which is the strongest signal that parse failed server-side. If only one environment (e.g. production) uses the reusable workflow, the breakage can persist undetected for many deploys while staging stays green (this happened 2026-05-23: tags v3.9.225–231 all silently dropped because production.yml calls playwright-smoke.yml but staging.yml does not). Cause is usually a squash-merge that re-introduces a line already added on master. Detection: actionlint .github/workflows/*.yml flags it instantly as syntax-check. Run actionlint locally before merging any PR that touches workflows. Recovery: the broken workflow file is baked into existing tags' commit trees, so previously-pushed tags cannot be retroactively deployed — once fixed on master, you must cut a fresh tag with scripts/release.sh to roll forward.

Secret scanner silently suppresses job outputs matching secrets

• GitHub Actions secret scanner suppresses job outputs containing secret substrings — At Complete Job time, GitHub Actions scans job-level outputs: and silently drops any output whose value matches a repo secret (e.g. an API key, deployment token). Firebase preview channel URLs contain the site ID (enkidu-app), which may collide with a secret value in the repository, triggering suppression — the output becomes an empty string with no error, no annotation, and no indication that suppression occurred. This is the same silent-failure family as the duplicate-key gotcha: CI passes locally, but breaks in GitHub. Fix: extract only the URL hash (not the full enkidu-app URL) as the job output, then reconstruct the full URL at the playwright-smoke with: call site using github.event.number. The hash alone does not match any registered secret, so it crosses the job boundary without suppression. Step-level $GITHUB_OUTPUT writes within the same job (e.g., for PR comments) are unaffected by the scanner. Example:

  # ❌ unsafe — full enkidu-app URL as job output is suppressed by the scanner
  -  outputs:
  -    url: ${{ steps.set-url.outputs.url }}
  -  ...
  -  playwright-smoke:
  -    with:
  -      frontend_url: ${{ needs.deploy-preview.outputs.url }}  # receives ""
  
  # ✅ safe — extract only the hash (no enkidu-app, no suppression); reconstruct at call site
  +  outputs:
  +    url_hash: ${{ steps.set-url.outputs.url_hash }}
  +  ...
  +  # in set-url step:
  +  URL_HASH=$(echo "$ACTUAL_URL" | sed 's/.*-\([a-z0-9]*\)\.web\.app$/\1/')
  +  echo "url_hash=$URL_HASH" >> "$GITHUB_OUTPUT"
  +  ...
  +  playwright-smoke:
  +    with:
  +      frontend_url: https://enkidu-app--pr-${{ github.event.number }}-${{ needs.deploy-preview.outputs.url_hash }}.web.app

pull_request checkout without ref: builds a synthesized merge

• pull_request-triggered jobs without an explicit ref: build a synthesized merge commit, not the PR head — GitHub Actions' default checkout target for a pull_request event is not the PR branch's head commit, it's an ephemeral, GitHub-synthesized merge commit (refs/pull/<PR>/merge, the PR head merged into the base branch's current tip), regenerated fresh on every run/re-run from whatever the base branch looks like at that moment. If two independent, non-overlapping line insertions land in the same block — one on the PR branch, one on the base branch, same day — git's merge produces no textual conflict and silently keeps both, even when that produces a semantically-duplicate key/property a downstream parser (TypeScript, YAML, etc.) then flags. The resulting CI failure cites a line/position that doesn't exist in either branch's own diff, so it reads as flaky infra or a phantom bug rather than what it is. Same silent-collision shape as the reusable-workflow env: duplicate-key gotcha above, different layer: git's merge — unlike GitHub's YAML mapping-key rejection — produces no error of its own, so the collision stays invisible until whatever downstream tool chokes on the duplicate. Concrete example (OBJ-2157, PR #2102): Codi's commit 9ee0e6128 on tech-debt/chip-destructive-variant added environment: 'jsdom', before the include: [...] line in design_system/vite.config.ts's test: {} block; the same day, an unrelated master commit dd12cdc5e (OBJ-2156, aria-label fix) added the identical line after include: [...]. Both are pure insertions relative to their common ancestor, so the merge kept both, and design-system.yml's validate-design-system job — actions/checkout@v6 with no explicit ref: — silently built that merge artifact, failing vue-tsc -b with error TS1117: An object literal cannot have multiple properties with the same name. at a line absent from the PR's own diff. Verified three ways: git merge-tree --write-tree <pr-head> origin/master reproduces it locally; git fetch origin pull/<PR>/merge && git show FETCH_HEAD:<file> reproduces it byte-for-byte; the failed run's own actions/checkout log line (HEAD is now at <merge-sha> Merge <pr-head> into <base-tip>) confirms what was actually built. Detection: if a CI failure cites a file/line that doesn't exist in the PR's own diff, fetch refs/pull/<PR>/merge directly and diff it against the PR head before assuming flaky infra. Fix: merge/rebase the PR branch onto current master and dedupe the colliding block — this also clears the underlying "N behind master" drift, so the next pull_request/merge checkout matches the branch head again.

Preview deploy swallows Firebase diagnostics under errexit

• Deploy Frontend Preview workflow swallows Firebase deploy diagnostics under errexit (OBJ-1336) — preview.yml's "Deploy to Firebase Preview Channel" step ran under GitHub Actions' default bash -e. FIREBASE_EXIT=$? sat directly on the line after the firebase-tools hosting:channel:deploy invocation with no set +e guarding it, so a non-zero exit tripped errexit and killed the script at the failing command — before FIREBASE_EXIT=$? was ever assigned and before the step's own stderr/deploy_output.json diagnostic dump could run. Every real deploy failure surfaced in CI as a bare "Process completed with exit code 1" with zero captured Firebase output (observed on PR #1443, run 29142565121, reproduced identically on manual re-run). Fix: wrap the deploy call in set +e / set -e so the capture and diagnostic dump are reachable — confirmed working against a real CI failure on the fix's own PR (Firebase HTTP 429: channel quota reached on PR #1445 printed its full deploy_output.json diagnostic instead of a bare exit code). Guardrail: ci.yml's lint-workflows job now has a "Guard against errexit swallowing $? exit-code captures" step that scans every .github/workflows/*.yml file for a bare VAR=$? capture not preceded by set +e and not inside a command || { ... } guard (the pattern already used correctly in preview_teardown.yml's "Delete Firebase Preview Channel" step) — this bug class can't land silently again — the guard resets its guarded/or_depth state at every step boundary (- name: / run: |), so a set +e left unclosed in one step can't mask an unguarded capture in a later step of the same file (OBJ-1338, PR #1447). If you add a VAR=$? capture anywhere in .github/workflows/, guard it with set +e/set -e or an || { } block. (Root cause: OBJ-1336, PR #1445.)

Preview teardown swallowed delete failures behind || echo

• Preview teardown swallowed cloud delete failures behind || echo, leaking resources for 3.5 months (OBJ-1450) — The pre-April preview_teardown.yml suffixed every gcloud/firebase-tools teardown call with an unconditional || echo "might already be deleted". A real delete failure was indistinguishable from a legitimate "already gone" no-op — the step, and the run, reported green either way. Between March and July 2026 this leaked ~20 orphaned Cloud Run resources (enkidu-api-pr-* services + enkidu-db-setup-pr-* jobs for PRs 252–296) with zero CI signal; discovered and cleaned up manually on 2026-07-17. Guardrail: ci.yml's lint-workflows job now has a "Guard against silently-swallowed cloud teardown failures" step that fails CI when a cloud resource teardown command (gcloud run services delete, gcloud run jobs delete, gcloud compute instances delete, gcloud sql instances delete, hosting:channel:delete) is suffixed with a same-line || echo, || true, or || : — same continuation-folding approach as the errexit guard above, so a backslash-continued command (the actual historical bug shape) is still caught. An || { ... } block that inspects the error before treating it as benign — the pattern already used correctly in preview_teardown.yml's "Delete Firebase Preview Channel" step — remains the sanctioned idiom and is not flagged; gcloud artifacts docker tags delete ... || true in production.yml is a best-effort re-tag for idempotency, not resource teardown, and is deliberately out of scope. If you add a new cloud teardown command anywhere in .github/workflows/, guard its failure path with an || { ... } block that inspects the error rather than swallowing it unconditionally. Known limitation (non-blocking): the guard's mid-command .* is unanchored to a single statement, so a backslash-continued logical line containing an unrelated command || true after a teardown call with no swallow of its own would false-positive — fails safe (over-flags), not a functional gap. (Root cause: OBJ-1450, PR #1578.)

continue-on-error needs a terminal re-raise step

• GitHub Actions continue-on-error requires a terminal re-raise step — Using continue-on-error: true on a job step (to allow artifact upload or post-failure cleanup) will cause the entire job to report green even on test failure unless you add a final step that checks if: steps.<id>.outcome == 'failure' and exits with code 1. Without this re-raise, the GitHub Actions job always exits 0 and masks the failure. Guard (OBJ-2481): production.yml's deploy-backend and staging.yml's deploy-backend/seed-staging-data jobs each had 5 continue-on-error steps (Reset Demo Account, Seed Goal Templates ×2, Provision Flag-Check Console Job) with no id/no terminal check — all fixed, and ci.yml's lint-workflows job now has a "Guard against continue-on-error steps missing a terminal outcome check" step scoped to those three jobs that fails CI if a new continue-on-error step there lacks either a terminal re-raise or the # ci-allow-continue-on-error: <reason> opt-out marker (same marker convention as the "unmarked continue-on-error in required-check jobs" guard below). Not yet extended repo-wide — see that guard step's own comments for the other files with differently-shaped, already-reviewed advisory continue-on-error usage.

cancel-in-progress: false alone isn't a FIFO queue — needs queue: max

• cancel-in-progress: false alone is not a durable FIFO queue — it needs queue: max — Without an explicit queue: key, a GitHub Actions concurrency: group defaults to queue: single, which only guarantees one pending (not-yet-started) run survives: the instant a newer run queues behind it in the same group, the older pending run is evicted, even though cancel-in-progress: false was set. cancel-in-progress: false only protects a run that's already executing; it does nothing for a run still waiting in the queue. Under this milestone's parallel-PR volume this starved a job indefinitely (PR #1414: the Preview Playwright smoke job was evicted 3+ times with steps: [], 0 steps ever executed, and re-running it didn't help — the re-queued run just lost the race again). Fix: add queue: max alongside cancel-in-progress: false to switch the group to GitHub's real FIFO queue (up to 100 pending runs), shipped in the 2026-05-07 concurrency-groups update. This is applied to the two remaining global smoke-lane concurrency groups — smoke-staging-demo (.github/workflows/staging.yml) and smoke-production-demo (.github/workflows/production.yml) — and enforced by a guard step ("Guard smoke-lane concurrency groups against pending-run cancellation", .github/workflows/ci.yml's lint-workflows job) that fails CI if either regresses to cancel-in-progress: false without queue: max. smoke-preview-lane no longer belongs to this pairing: Milestone v4.4 (OBJ-1322/OBJ-1326, per-PR smoke lane isolation) re-scoped it to smoke-preview-lane-${{ github.event.number }} per-PR with cancel-in-progress: true — with the group no longer global, no two PRs ever queue behind each other in it, so there's nothing left for queue: max to protect there; the lint-workflows guard's group-name match is unanchored specifically so it still finds this lane by its constant prefix without flagging it. See Preview Smoke Lane Isolation — decision record. If you add a new global cancel-in-progress: false concurrency group anywhere, apply the same queue: max pairing. (Root cause: OBJ-1291, PR #1414.)

A continue-on-error gate is an absent gate

• A continue-on-error gate is an absent gate, and it reads as coverage (resolved, OBJ-1734) — ci.yml's schema-drift job used to carry continue-on-error: true on both of its admin_dashboard steps ("Regenerate admin_dashboard generated types" and "Fail on admin_dashboard codegen drift"), each with a TODO to remove it. The effect: admin_dashboard/src/generated/graphql.ts went unregenerated from 2026-05-30 while its GraphQL documents kept changing, and master stayed green throughout. The staleness hid a real defect — UPDATE_USER_ROLES sent updateUserRoles(input: {...}) against an API that takes flat userId/roles arguments, so admin's role-update mutation was failing in production — found only when scripts/check-admin-schema-compat.mjs validated the same documents against the live schema. Same family as the continue-on-error-without-terminal-re-raise gotcha above: the job reported success, the dashboard read as covered, and nothing was enforced. Fix (shipped): PR #1773 regenerated the stale types, fixed the mutation, and removed both continue-on-error: true lines — the admin_dashboard schema-drift steps are hard gates today, no exception. ionic_frontend's matching pair got the same treatment shortly after (OBJ-1750, PR #1784), and the unmarked-continue-on-error guard below (OBJ-1736) now stops this class of TODO-flagged exception from recurring in any required job. If you add continue-on-error to a gate, the TODO to remove it needs an owner, or write no gate at all. (Root cause: OBJ-1734, PR #1773.)

CI guard against unmarked continue-on-error in required jobs

• CI guard against unmarked continue-on-error: true in required-check jobs (OBJ-1736) — A step or job carrying continue-on-error: true inside a job that backs a required status check produces a green check that verifies nothing: the check reports success no matter what the step/job actually did. This was the root cause of the PR #1773 incident (a TODO comment promising future removal of two continue-on-error: true lines in the ionic schema-drift job was not a mechanism — the schema-drift catch-up work, OBJ-1734/OBJ-1750, is what actually closed those lines). Guardrail (shipped): ci.yml's lint-workflows job has a "Guard against unmarked continue-on-error in required-check jobs" step that scans every .github/workflows/*.yml file for continue-on-error: true inside a required job, at either the step level or the job level (job-level — set directly under a job's runs-on:, before any step — is the more dangerous variant, since GitHub forces the whole job's conclusion to success regardless of any step failure). Required-job set is a hardcoded allowlist of job ids (rspec vitest vitest-admin schema-drift a11y cross-app-helper-sync posthog-flag-drift lint-workflows storybook-test frontend-lint), sourced from GitHub ruleset 17927598 ("master CI gate") and kept in sync manually, not via a live API read. Escape hatch: a step or job may keep the flag only if it also carries an inline opt-out marker comment, e.g. # ci-allow-continue-on-error: <reason> — no marker + the flag inside a required job fails CI with "Unmarked continue-on-error found in a required-check job — a required status check must not silently pass on a failed step or job." Job-boundary detection is scoped to only start tracking after a ^jobs:$ line is seen, so a same-named 2-space-indented key elsewhere in the file (e.g. under on:) can't spuriously toggle which job a hit is attributed to. If you add a new continue-on-error: true anywhere in .github/workflows/, either remove it, or add the marker comment with a real reason if the exception is intentional. (Root cause: PR #1773 incident, OBJ-1734. Guard: OBJ-1736/OBJ-1751, PR #1792.)

CI guard against bare github.event.number with no inputs.pr_number fallback

• CI guard against bare github.event.number missing an inputs.pr_number fallback (OBJ-2380) — A workflow that declares a workflow_dispatch trigger with a pr_number input promises manual dispatch as a real retrigger path, but github.event.number is only populated by pull_request events — it resolves to an empty string under workflow_dispatch, where only inputs.pr_number carries the PR number. A template expression that reads github.event.number with no || inputs.pr_number fallback silently breaks under manual dispatch: preview.yml had 5 such sites (Firebase channel deploy, concurrency group, preview URL reconstructed twice, UI-evidence artifact name) alongside 5 correct ones. The Firebase channel deploy site's failure mode wasn't just breakage — it deployed to a real, wrong channel literally named pr-. The only workaround before this fix was pushing an empty commit to force a synchronize event instead of using the dispatch input that already existed for this (surfaced diagnosing OBJ-2372, PR #2234's UI-evidence recapture). Guardrail (shipped): ci.yml's lint-workflows job has a "Guard against bare github.event.number missing an inputs.pr_number fallback" step that scans every .github/workflows/*.yml file, scoped to only files whose workflow_dispatch trigger itself declares a pr_number input (today, only preview.yml — playwright-smoke.yml also has a workflow_dispatch trigger, but its inputs are environment only, and it receives pr_number as a workflow_call input from its caller instead, so it's correctly out of scope for this specific guard), then fails CI on any real template expression (${{ ... }}) reading github.event.number without an || inputs.pr_number fallback on the same line — detection requires the actual expression opener so plain-text mentions in comments/::error:: messages documenting the fallback itself don't false-positive. Escape hatch: none — every github.event.number read in a pr_number-dispatchable workflow must carry the fallback; there's no legitimate reason to omit it. If you add a new workflow_dispatch trigger with a pr_number input, every github.event.number read in that file needs the || inputs.pr_number fallback, or the guard fails you. The Firebase deploy site additionally got a fail-fast ::error:: + exit 1 guard ahead of the deploy call, since that's the one site where an unresolved PR number causes a real wrong-target side effect rather than just a malformed downstream value. Caveat (OBJ-3703): "correctly out of scope" above is true only for this guard's specific variable (github.event.number/inputs.pr_number) — it does not generalize to github.event_name. playwright-smoke.yml turned out to have the sibling bug on that different variable; see the next entry. (Root cause: OBJ-2372 investigation. Guard: OBJ-2380, PR #2246.)

CI guard against github.event_name/github.event.* inferring a reusable workflow's own invocation kind

• CI guard against github.event_name/github.event.* inferring a reusable workflow's own invocation kind (OBJ-3703) — Same class of bug as the github.event.number gotcha above, on a different context variable: github.event_name reflects the top-level triggering event of the whole run, not whether this invocation of a reusable workflow is a direct workflow_dispatch or a workflow_call. playwright-smoke.yml's "Resolve URLs" and "Resolve smoke environment" steps branched on github.event_name to decide whether to trust their own workflow_call inputs or fall back to hardcoded URLs — but when caller preview.yml was itself manually re-run via workflow_dispatch (the documented mechanism for retriggering UI-evidence capture after a PR-body-only edit, since that edit doesn't fire synchronize), github.event_name read workflow_dispatch inside the callee even though it had received real workflow_call inputs from preview.yml. inputs.environment is empty on a workflow_call invocation, so the branch fell to else and hardcoded production URLs. visual-regression.yml has the identical dual-trigger shape and carried the same bug. Confirmed impact: Playwright Smoke Tests (Preview) failed on PR #3129 (OBJ-3688) with a misleading UNAUTHORIZED — job log showed api.objectuve.com/IS_EPHEMERAL_PREVIEW: false, GCP Cloud Logging showed zero staging /graphql traffic, and production's AuthenticateSessionToken#fallback_auth correctly rejected the staging-only smoke account. The same misroute also defeated the is_ephemeral_preview advisory-only escape hatch, since the ephemeral-preview URL pattern (*--pr-*.web.app) never matched the hardcoded production frontend URL either — the check hard-failed instead of warning. Fix (shipped): both files now key their branch off presence of one of their own workflow_call inputs instead — playwright-smoke.yml on [ -z "${{ inputs.api_url }}" ], visual-regression.yml on [ -z "${{ inputs.frontend_url }}" ] — since each is required: true with no default under workflow_call (always non-empty there) and never declared under workflow_dispatch (always empty there), unlike the run-level github.event_name. Guardrail (shipped): ci.yml's lint-workflows job has a "Guard against github.event_name/github.event.* inferring a reusable workflow's own invocation kind" step, sibling to the OBJ-2380 guard above, that fails CI on any workflow declaring both workflow_call: and workflow_dispatch: triggers whose body contains a real template expression (${{ ... }}) referencing github.event_name or github.event.* — detection requires the actual expression opener, so a plain-text comment mentioning the string doesn't false-positive on its own. If you add a new dual-trigger (workflow_call + workflow_dispatch) workflow, key any invocation-kind branch off presence of one of its own required workflow_call inputs, never off github.event_name/github.event.*. (Root cause: OBJ-3688 investigation, PR #3129. Guard: OBJ-3703, PR #3137.)

Force-pushing a PR branch 3+ times can break CI triggers

• Force-pushing a PR branch 3+ times rapidly can permanently break GitHub Actions CI triggers — Pushes after the third are silently dropped; CI never fires and the branch stays yellow with no error. Fix: push an empty commit (git commit --allow-empty -m "retrigger CI") or close and reopen the PR. See docs/operations/github-actions-force-push-gotcha.md.

Force-push iteration: use a throwaway local branch

• Force-push iteration: work on a throwaway local branch; force-push to the PR branch only once when ready — Repeated force-pushes during iteration (fixups, rebases mid-review) burn the three-push budget. Accumulate all changes locally or on a scratch branch, then do a single force-push when the work is stable. See docs/operations/github-actions-force-push-gotcha.md.

ci.yml's branch allowlist must never pin a milestone version

• ci.yml's pull_request.branches allowlist must never enumerate specific milestone-branch versions (e.g. 'gsd/v2.*') — The main CI workflow (RSpec/Rubocop, Vitest/ESLint, a11y, Cypress E2E, schema-drift, PostHog flag-drift) only runs on PRs whose base branch matches its pull_request.branches filter. Every milestone lives on a gsd/vX.Y-name branch, and the allowlist had been patched with an exact version glob per milestone round (gsd/v1.10-*, then gsd/v2.*) — so it silently stopped matching the moment v3.x/v4.x branches shipped, and no error surfaced: preview.yml/playwright-smoke.yml have no branch filter at all, so those checks still went green, making the PR look CI-clean while RSpec/Rubocop/ESLint/vue-tsc/coverage/posthog-flag-drift never ran. This recurred twice (a stale chore/ci-allow-gsd-milestone-prs branch shows the same v1.10-only patch predating even the v2.* glob) before being fixed for good in OBJ-1145/PR #1271's CI-config follow-up. Fix (the actual guardrail): branches: [master, 'gsd/*', 'feat/*'] — a single version-agnostic glob that matches every gsd/vX.Y-* milestone branch (and any other single-level gsd/* branch) forever. Do not narrow this back to a version-specific pattern for a "cleaner" allowlist; if a future milestone truly needs an exception, add it as an additional entry, never as a replacement for the general gsd/* glob. When adding a new CI-gated workflow, always ask whether its trigger should be scoped by branch name at all — preview.yml's no-filter approach is why it never went stale.

Splitting a required-check job silently drops its gate strength

• Splitting a required-check job silently drops its gate strength until the ruleset is updated to match (resolved) — OBJ-1942 split ci.yml's combined ~12m vitest job (ESLint + npm audit + guards + Vitest) into two parallel jobs: frontend-lint (new) carries ESLint/audit/guards, vitest keeps its exact job id/name and now runs only the coverage-critical unit tests. GitHub ruleset 17927598 ("master CI gate") required-status-checks initially only listed "Vitest + ESLint (ionic_frontend)" (vitest's display name, deliberately unchanged so the existing required-check reference kept resolving) — "Frontend Lint + Guards (ionic_frontend)" (frontend-lint's display name) was missing from that list, so a failing ESLint/npm audit/button-border/detached-wait check reported red on the PR but did not block merge, a real if narrow gate-strength regression versus the single combined job it replaced. Status: fixed. "Frontend Lint + Guards (ionic_frontend)" is now present in ruleset 17927598's required_status_checks alongside "Vitest + ESLint (ionic_frontend)" (confirmed via gh api repos/objectuve-softworks/enkidu/rulesets/17927598 on 2026-09-07) — both jobs are required checks today. General rule (covers renaming too, not just splitting): the ruleset matches required checks on the job's name: display string (a check-run context), not its job id — splitting a required-check job into two, or renaming one job's name:, always needs a paired ruleset update (gh api repos/objectuve-softworks/enkidu/rulesets/17927598 to verify) in the same change, or land it same-day — a required check that silently stops covering what it used to cover is the same failure shape as an unmarked continue-on-error: true (gotcha above), just introduced by a job split or rename instead of a flag. (Root cause: OBJ-1942, PRs #1911/#1912/#1913. Ruleset gap closed some time after 2026-07-31; confirmed closed 2026-09-07.)
• Rename addendum (OBJ-3485, won't-fix) — SUPERSEDED, historical only. The rename it declined was later made anyway by PR #2976 without the paired ruleset update, producing exactly the outage this addendum warned about; see the OBJ-3613 entry below for what actually happened and the guards now in place. vitest's name: reads "Vitest (ionic_frontend)" today. Preserved as written for the record: vitest's name: still reads "Vitest + ESLint (ionic_frontend)" after OBJ-1942 moved ESLint out to frontend-lint — cosmetically misleading, but renaming it alone would strand "Vitest + ESLint (ionic_frontend)" as a required context no job ever reports again, blocking every future PR to master. OBJ-3485 confirmed this against the live ruleset (2026-09-07) and declined the rename: not worth a live branch-protection mutation for a low-priority cosmetic fix. Safe rename sequence, if ever wanted (unavoidably has a brief unattended-gate window — do it attended, by a repo admin, and fast):
  1. Remove the old context (e.g. "Vitest + ESLint (ionic_frontend)") from ruleset 17927598's required_status_checks. Gate is now open on that job — this is a stopwatch, not a stopping point.
  2. Merge the ci.yml rename PR — pre-approved and green before step 1.
  3. Add the new context (e.g. "Vitest (ionic_frontend)") to the ruleset; confirm via gh api repos/objectuve-softworks/enkidu/rulesets/17927598.
  4. Open a throwaway PR and confirm the new context appears as required.

A job rename can orphan a still-required status-check context with no error (OBJ-3613)

• A job rename can orphan a still-required status-check context, leaving every PR permanently BLOCKED on "Expected — waiting for status" with no error surfaced (OBJ-3613, resolved) — Commit 43d05ad80 (PR #2976) renamed the vitest job's name: display string from "Vitest + ESLint (ionic_frontend)" to "Vitest (ionic_frontend)" as part of the OBJ-1942 job split documented in the entry above, but ruleset 17927598's ("master CI gate") required_status_checks list still named the old string. Because no job ever emitted that context again, the ruleset had nothing to reconcile against — same failure shape as the workflow-level paths: filter gotcha below, but caused by a job rename instead of a trigger scope, and here the check context wasn't just missing from one PR, it was permanently gone for all of them. Every open PR sat BLOCKED for ~2 days before anyone noticed; Josh fixed the ruleset by hand (repo-admin action, 2026-09-10). Guardrail (shipped): scripts/check-ruleset-required-contexts.mjs (+ .test.mjs), wired as two steps in ci.yml's lint-workflows job, diffs a hand-maintained snapshot of ruleset 17927598's 14 required contexts against every job name: emitted across .github/workflows/*.yml and fails with the exact orphaned-context name if any required context has no matching job. Same hardcoded-snapshot constraint as the continue-on-error required-jobs guard above: GitHub's ruleset-read API needs the "Administration" repo permission, unavailable to the default GITHUB_TOKEN, so REQUIRED_CONTEXTS in the script must be updated by hand (gh api repos/objectuve-softworks/enkidu/rulesets/17927598 run by a repo admin) whenever the ruleset's required-context list changes. That PR-time guard catches only the OBJ-3613 direction — a job rename/removal orphaning a still-required context — not the reverse (a ruleset edit made directly in the GitHub UI, independent of any workflow-file change). .github/workflows/required-status-checks-drift.yml (OBJ-3613) closes that reverse half: a daily scheduled job that reads the live ruleset with RELEASE_TOKEN (the one credential already provisioned that can hold Administration read) and diffs it against the same REQUIRED_CONTEXTS snapshot, mirroring clerk-auth-config-drift.yml's file-or-close-a-tracking-issue pattern and distinguishing an API/auth/network failure from real drift so a transient outage is never misreported. Caveat — this one check can go dormant silently: if RELEASE_TOKEN is ever revoked or unset, the live-check job warns and skips, and the scheduled run still ends green; a wrong-scope token fails loud (result=api_failure), but a fully absent one does not. The PR-time guard is unaffected either way. Second caveat — an emptied or truncated REQUIRED_CONTEXTS would pass vacuously, so the guard carries a fail-closed floor: findMissingContexts filters the required list against emitted job names, so a list accidentally trimmed to [] has nothing missing and would print ✓ and exit 0. MIN_REQUIRED_CONTEXTS = 10 (PR #3067) closes that window: main() checks the snapshot's length before the comparison and exits 1 if it has fallen below the floor, treating a short list as a truncated/corrupted edit rather than a legitimate ruleset shrink. The floor is a corruption tripwire, not a count to keep in sync with the live ruleset (currently 14) — raise it only if the real required-context list ever drops near it. The daily drift cron remains the complementary control for the other direction (an out-of-band ruleset edit). (Root cause: OBJ-3613/commit 43d05ad80, PR #2976. Guards: OBJ-3615, PR #3063; drift cron, PR #3067.)

A required-status-check workflow must never gate on a workflow-level paths: filter

• A required-status-check's workflow must never gate on a workflow-level on.pull_request.paths filter — GitHub only creates a check-run context for a workflow once it actually triggers; a paths: filter under on.pull_request means the workflow (and every job in it) never runs at all on a non-matching PR, so the check context is never created, and a PR requiring that check gets stuck on "Expected — waiting for status" forever, since GitHub has nothing to reconcile against. mobile-ios.yml hit exactly this: its workflow-level paths: ['ionic_frontend/ios/**', 'ionic_frontend/fastlane/**'] meant ios-signing-check/ios-widget-extension-build/ios-widget-snapshots (including "Widget Snapshot Tests") never posted a status at all on a PR that didn't touch those paths — structurally incompatible with ever adding them as required checks. The safe pattern (ci.yml's detect-cypress-changes, preview.yml's detect-app-changes, and now mobile-ios.yml's detect-ios-changes): no workflow-level paths: filter — the workflow triggers on every PR — plus an internal detect-*-changes job that diffs origin/<base>...HEAD and gates the downstream jobs via if: needs.detect-*-changes.outputs.*_changed == 'true', so the check context always reports (success/failure/skipped) even when nothing relevant changed. When adding a new path-scoped PR check that might later become required, build it on this pattern from the start rather than a workflow-level paths: filter. (Root cause/fix: OBJ-3294, PR #2859.)

validate-secrets.sh never runs on PRs — only at deploy time

• validate-secrets.sh is a deploy-time-only static check — it can be green on every required PR check and still guarantee a broken merge, because it never runs on pull_request — scripts/validate-secrets.sh (the round-1 gate referenced in the stranded-milestone-branch and PR-scope gotchas above) only runs as part of the staging/production deploy workflows, not as a PR check. A bare ENV.fetch('STRAVA_CLIENT_ID') with no default and no provisioned secret passed every required PR check on the v4.14 Activity Nexus promotion (OBJ-1982, PR #2003) — RSpec, Rubocop, Vitest, pr-scope-guard, all green — and then failed the very next master deploy, since the script only runs once the merge has already landed. Flag-off does not make a branch safe to merge — connected_apps_enabled being 0%/fail-closed means the code path is unreachable at request time, but it says nothing about whether the deploy itself succeeds; a missing required secret with no default breaks the deploy pipeline for the whole team regardless of any feature flag. Fix applied (temporary): the three affected ENV.fetch calls in rails_api/lib/integration_provider/strava.rb were given nil defaults, marked with an inline # OBJ-1982 comment noting they must revert to a bare ENV.fetch once the real secrets are provisioned (restoring validate-secrets.sh's zero-presence protection for these vars like every other required secret). Follow-up (shipped, OBJ-2073, PR #2021): validate-secrets.sh now runs on pull_request too, via the new validate-secrets job in ci.yml — path-gated to rails_api/app/, rails_api/lib/, deploy/, .github/workflows/{staging,production}.yml, and scripts/validate-secrets.sh itself, reusing the same proven origin/<base>...HEAD fetch pattern as the existing detect-cypress-changes job. Correction (2026-09-09, OBJ-3419): this entry previously said the script "never runs on PRs — only at deploy time," which is now stale — the trigger described above has existed since OBJ-2073. What's actually still true, and the real gap: that PR-time trigger's coverage, not its existence, stops at validate-secrets.sh's own candidate set (bare ENV.fetch('X')/ENV['X'] under rails_api/) — it was never designed to catch, and still doesn't catch, a secret referenced only as a GitHub Actions workflow-step env: value (see the next entry). (Root cause: OBJ-1982 attempt 1, PR #2003 reverted. Follow-up: OBJ-2073, PR #2021.)

A fourth blind spot: a secret consumed only as a workflow-step env: value

• A secrets.X reference in a workflow step's env: block is invisible to every guard that predates it, for a fourth structurally distinct reason — none of the previous three guards read .github/workflows/ at all. PR #2997 (OBJ-3419) added a staging.yml step calling gcp_setup/16_cloud_run_startup_probe_alerts.sh with NOTIFICATION_CHANNEL_ID: ${{ secrets.MONITORING_NOTIFICATION_CHANNEL_ID_STAGING }}. That secret was never provisioned. The script correctly exit 1s on an empty NOTIFICATION_CHANNEL_ID (see the comment now at gcp_setup/16_cloud_run_startup_probe_alerts.sh near the guard — the loud failure was never the bug), so every master push red-lined the staging deploy and paged Slack for roughly 24 hours before anyone provisioned the secret. This is the same recurring shape as Three separate guards missed the same unwired secret, each for a structurally different reason (OBJ-2894) — a fourth guard-blind-spot, structurally distinct from all three there:

  1. scripts/check-deploy-secret-drift.mjs parses exactly six Cloud Run deploy manifests (deploy/{service,worker,crono}.{staging,production}.yaml) — it never reads .github/workflows/ at all, so a secret referenced only in a workflow step's env: block never enters its candidate set by construction, the same way a var wired into zero manifests was invisible to it in the OBJ-2894 incident.
  2. scripts/validate-secrets.sh's OBJ-1340 candidate set is ENV.fetch('X') / ENV['X'] greps under rails_api/app and rails_api/lib — a Ruby-source-only detector. grep -rn MONITORING_NOTIFICATION_CHANNEL_ID rails_api/ returns zero hits, because the value never reaches Rails at all; it's consumed entirely inside a bash script invoked from the workflow step, one layer outside anything this script was ever built to scan.
  3. Neither guard runs against .github/workflows/*.yml's own text, so a workflow author adding a brand-new secrets.SOME_NEW_NAME reference to any step had (before this fix) zero PR-time signal that the name existed on GitHub as a real secret, versus a typo or an unprovisioned placeholder — the only feedback was the consuming step failing at run time, which for a step that runs unattended on every push (unlike a one-off manual bootstrap script) means every push fails identically until a human notices and provisions it.

  Fix (shipped, OBJ-3419): scripts/check-workflow-secret-refs.mjs parses every tracked .github/workflows/*.yml/*.yaml file for secrets.NAME and secrets['NAME']/secrets["NAME"] references (excluding the auto-provided GITHUB_TOKEN) and fails the build on any name not listed in the checked-in .github/provisioned-secrets.json manifest's provisioned or known_absent arrays. Wired into ci.yml's lint-workflows job (no path filter, no continue-on-error) as a gating, not informational, check — deliberately not a live gh secret list call in that gating path, because the default PR-scoped GITHUB_TOKEN can't list Actions secrets, an org-scoped secret 403s even a maintainer's own non-org-admin PAT (gh api orgs/objectuve-softworks/actions/secrets), and a fork PR gets zero secrets regardless — any of those would make a live check either fail open (an absent-gate, per the continue-on-error gotcha) or false-positive on a legitimately working org secret. A live reconcile against gh secret list is available as the separate --reconcile flag, explicitly non-gating (never changes the exit code, prints an UNVERIFIED line naming what it structurally cannot check). At ship time, 8 secrets referenced on master were already unprovisioned with no incident yet reported (ACHIEVEMENTS_EMPTY_EMAIL/ACHIEVEMENTS_POPULATED_EMAIL/CLERK_SECRET_KEY_PROD in playwright-smoke.yml, CODECOV_TOKEN in ci.yml, ITC_TEAM_ID in mobile-ios.yml, RAILS_WEBHOOK_SECRET in load-test.yml, SENTRY_ISSUES_API_TOKEN in staging.yml, SLACK_BUDGET_ALERT_WEBHOOK_URL in scheduled-release.yml) — these were baselined into known_absent (with a real reason and consuming line each) so the guard could ship without red-lining every PR on day one; they remain a live, tracked backlog, not an approval to leave them unprovisioned indefinitely. General rule: this is now the fourth time a secret-provisioning gap has slipped past every existing guard because each guard's candidate set was scoped to one syntax/directory/file-type and none of them covered workflow-step env: values — when adding a new secret-related guard, name explicitly which surfaces it does not cover, the same way this entry and its three OBJ-2894 siblings do, rather than letting "we have secret guards" read as "every secret-consuming surface is covered." (Root cause/incident: OBJ-3419, PR #2997. Fix: scripts/check-workflow-secret-refs.mjs, .github/provisioned-secrets.json.)

Three separate guards missed the same unwired secret, each for a structurally different reason

• An unwired env var can pass three independently-designed guards at once, each blind to it for a different structural reason — none of them wrong on their own terms, and none of them overlapping with what would actually have caught it — PR #2567 (47782f871, merged 2026-08-22, OBJ-2822) added rails_api/config/environments/staging.rb:19-23's ENV['MAILTRAP_SANDBOX_SMTP_USERNAME'] / ENV['MAILTRAP_SANDBOX_SMTP_PASSWORD'] bracket-access references, wired into zero deploy manifests and zero workflows. All three of this repo's existing secret-related guards passed the PR anyway:

  1. scripts/validate-secrets.sh's OBJ-1340 zero-presence check — wrong directory, wrong syntax. collect_bare_env_fetches()'s BARE_FETCH_DIRS=( rails_api/app rails_api/lib ) (scripts/validate-secrets.sh:114) never scans rails_api/config, and its detector regex ENV\.fetch\(['"][A-Z][A-Z0-9_]*['"]\) (scripts/validate-secrets.sh:401) only matches the bare-.fetch call shape — it has no pattern for ENV['X']/ENV["X"] bracket access at all. The staging.rb reference is wrong on both axes the check covers, so it never entered the check's candidate set. Bracket access is arguably the higher-risk of the two missed shapes — it returns nil silently instead of raising KeyError, the same silent-nil risk already named in the if ENV['X'] truthiness-guard gotcha above.
  2. rails_api/spec/config/staging_mailer_spec.rb — asserts on source text, not on wiring. Its second example (staging_mailer_spec.rb:18-20) does expect(staging_source).to include('MAILTRAP_SANDBOX_SMTP_USERNAME') — a grep against the config file's own source, added in the same PR as a regression guard for the previous bug (staging inheriting production's live-SMTP block wholesale). A source-text assertion can never detect an unwired env var: it passes precisely because the name exists in the file, regardless of whether anything downstream ever populates it. It was never designed to catch this failure mode and structurally cannot.
  3. scripts/check-deploy-secret-drift.mjs — compares staging against production, not either against "wired at all". findLiteralDrift() (scripts/check-deploy-secret-drift.mjs:127) flags an env var whose secret ref or literal value is identical on both sides of a staging/production pair — built to catch staging accidentally sharing a live production credential (OBJ-1225) or resource name (OBJ-1341). A var absent from both environments' manifests never enters either side's map, so it's invisible to a same-vs-different diff by construction — the tool's entire design assumes both sides define the var and asks whether they collide, not whether either side defines it.

  The gap sat live for two days until staging's Email::DeliveryCanarySendJob actually exercised the SMTP path on the first Crono tick after the merge (2026-08-23T07:00Z) and raised ArgumentError: SMTP-AUTH requested but missing user name (OBJ-2893, full diagnosis on that issue's comment thread). This is the same "detector matches one syntax/location/comparison, misses the equivalent-risk sibling" shape already documented in this file for ?? vs || and jq -r vs // empty — just spanning three separate tools instead of two variants of one. Fixes tracked separately, not part of this doc entry: OBJ-2894 (Codi) extends validate-secrets.sh's BARE_FETCH_DIRS to include rails_api/config and its regex to match bracket access — a pure detection-tooling change with no secret dependency. The staging_mailer_spec.rb rewrite (assert the staging deploy manifests carry both var names, not that the string exists in the config file) and the check-deploy-secret-drift.mjs allowlist entry (once the secrets are wired staging-only, the drift check will correctly flag them as staging/production asymmetric — same shape as the existing EMAIL_CANARY_RECIPIENT allowlist entry) are bundled into the secret-provisioning + wiring fix tracked on OBJ-2893, since both depend on the secrets actually being wired first. General rule: "a guard exists for this class of risk" is not the same claim as "this guard would catch this specific failure." Before trusting a green check as coverage, read what it actually asserts — string presence in source, cross-environment identity, or real wiring — against the specific failure shape in front of you; three guards can each correctly do their stated job and still jointly miss a var that's wired nowhere at all. (Root cause: OBJ-2893, Dave + Josh's diagnosis. Fixes: OBJ-2894 for validate-secrets.sh; the spec rewrite and drift-allowlist entry ride the OBJ-2893 wiring fix.)

firebase-tools steps need a checkout step or they fail every time

• Teardown PR Preview fails on every PR — firebase-tools needs a checkout step, unconditionally, for every command, even hosting:channel:delete with an explicit --site/--project — preview_teardown.yml's teardown-preview job never had an actions/checkout step; it went straight from job setup into Authenticate to Google Cloud and then npx firebase-tools hosting:channel:delete. firebase-tools requires firebase.json in the working directory for every invocation (requireConfig/Config.load runs synchronously, unconditionally, before any network call), so with an empty $GITHUB_WORKSPACE the Delete Firebase Design System Preview Channel step failed fast (~2s) with Error: Not in a Firebase app directory (could not locate firebase.json) on every PR close. Masked twin bug: the earlier Delete Firebase Preview Channel step (enkidu-app) hit the identical missing-config error, but its own grep -qi "not found\|does not exist" catch happened to match npm's install-phase stderr noise on a cold npx cache — so it reported green while the enkidu-app preview channel was, in fact, never being torn down either (same shape as the OBJ-1450 silently-swallowed-teardown gotcha above, via a false-positive catch instead of an unconditional || echo). Fix (shipped): added actions/checkout@v6 as the job's first step (.github/workflows/preview_teardown.yml:26) — firebase.json lives at repo root, so a plain checkout covers both delete steps in the job, no working-directory: needed. Guardrail (shipped): ci.yml's lint-workflows job has a "Guard against firebase-tools steps missing a preceding checkout" step (.github/workflows/ci.yml:402-448) that scans every workflow for a npx firebase-tools invocation with no actions/checkout step earlier in the same job; verified clean against every other firebase-tools-invoking workflow in the repo. If you add a new job that calls firebase-tools, give it a checkout step before the first invocation, or the guard fails you. (Root cause/fix: OBJ-2261, PR #2165.)

A watch job's real coverage window starts when its needs: chain finishes, not at deploy

• A polling job gated behind a slow needs: chain covers [chain_duration, chain_duration + window], not [0, window] — read the deadline off the code, not off the constant — staging.yml's release-health-watch job only starts once its needs: chain (build → migrate → deploy → smoke) finishes; before OBJ-2267 its poll loop was a hardcoded seq 1 7 × sleep 300 measured from job start, not deploy time. RELEASE_HEALTH_WINDOW_MINUTES (30 min default) named a [0, 30]-minute contract that the code never actually enforced — real coverage was an emergent [~31, ~61]-minute window, entirely dependent on how long the smoke suite happened to take that day. Three independent readings of the code (Codi, Roy, and Orion on first pass, while reviewing OBJ-2264/OBJ-2267) modeled it as [0, 30] and concluded a recurrence of the OBJECTUVE-CLIENT-B8 incident's real ~56-minute discovery time (OBJ-1982) would have exited the gate clean, 26 minutes before B8 crossed the event threshold — false; deploy+56 sat inside the actual [31, 61] window and would have matched. Fix (OBJ-2267): the poll deadline is now anchored to the workflow run's own run_started_at (via gh api), not to job-start or upstream chain duration, so coverage is a stated deploy+0 → deploy+RELEASE_HEALTH_WINDOW_MINUTES contract (default raised 30→90) independent of needs: chain speed. A job that starts after its own deadline (queue backlog) logs ::warning:: naming the unwatched release and exits 0 rather than reading as silently clean; a workflow-level assertion after the poll loop fires ::error:: + exits 1 if it ever exits clean without a poll reaching the deadline — the "coverage was actually enforced" property is checked, not just documented. General rule: for any polling/watch job with a needs: dependency, always ask whether the window constant in the code and the actual wall-clock coverage are the same thing — an emergent property that happens to look right today silently shrinks the day the upstream chain gets faster. (Root cause: OBJ-2264's header comment first mismodeled this; corrected then re-diagnosed and fixed under OBJ-2267, PR #2334.)

guard_stale's skip-on-stale run reads identically to a real pass, and can misdirect a CI bisection

• A guard_stale-skipped deploy run reports the same all-green run-list appearance as a genuine pass — "skipped" and "verified and passed" are visually indistinguishable at a glance, and that ambiguity can misdirect someone bisecting "which commit is CI red on." staging.yml's guard_stale step (see Staging deploy race for the full mechanism) correctly no-ops a deploy when a newer commit has already superseded it on origin/master — the right behavior, since deploying a stale build would overwrite a newer one. But the run itself still finishes with every job green, Deploy to Cloud Run: skipped reading the same as any other conditionally-skipped step in the UI, with no distinct visual treatment marking the skip as "this run never actually verified anything real." During the 2026-09-06 ec33980f/OBJ-3424 puma.rb incident, ec33980f's own Staging Deployment run got stale-skipped (a newer commit had already landed), so the run that carried the actual regression read as fully green — the real first deploy attempt landed on the next commit in the queue instead, a docs-only PR that got investigated and correctly ruled out first, costing roughly two hours before the trace led back to ec33980f. This is a distinct failure mode from every other guard_stale entry in this doc (completion-order races, mid-flight interruption, validation-vs-mutation races) — those are about the guard failing to prevent a bad deploy; this one is about a correct skip being visually unreadable as "this commit's own run proved nothing," which matters specifically when someone is scanning run history to attribute a regression to a commit. Not yet fixed: no distinct badge or ::notice::-level run summary currently marks a stale-skip run as distinct from a real pass in the run list — tracked as OBJ-3545 (Codi/Tess-shaped: a job-summary annotation on guard_stale's skip path, so it doesn't read as verification). (Incident: OBJ-3424/ec33980f, PR #2947. Guard mechanism: OBJ-1144/OBJ-1992/OBJ-2751, see the "Staging deploy race" entries above. Follow-up: OBJ-3545.)

• Addendum (OBJ-3879, 2026-09-16) — the OBJ-3545/OBJ-3597 fix closes the single-run read, not the multi-run scan a release preflight actually performs, and a brand-new spec can go from merge to release-blocking while reading clean the entire time. OBJ-3597 shipped smoke-result-summary (.github/workflows/staging.yml:1243-1265), a per-run Step Summary annotation that correctly distinguishes a stale-skip from a real pass — but nothing aggregates that annotation across runs. Both Vicki's 2026-09-16 release-train preflight and Maggie's routing instead read gh run list-style conclusion fields across recent runs and concluded "staging is green," which is exactly the ambiguity this doc's base entry describes, just hit by tooling instead of a human bisecting by eye. Dave's diagnosis (checkpoint dave/obj3879-2026-09-16-guard-stale-misattribution) walked every staging.yml run backward from origin/master's tip and found the real picture: 8 consecutive real (non-stale-skipped) Playwright runs over 7+ hours all failed with the same two-spec-file signature, while roughly 20 runs interleaved among them were stale-skipped and displayed identically green in the run list — including the specific run (0fb55a45b) the preflight cited as "last fully green," which had never actually executed Playwright at all. New wrinkle this incident surfaced: a spec file can go from merge to "blocking a release" without ever having had a real passing run in between. specs/dashboard/offline-cold-start.spec.ts (OBJ-3868) was added in commit a0444df2c; its first real (non-stale-skipped) staging run (c9c1ec37f) already failed both of its cases, yet high deploy velocity meant most runs landing after its merge were stale-skipped, not passed — so the spec read as clean in every surface a human or preflight tool would normally check, right up until it blocked the 2026-09-16 train. Confirmed second occurrence (2026-09-17): the identical gap recurred one calendar day later against a different spec — specs/platform/pwa/02-offline-cold-start.spec.ts (OBJ-3853/PR #3227, merged 8f6ae5762 2026-09-16 11:45 UTC; reverted by PR #3291, so the file no longer exists on master) failed on every real staging run since merge, for the same underlying reason: no CI mechanism distinguishes a new spec's first real run from a regression, and PR #3227's own body confirms its CI never ran the new spec against a live environment (Preview/Playwright-Preview jobs skip on a test-infra-only diff). Dave's diagnosis (OBJ-3913, checkpoint dave/obj3913-2026-09-17-flags-route-and-staleness-stamp) found the failure itself was two independent spec-authoring defects, not a product regression — but the mechanical gap this addendum describes is exactly what let it reach release-blocking status undetected either way, with no guard shipped in the one day between occurrences. The spec was reverted (PR #3291, merged 8f33bbee1 2026-09-17T13:40:02Z) independent of this diagnosis, so no live blocker remains — but this is now a confirmed recurrence, not just an open risk. Shipped (OBJ-3917): scripts/check-smoke-spec-provenance.mjs is the "last real (non-stale-skip) Playwright result" query/annotation this addendum called for. It derives a green watermark from real (non-guard_stale-skipped) staging.yml push runs and classifies every spec as proven (introduced before the watermark), unproven (introduced after — never had a real passing run), or unknown (fail-closed default when the watermark walk is exhausted). Wired into playwright-smoke.yml's Step Summary, staging.yml's notify-smoke-failure Slack alert, and a report subcommand that release preflight reads directly instead of raw gh run list conclusions. See Spec provenance: first real run vs. regression.

A job failure invisible to both of staging.yml's alert jobs is a recurring shape, not a one-off

• A job falling through both of staging.yml's Slack alert jobs is a recurring shape, not a one-off — four occurrences, same root cause each time — staging.yml has two alert jobs: notify-smoke-failure (staging.yml:952, fires on needs.playwright-smoke.result == 'failure') and notify-deploy-failure (staging.yml:988, fires when any of its needs: jobs report failure). A job added to the critical path between them isn't automatically covered by either — it has to be explicitly added to notify-deploy-failure's needs:/if:, or its own downstream job has to always()-propagate its result so notify-smoke-failure can see it. Four jobs have fallen through this exact gap: deploy-backend/deploy-litellm (OBJ-3164 — a transient regional Cloud Run network-attach failure sat unrouted for 87+ minutes, only surfaced by Dave's external /health-polling autopilot), playwright-smoke itself (OBJ-2147 — smoke's own red run was a post-deploy informational job with no alerting at all, letting a known-broken assertion ship to production undetected for ~24h), seed-staging-data (OBJ-3492 — its terminal re-raise step correctly flipped the job to failure, but notify-deploy-failure's needs: didn't watch it, and playwright-smoke, which depends on it, came back skipped rather than failure upstream, so notify-smoke-failure didn't fire either), and provision-cloudsql-alert-policy (OBJ-3503 — added by 2add3494d; its PERMISSION_DENIED failure was visible only in the GitHub Actions UI from the day it merged until a human, not an alert, caught it during unrelated ship verification on OBJ-3492). All four fixes shipped the same shape: add the new job to notify-deploy-failure's needs:, if:, and message-building (see the OBJ-3164/OBJ-3492/OBJ-3503 comments at staging.yml:991, staging.yml:1000, and staging.yml:1006). General rule: any job added to staging.yml's critical path needs an alert-routing check as part of its own PR — don't wait for the next job downstream to fall through the same gap and rediscover it. Prose alone hasn't worked — this general rule has now been restated after each of three prior instances (OBJ-3164, OBJ-2147, OBJ-3492) and still missed a fourth job. The mechanical check that would actually close this (not yet built, per Dave's OBJ-3503 diagnosis): a CI lint/actionlint step that fails whenever a job name appears on staging.yml's critical path (i.e. deploy-*/seed-*/provision-*-shaped jobs gating production readiness) but is absent from notify-deploy-failure.needs, so a missing wire-up fails the adding PR instead of waiting to be rediscovered by an operator. (Root cause/fixes: OBJ-3164, OBJ-2147, OBJ-3492, OBJ-3503.)

An alert job's needs/if must cover the full transitive needs chain, not just the named jobs' own result

• An alert job's needs:/if: must cover the full transitive needs chain of the jobs it watches — checking only the named jobs' own == 'failure' result misses the case where a named job resolves to skipped because its OWN upstream dependency failed — GitHub Actions resolves a job's result to skipped (not failure) whenever a job it needs: doesn't succeed, even though the skipped job never ran and never had the chance to fail itself. An alert if: that only checks needs.<job>.result == 'failure' for the jobs directly named in its own needs: list is blind to a failure one hop further upstream, because that failure surfaces downstream as skipped, not failure. This is a narrower, more mechanical version of the "job falls through both alert jobs" gap documented above — it has already recurred twice on staging.yml's notify-deploy-failure job (staging.yml:988, if: at staging.yml:1010) alone:

  • Direct-failure case (OBJ-3164): deploy-backend/deploy-litellm themselves failed; checking their own .result == 'failure' was sufficient because nothing further upstream skipped them.
  • One-hop-upstream case (OBJ-3500, PR #2986): migrate failing (OBJ-3425, PERMISSION_DENIED) resolves deploy-backend/deploy-litellm — both declare needs: migrate — to skipped, not failure. notify-deploy-failure's if: only checked deploy-backend/deploy-litellm's own result, so 15+ consecutive staging.yml runs (2026-09-06T20:00Z–2026-09-07T14:25Z) had migrate fail with zero Slack alerts, caught only ~19h later by the staging health-check autopilot.

  Fix pattern (Tess, commit fee8b85d1, PR #2986 — open, not yet merged as of this writing): add the upstream job to the alert job's own needs: list, and OR its .result == 'failure' into the if: alongside the jobs already watched:

     notify-deploy-failure:
  -    needs: [deploy-backend, deploy-litellm]
  +    needs: [migrate, deploy-backend, deploy-litellm]
  -    if: always() && (needs.deploy-backend.result == 'failure' || needs.deploy-litellm.result == 'failure')
  +    if: always() && (needs.migrate.result == 'failure' || needs.deploy-backend.result == 'failure' || needs.deploy-litellm.result == 'failure')

  General rule: when wiring a new job into an alert job's needs:/if: (per the gotcha above), don't stop at the job named on the critical path — walk that job's own needs: chain one level at a time and watch every job whose failure could resolve it to skipped, not just its own failure result. Tess deliberately didn't build a generic graph-walking guard workflow for this class of gap on OBJ-3500 (flagged as low-priority for a one-file fix); a third recurrence is the signal to revisit that call. (Root cause/fixes: OBJ-3164, OBJ-3500.)

A | tee pipeline under default bash -e {0} swallows the piped command's exit code

• A | tee pipeline under GitHub Actions' default unspecified shell (bash -e {0}) reports tee's exit code, not the piped command's — a real crash still reads as steps.<id>.outcome == 'success' (OBJ-2521) — GitHub Actions' default shell: for a run: block resolves to bash -e {0} — -e (errexit) only, no pipefail. Under that shell, a pipeline's exit code is the last command's; tee almost always succeeds (writing the log file rarely fails), so <command> 2>&1 | tee <log> reports success even when <command> crashed. Any downstream step gated on steps.<id>.outcome == 'success' (rather than a PIPESTATUS-derived output) then runs against a job that silently produced nothing. Concrete example: preview.yml's capture-ui-evidence job ran node scripts/capture-ui-evidence.mjs ... 2>&1 | tee /tmp/ui-evidence-capture.log — when the script crashed (process.exit(1), zero PNGs produced), the step's outcome still read success, so the downstream check-png-legibility.mjs step (itself gated on that same outcome) found nothing to check and only logged a ##[warning] instead of failing the job. Confirmed on PR #2336: a 603-byte artifact holding only the capture log, down from a normal ~95KB of real screenshots — the PR still got a false "UI Evidence Captured" comment. admin.yml's sibling capture-ui-evidence job (playwright test ... | tee, same steps.capture.outcome == 'success' gate shape) had the identical bug. This is the same silent-success family as the errexit-swallows-diagnostics gotcha above, one layer further down the pipe. Fix (shipped): set -euo pipefail added before the pipeline in both preview.yml:986 and admin.yml:258, matching this repo's existing set -euo pipefail convention (15+ prior usages elsewhere in .github/workflows/) — confirmed the fix reports failure via isolated bash repro (false | tee /dev/null exits 0 under bash -e alone, exits 1 under set -o pipefail) and confirmed no happy-path regression via real CI artifact sizes on the fix's own PR (1.25MB and 682KB, consistent with genuine screenshot sets). Guardrail (shipped): scripts/check-ui-evidence-capture-pipefail.mjs extracts each workflow's live "Capture UI evidence" step body and asserts pipefail precedes the tee pipe; .github/workflows/ui-evidence-capture-pipefail-guard.yml runs it path-filtered to preview.yml/admin.yml/the guard script itself, so a future refactor that drops pipefail from either step fails CI instead of shipping silently. Not every | tee site in this repo needs this fix — admin.yml:553, staging.yml:1630, photo-purge-latency.yml:100, and playwright-smoke.yml:422 already capture the piped command's real exit code via ${PIPESTATUS[0]} rather than trusting step outcome; leave those alone. General rule: any new <command> | tee <log> pipeline in .github/workflows/ needs either an explicit set -o pipefail (or set -euo pipefail) before it, or a ${PIPESTATUS[0]}-derived output if the step's failure gates a downstream step. (Root cause: OBJ-2521, found re-verifying OBJ-2516/PR #2336's evidence-gate. Fix: PR #2337.)

An npm registry audit-endpoint outage hard-fails required CI checks repo-wide, and reads as a dependency vulnerability

• npm audit exits non-zero identically for "the registry is unreachable" and "a real high-severity advisory exists" — the log body, not the exit code, is the only discriminator. ci.yml has three npm audit --audit-level=high --omit=dev steps, each continue-on-error: false: "Audit frontend dependencies (production only)" (line 1105, job frontend-lint), "Audit admin dashboard dependencies (production only)" (line 1187, job vitest-admin), and "Audit agent runner dependencies (production only)" (line 1208, also vitest-admin). Only vitest-admin actually blocks the tests it shares a job with: its two audits sit before npm run test:unit -- --run (1202) and npm test (1223), so an outage there fails a check literally named Vitest admin_dashboard + agent_runner with no vitest ever executed. frontend-lint has no test step at all — its audit instead blocks ESLint (1118) and the guard scripts that make up the rest of that job; the frontend's actual unit tests live in the separate vitest job (1130–1165), which has no audit step and is unaffected. Two of the fourteen required status checks in ruleset 17927598 ("master CI gate") are Frontend Lint + Guards (ionic_frontend) and Vitest admin_dashboard + agent_runner, so this gates every PR, not just the one you're looking at. Observed failure text: npm error audit endpoint returned an error — a registry-side outage, not an advisory; a real finding instead prints an advisory table with severity counts. Exit code alone can't tell these apart; read the log body. Repo-wide, not per-PR: confirmed by checking master's own latest CI run failed on the identical step at the same time as an unrelated docs-only PR (run 33814746931 vs. PR #2894, both 2026-09-03) — checking master's latest run is the cheap first probe before reading your own diff for a cause. Mitigation shipped (OBJ-3340, commit 0469ec295, PR #2900): all three audit steps now wrap npm audit in a 3-attempt retry with a 30s backoff (ci.yml:1105-1116 for frontend-lint, 1187-1198 for the admin dashboard audit, 1208-1219 for the agent runner audit) — continue-on-error stays false, so a real high-severity finding still fails the step once retries are exhausted. Recovery: the retry loop already absorbs a brief outage automatically — a red here today means the outage outlasted the built-in ~90s (3×30s) of retries, or it's a genuine advisory. Read the log body first (see below) to confirm it's still the outage shape, then re-run the job (gh run rerun <id> --failed) — there is nothing to fix in the diff for that case. Distinct from a genuine master-wide advisory failure: this repo has separately hit a true positive (OBJ-1523, 2026-07-20 — a real transitive high-severity advisory, brace-expansion GHSA-3jxr-9vmj-r5cp, genuinely failing on master), where the trap was reproducing against a stale branch checkout and misreading an already-cleared gate as still-live; that failure mode has an advisory table in the log and a real fix (bump/patch the dependency). This entry is the opposite shape: no advisory at all, registry simply unreachable, and the only correct action (beyond the built-in retries) is re-running, not touching the diff. Don't conflate the two — check the log body first. Same class of exposure, now observed on the Ruby side too: the three required rspec shards' bundle audit check --update step (ci.yml:965-970) fetches a remote advisory database (RubySec, not npm) — structurally the identical exposure predicted here, and unlike the three npm audit steps it still has no retry wrapper as of this writing. It fired for real via OBJ-3400/CVE-2026-85396 (rubyzip) — see the bundler-audit entry below for that occurrence's specifics, including why the fix path differs from a plain npm audit fix. Fix: the retry-with-backoff mitigation (OBJ-3340) is the fix for the transient case; this entry documents the residual trap — a red that survives the built-in retries — so it's triaged by reading the log body instead of chased as a phantom vulnerability or code regression. (See the OBJ-1942 entry above for how frontend-lint came to carry the frontend audit without carrying its tests. Observed 2026-09-03 during OBJ-3325 / PR #2894; no root-cause ticket for the original outage — the trigger was registry-side. Retry mitigation: OBJ-3340, PR #2900.)

A ruby-advisory-db update can hard-fail bundler-audit, a required pre-gate, repo-wide with no code change

• bundler-audit can start failing against a lockfile that has been green for months, with zero repo changes, the moment ruby-advisory-db picks up a new advisory against an already-pinned dependency — and because it's a hard pre-gate, everything after it in the same job is skipped, not failed. The "Audit Ruby dependencies" step (ci.yml:965-970, if: matrix.shard == 0) runs inside the rspec job (display name RSpec + Rubocop (shard N/3)) before "Run Rubocop" (972), "Setup database" (984), "Run RSpec (shard N/3)" (990), "Check auth spec coverage" (994), the "Regression guard — full-stack HTML routes outside RSpec (OBJ-2100)" step (1006), and the SimpleCov artifact upload (1011) — none of those continue-on-error, so a failed audit step aborts the job and every step after it shows skipped, not red. The only surfaced annotation is a generic Process completed with exit code 1., with no hint that a dependency advisory (as opposed to a real Rubocop/RSpec regression) is the cause — the same misleading shape as the npm-audit entry above, just triggered by a genuine new advisory landing in RubySec's database instead of a registry outage.
• Reference case (OBJ-3400): ruby-advisory-db added CVE-2026-85396 / GHSA-47m2-wp7j-p9vc (High) against rubyzip 2.4.1 — a path-traversal fix requiring >= 3.4.0 — and bundler-audit started failing on master itself with no preceding commit to blame; confirmed by checking master's own latest CI run against its five green predecessors, the same cheap first probe the npm-audit entry recommends. rubyzip is a direct, pinned dependency (rails_api/Gemfile:32: gem 'rubyzip', '~> 2.4', require: 'zip'), and ~> 2.4 cannot resolve to >= 3.4.0 — so a plain bundle update cannot clear this class of finding. The constraint itself has to move (~> 2.4 → ~> 3.4), which is a 2.x → 3.x major version bump, not a patch. Fixed by bumping the Gemfile constraint and regenerating Gemfile.lock to rubyzip 3.6.0 (PR #2931) — the two Zip::File.open call sites (app/services/gdpr/export_builder.rb:57 and its spec) were re-verified against the installed 3.6.0 gem source and changelog before merging, since this feeds the GDPR data-export path.
• General rule: before assuming a red bundler-audit/bundle audit check --update step reflects a real code regression, check whether it's the advisory-DB-update shape — no diff in your branch, and master itself failing on the identical step. If so, read the advisory table in the log (gem name, current version, required fix version), then check the affected gem's Gemfile constraint: if it's a direct/pinned dependency (not merely transitive) and the fix version falls outside the existing ~> range, the fix is a Gemfile constraint edit plus a lockfile regen — potentially a major bump — not a routine bundle update, and any call sites against that gem's API should be checked against the target major version's changelog before merging. (Reference case: OBJ-3400, PR #2931. Documented: OBJ-3401.)

Replay-style test fixtures must pin the landing commit on master, never a PR head SHA

• A replay-style test fixture that pins a PR-head SHA is guaranteed to fail on a timer, with no offending diff (OBJ-3092) — scripts/check-milestone-close-ordering.test.mjs replays real history by reading .planning/ROADMAP.md at six pinned commit SHAs. Three of the six (PR_2386_HEAD, PR_2631_HEAD, PR_2601_HEAD) were the PR head commits of their respective historical PRs, not the commits that actually landed on master — a squash merge leaves the head commit reachable only for as long as its source branch still exists. PR #2386 squash-merged 2026-08-16; its source branch docs/v4.37-milestone-close was deleted per this repo's normal post-merge branch hygiene (~14 days later, confirmed 404 via gh api .../branches/docs%2Fv4.37-milestone-close), which made d8381fed… unreachable from any ref. actions/checkout@v6 with fetch-depth: 0 only fetches objects reachable from fetched refs, so a fresh CI clone doesn't have the object: gatherFileAtRef (check-milestone-close-branch.mjs:281-286) does git show <sha>:path, which fails and silently returns null, and extractShippedVersions(null) returns [] — the test asserted deepEqual(flipped, ['v4.37']) and got actual: []. Because nothing about this depends on any code change, the check (Regression tests (proves this would have caught OBJ-2947), .github/workflows/milestone-close-ordering-guard.yml) failed identically on every branch in the repo the moment the branch-deletion clock ran out — a repo-wide red with no offending diff, same failure shape as several other entries in this file. The other two PR-head pins passed only because their source branches happened to still exist; they were subject to the identical clock, just not yet expired. Fix (shipped, PR #2735): repointed all three *_HEAD pins to the permanent squash-landing commits on master (#2386 → e34352e4… "close v4.37 Flow Integrity & Interstitial Arbitration", #2631 → 016d169d… "close v4.45 Live Surfaces Beyond Notifications", #2601 → 14de9286… "close v4.43 Email Delivery Canary") and their matching *_BASE parents — verified the .planning/ROADMAP.md blob at every new pin is byte-identical to the PR-head pin it replaces, so every assertion is unchanged in substance; the tests still replay the same real history, just against refs that cannot disappear. No skip-when-fixture-missing escape hatch was added. Guardrail: the fix added an in-file comment stating the rule directly above the pins: pin the squash commit from master (git log origin/master --grep='(#<PR>)'), never the PR head. General rule: any test fixture that replays a historical PR by reading file content at a pinned commit SHA must pin the commit that landed on master (the squash-merge commit, findable via git log origin/master --grep='(#<PR-number>)'), never the PR's own head commit — a PR-head pin is a latent design flaw that WILL start failing repo-wide once that PR's source branch is deleted, regardless of any code change, and this repo's branch hygiene deletes merged source branches routinely. Known sibling, not yet fixed as of this entry: branch-promotion-guard.yml's check-branch-promotion.test.mjs:123 has the same defect class in a different file (branch-promotion-guard.yml red on master since 2026-08-24) — tracked separately as OBJ-3094, deliberately out of scope for PR #2735. (Root cause/fix: OBJ-3092, PR #2735.)

A pinned-fixture-SHA CI guard needs a fetch-by-object-ID step

• A replay-style test fixture that reads a pinned commit SHA needs an explicit fetch-by-object-ID step in its workflow — fetch-depth: 0 alone isn't enough once that SHA's source branch is deleted (OBJ-3071) — generalizes the entry above into the fix pattern itself. actions/checkout@v6 with fetch-depth: 0 only fetches full history for refs origin still advertises; a commit whose only reachable ref (a PR-head branch) has since been deleted becomes unreachable from any ref origin exposes, however deep the fetch. git show <sha>:<path> against that commit fails and silently returns nothing — the calling test collapses to a wrong-but-not-absent result ([] instead of the expected diff), not a fetch error. Confirmed independently by Tess while re-diagnosing the same check-milestone-close-ordering.test.mjs failure OBJ-3092 fixed: git fetch origin <sha> still succeeds even with zero refs pointing at the object, as long as GitHub hasn't pruned it — this is the fix's load-bearing fact. Three fix patterns exist for this defect class — pick by shape, don't default to the same one every time:
  1. Fetch the pinned SHA by object ID before running the suite (default choice — cheapest, most general). Template: .github/workflows/milestone-close-ordering-guard.yml's self-test job, Fetch pinned regression fixtures by SHA step (milestone-close-ordering-guard.yml:71-83, shipped PR #2726, OBJ-3071). It grep -oE '[0-9a-f]{40}'s every 40-hex-char SHA straight out of check-milestone-close-ordering.test.mjs itself — not hand-copied into the workflow — then git fetch --no-tags origin "$sha" for each, best-effort (a failed fetch ::warning::s instead of hard-failing the job, since the regression test's own assertion below is what surfaces a truly-pruned fixture clearly). Parsing the SHA list from the test file means the step can't drift out of sync as fixtures are added, changed, or replaced — apply this pattern whenever a test just needs a pinned commit's blob content, regardless of what the pin is being compared against.
  2. Repoint the pin to its master squash-landing commit instead of the PR head. Use only when the two pinned commits are diffed against each other and repointing doesn't change which commit is whose ancestor. This is OBJ-3092's fix (PR #2735, entry above) — a squash-landing commit on master can never be deleted, unlike a PR-head commit on a branch that gets cleaned up post-merge.
  3. Synthesize the fixture from git plumbing instead of pinning real history. Needed when pattern 2 would collapse the assertion to a false pass: if the corrected landing commit becomes an ancestor of the tip it's diffed against, git diff --diff-filter=A between them returns empty regardless of whether the code under test still works. This is OBJ-3094's fix (PR #2739) — check-branch-promotion.test.mjs's createAddedFileCommit() helper builds a real, unreferenced commit via read-tree/update-index/write-tree/commit-tree off a parent chosen to predate the scenario, so the test still exercises the real code path without depending on any specific historical SHA staying reachable. Dropped the V41_PHASE1_TIP pin entirely.
• Sweep status as of 2026-08-31 (OBJ-3108), fixed 2026-09-01 (OBJ-3109, PR #2748): grepping every scripts/*.test.mjs for a hardcoded 40-hex-char SHA constant read via git show <sha>:<path> (or equivalent) found the pattern in eleven files beyond check-milestone-close-ordering.test.mjs (fixed earlier, OBJ-3071/PR #2726). All eleven now carry the appropriate fix: pattern 1 (grep-and-fetch step) landed on check-branch-promotion.test.mjs (branch-promotion-guard.yml — this closes out the MASTER/V414_TIP/V45_TIP/V43_TIP/two SHELVED_* pins OBJ-3094's fixture synthesis had left exposed for one subtest), check-milestone-close-branch.test.mjs (milestone-close-branch-guard.yml), check-pr-scope.test.mjs (pr-scope-guard.yml), check-arch-domain-coverage.test.mjs (arch-domain-coverage-guard.yml), check-doc-surface-coverage.test.mjs and capture-doc-evidence.test.mjs (one shared step in doc-surface-coverage-guard.yml), check-marketing-referenced-undefined.test.mjs (marketing-contrast-guard.yml, scoped to the referenced-undefined job only — its other three jobs have no 40-hex fixtures), check-mutation-guard-recurrence.test.mjs (mutation-guard-recurrence.yml), and sync-posthog-feature-flags.test.mjs (posthog-flag-reverse-drift.yml — ci.yml's own posthog-flag-drift job never runs this self-test, so it needed no change). Three pairs needed a variant, all confirmed correct by review: check-revert-resurrection.test.mjs (revert-resurrection-guard.yml) excludes its INCIDENT_PATCH_ID pin by name — it's a git patch-id content hash, not a commit object, so fetching it by SHA would fail every run, not just once a real fixture is pruned; check-release-integrity.test.mjs (release-integrity-guard.yml) needed no fetch step at all — independently re-verified via git merge-base --is-ancestor <sha> origin/master that all seven pinned SHAs/tags are already master-reachable and permanent, documented inline rather than left as a silent no-op; check-store-asset-copy-drift.test.mjs (store-asset-copy-drift-guard.yml) reads its SHA from copy-manifest.json's capturedFrom.commit field at runtime rather than a literal in the test file, so it got a variant step resolving that field via node -pe and fetching it directly instead of the grep-the-test-file mechanic. check-clerk-auth-config.test.mjs, named in OBJ-3071's original sweep, does not actually match this pattern (no pinned-SHA constants, no git show) — verified false positive, not a fix candidate.
• General rule: any new test fixture in scripts/*.test.mjs that reads real repo history by a hardcoded commit SHA needs at least pattern 1 (add/extend a Fetch pinned regression fixtures by SHA-style step in its guard workflow) — a guard whose own fixture depends on undocumented ref reachability will eventually flake for reasons that look identical to a real regression, on a timer nobody can predict in advance. (Root cause: OBJ-3071, PR #2726. Sibling fixes: OBJ-3092/PR #2735, OBJ-3094/PR #2739, OBJ-3109/PR #2748. Sweep: OBJ-3108.)

A git-backed test that gates on a live branch name needs to own that branch, not borrow one

• A CLI exit-code test that points at a real origin/gsd/* branch by name silently stops asserting anything the moment that branch is legitimately archived or deleted (OBJ-3124) — scripts/check-branch-promotion.test.mjs's CLI: check exits 1 when a branch is stranded, but report and body still exit 0 subtest used STRANDED_IDLE_DAYS: '0' to assert the CLI's exit-code contract, but depended on gsd/v4.14-activity-nexus-connected-apps and gsd/v4.3-keystone-arch-sweep existing as live branches on origin to actually trip the "stranded" condition — gatherCandidateBranches (scripts/check-branch-promotion.mjs:170-179) enumerates refs/remotes/origin/gsd/* directly off the clone, so once both branches were legitimately retired (v4.3-keystone-arch-sweep archived to refs/heads/archive/gsd-v4.3-keystone-arch-sweep-phase-1-goal-tracking-superseded-by-2110; v4.14-activity-nexus-connected-apps deleted outright), the sweep found nothing stranded and check started exiting 0 instead of the asserted 1. Nothing about the code under test had changed — the fixture assumption just drifted out from under the test once those branches aged out of this repo's normal branch hygiene, failing identically on master itself for 5+ consecutive daily runs (confirmed run 33401285849 on 8/31 back through 33100143299 on 8/27). This is a different defect class from the pinned-fixture-SHA entry above: that family needs a specific commit object to stay fetchable by SHA — fetch-by-object-ID fixes it. This one needed a branch name to still exist as a ref at all; no amount of fetching-by-SHA helps once nothing points at the branch anymore. Fix (shipped, PR #2755, commit e2f61156f): added withTempCandidateBranch (scripts/check-branch-promotion.test.mjs:158-192), a helper that builds a real, never-pushed synthetic commit off origin/master via the file's existing createAddedFileCommit plumbing and registers it at refs/remotes/origin/gsd/<slug> — exactly the ref shape gatherCandidateBranches reads — then deletes the ref in a finally. The exit-code subtest (scripts/check-branch-promotion.test.mjs:705-721) now drives its failure off this test-owned branch instead of two branches it didn't control the lifecycle of. Verified: 8/8 green runs with the fix vs. deterministic failure on unmodified master across 3 runs; git for-each-ref refs/remotes/origin/gsd/ clean of test fixtures after teardown. General rule: a git-backed test that gates on a specific branch name existing on origin (not just a commit being reachable by SHA) must synthesize and own that branch itself — the same synthesize-the-fixture approach as pattern 3 in the entry above — never point the assertion at a real long-lived branch name, since this repo's branch hygiene (see "Branches, Releases & Planning" below) deletes and archives merged or superseded branches routinely, on a timeline the test has no visibility into. (Root cause/fix: OBJ-3124, PR #2755.)

An upstream action transferred to another org fails at Set up job, invisible on a monthly cron, and actionlint can't catch it

• A third-party GitHub Action that gets transferred to another org and archived stops resolving — the caller's job dies at Set up job, before any step runs, and actionlint is structurally blind to it (OBJ-3591) — .github/workflows/load-test.yml used grafana/k6-action@v0.3.1 at five call sites (L1-L5). The action was never deleted; it was transferred to the grafana-cold-storage org and archived (grafana-cold-storage/k6-action, archived: true, last push 2024-07-16). GitHub's action resolver does not follow the org transfer, so grafana/k6-action@v0.3.1 became permanently unresolvable, failing every run with Unable to resolve action. Repository not found: grafana/k6-action. Because the failure happens at Set up job — before the job's own steps start — nothing downstream (including a later Verify required secrets gate) ever executes, so an unrelated missing-secret fix cannot make the workflow green on its own. Why this was invisible for two months: load-test.yml only runs on a monthly cron plus manual dispatch, no pull_request trigger, so nobody saw the red run in a PR check list. actionlint (run in ci.yml's "Lint GitHub Actions workflows" job) is a static/offline linter — it validates YAML shape and expression syntax but never resolves a uses: reference over the network, so a fully clean actionlint run coexisted with a 100%-dead workflow the entire time. Fix (shipped, PR #3040): swapped to grafana/setup-k6-action@v1 (pinned k6-version: '1.8.1') plus a native k6 run <script> --summary-export=<path> step at each of the five call sites, preserving step ids, continue-on-error, env: blocks, and --summary-export paths byte-for-byte — the old action was a Docker action whose entrypoint was literally sh -c "k6 $K6_COMMAND $INPUT_FILENAME $INPUT_FLAGS", so a native run: step is a behavioral equivalent, not a rewrite. --summary-export was verified still present and still legacy-shaped against k6 v2.2.0 source before relying on it. General rule: an actionlint-clean workflow is not proof a third-party uses: reference still resolves — that requires either a live dispatch/run or checking the action repo's own transfer/archive history directly. This risk is structurally worse on any workflow with no pull_request trigger, since there's no PR-check surface to surface the red run quickly. (Root cause/fix: OBJ-3591, PR #3040. A follow-up "action-resolvability" CI guard — enumerating every uses: across the repo's ~60 workflows and checking resolvability directly — was flagged as a separate, not-yet-scoped issue.)

A git command inside an if: block is exempt from errexit — its failure silently reads as the else branch

• A git diff call inside a GitHub Actions if: condition is exempt from errexit, so a no merge base failure silently reads as changed=false, not an error (OBJ-3902) — ci.yml's posthog-flag-drift job (now ci.yml:1679-1716) ran git diff BASE.SHA...HEAD (three-dot) inside a step whose result fed an if: guard. Three-dot diff needs a computable merge base; actions/checkout@v6's fetch-depth: 2 plus a --depth=1 fetch of base.sha only produces one when the PR happens to be 0 commits behind master at check time — the common case, not the rare one. Otherwise the command died with fatal: ... no merge base, and because that failure sat inside an if: expression, bash's errexit doesn't apply to it: control fell through to the else branch, changed=false was set, and the real flag-registry-vs-PostHog check (scripts/sync-posthog-feature-flags.mjs --ci) was skipped by its own if: steps.changed.outputs.changed == 'true' guard — the job still exited 0 and the required check reported green, having verified nothing. Confirmed impact: PR #3248 and #3260 both merged brand-new PostHog flags (data_import_enabled, feedback_tags_enabled) through this exact vacuous path — master carried two flags this gate exists to catch and never checked. PR #3255, the one PR in the affected window that happened to be 0 commits behind master, hit the three-dot diff's happy path and correctly failed. Fix (shipped, PR #3280): extracted the detection into scripts/check-featureflags-changed.mjs, which diffs git diff BASE HEAD (two-dot) — compares the two trees directly, needs no ancestry at all, so it works identically regardless of how far behind master the PR is — and lets any other git failure (bad sha, unreachable fetch) propagate as a thrown error and fail the step loudly instead of being read as "unchanged." Guardrail: scripts/check-featureflags-changed.test.mjs builds two genuinely unrelated git repos to reproduce the >0-behind-master / no-merge-base shape directly (the exact #3248/#3260 case) and asserts detection still works there, reports correctly on a real 0-behind PR, and rejects loudly on an unexpected git error — wired as an unconditional step (Run featureFlags-changed detection regression tests) that runs before the live check in every posthog-flag-drift run, so a regression to the detection logic can't silently stop it from reporting again. General rule: a git diff/git log/any git command whose result feeds a step's own if: condition (rather than being asserted against directly) needs either a diff form that can't fail on ancestry (two-dot over three-dot when you don't need the merge base itself) or an explicit non-zero-exit check before the if: branches on it — an if:-gated shell block reading a prior step's exit code is not equivalent to running under errexit, and a required check built on that pattern can report green while checking nothing. The two flags that merged unchecked (data_import_enabled, feedback_tags_enabled) turned out to need no reconciliation: the PostHog flag drift gate log on run 35061722517 (PR #3255) enumerates the full flag registry and reports both as (exists) in PostHog — the vacuous greens on #3248/#3260 were real gaps in the check, but they never produced live drift between the registry and PostHog. (Root cause/fix: OBJ-3902, PR #3280.)

Deployment & Infrastructure

Firebase CLI silently no-ops deploy on missing IAM role

• Firebase CLI silent empty deploy on IAM misconfiguration — firebase hosting:channel:deploy exits 0 and writes deploy_output.json even when the CI service account lacks the roles/firebasehosting.admin IAM role on the project. The output JSON contains "fileCount": 0 and the resulting preview channel URL returns HTTP 404 (or 200 + "Site Not Found" HTML) for the entire smoke window. There is no error message and no non-zero exit code — the only signal is fileCount: 0 in deploy_output.json. Detection: always print deploy_output.json in CI immediately after the deploy step (cat deploy_output.json), and assert fileCount > 0 before passing the URL downstream. Fix: grant roles/firebasehosting.admin to the CI service account on GCP Project enkidu-488723 — this is a one-time IAM grant in GCP Console or via gcloud projects add-iam-policy-binding enkidu-488723 --member=serviceAccount:<SA> --role=roles/firebasehosting.admin. This issue (OBJ-660) was the 1st occurrence; the root cause was missing IAM on the preview channel site enkidu-app.

Firebase preview channel URL hash drift (recurring)

• Firebase preview channel URL hash drift (recurring footgun) — Firebase preview channel URLs contain a project-specific hash (e.g. https://enkidu-app--pr-123-<HASH>.web.app). This hash is stable for a given Firebase project but is NOT documented anywhere in the repo. Hardcoding it drifts silently — Firebase returns HTTP 200 + "Site Not Found" HTML for any non-existent channel, bypassing HTTP-status checks. This has happened 4 times (p0xgy2fx, kphes86g, q4br84vp, tcb1dvro all seen in history). Symptom: CI smoke test polls 300s, all attempts succeed (HTTP 200) but return wrong content, timeout is the only failure signal. Correct pattern: always derive the URL from Firebase CLI's JSON output at deploy time — never hardcode the hash in any CI workflow, CLAUDE.md, or script:

  - name: Set preview URL
    id: set-url
    run: |
      ACTUAL_URL=$(jq -r '.result["enkidu-app"].url' deploy_output.json)
      if [[ -z "$ACTUAL_URL" || "$ACTUAL_URL" == "null" ]]; then
        echo "::error::Firebase deploy_output.json did not contain a valid URL"
        exit 1
      fi
      echo "url=$ACTUAL_URL" >> "$GITHUB_OUTPUT"
      echo "Firebase preview URL: $ACTUAL_URL"

Production deploy concurrency silently drops runs 3–N

• Production deploy concurrency silently drops runs 3–N — .github/workflows/production.yml used concurrency: { group: production-deploy, cancel-in-progress: false } with no queue: key, which (per the cancel-in-progress: false alone isn't a FIFO queue gotcha below) only guarantees 1 run executing and 1 pending — tag pushes 3–N were silently dropped by GitHub Actions without triggering the workflow. Previously mitigated only procedurally (push tags one at a time, or use scripts/release.sh to serialize releases). Fix (shipped, OBJ-2481): added queue: max alongside cancel-in-progress: false, same pattern as this file's own smoke-production-demo group and staging.yml's global groups — now a real FIFO queue instead of a single pending slot. Enforced by ci.yml's lint-workflows guard ("Guard smoke-lane concurrency groups against pending-run cancellation"), which now includes production.yml:production-deploy in its checked pairs. See Production deploy safeguards for the separate fail-fast behavior that catches image/tag validation failures (unrelated to this concurrency gap).

Staging deploy concurrency can race on completion order

• Staging deploy concurrency can race on completion order, not just overlap — .github/workflows/staging.yml's deploy-backend/deploy-litellm/deploy-frontend jobs originally each used a bare, job-name-scoped concurrency group with cancel-in-progress: true (deploy-litellm still does). That only cancels a run that's genuinely still in-flight when a newer one starts — it does nothing if an older commit's job simply finishes its publish step after a newer commit's job already finished and deployed. Two commits landing on master ~2 minutes apart (OBJ-1144) raced this way: the newer commit's deploy-frontend finished first, but the older commit's finished later and silently overwrote it on Firebase Hosting, with the only signal being a reproducible git-describe mismatch in 00-version-tag.spec.ts. This is distinct from the "drops runs 3–N" gotcha above — that one discards excess runs outright; this one lets all runs execute but the last one to finish wins regardless of commit recency. Fix (shipped): each job re-checks origin/master's tip against github.sha immediately before its publish action and skips if superseded (is_stale job output); playwright-smoke skips when any deploy job reports stale. Follow-up (OBJ-1992, shipped): guard_stale alone doesn't protect an in-flight, not-yet-stale run from being interrupted mid-publish — deploy-backend (3 sequential gcloud run services replace calls) and deploy-frontend (one firebase-tools deploy across 2 hosting targets) each publish more than one resource per run, so a cancel-in-progress: true SIGTERM landing mid-sequence could leave those resources split across two commits' builds; confirmed happening on deploy-frontend (run 30710897912, killed mid npm run build) during the burst that produced OBJ-1992's staging smoke failures. Both jobs flipped cancel-in-progress: true → false; a lint-workflows guard in ci.yml fails CI if either regresses without first splitting its multi-resource publish into single-resource jobs. deploy-litellm/staging-migrate/staging-seed-data were left on cancel-in-progress: true since none of them publish more than one resource per run. See Staging deploy race for the full write-up and guard-step code, and apply the same guards to any future job reusing this concurrency pattern.

Staging validation-vs-mutation race: mutation jobs can outrun a still-running smoke validation

• A later push's migrate/deploy-backend jobs could start and mutate the live staging environment while an earlier push's playwright-smoke job was still validating that same environment — distinct from the "Staging deploy concurrency can race on completion order" gotcha above. That one is a deploy-vs-deploy race on finish order between two deploy-* jobs of the same kind; this one is a race between two different pipeline stages of two different pushes — one push's environment-mutating jobs against another push's still-running environment-validating job. staging.yml's concurrency: groups were scoped per job type (staging-migrate, staging-deploy-backend, smoke-staging-demo, ...), not per pipeline run — a concurrency group is held only while its own job is running or pending, so an older run releases each group between its own jobs, and a newer run's migrate can grab the slot in the gap between the older run's deploy-frontend and playwright-smoke. guard_stale is powerless here — the earlier run wasn't stale when it started. The guard only checks whether this run's own commit is still origin/master's tip at the moment it runs; it has no way to know that some other, unrelated run is concurrently mutating the environment it's about to validate. Confirmed 2026-08-19: a 13:08 UTC run's playwright-smoke was still executing 13:30:50→14:15:09 while a 14:03 push's migrate (14:07:38–14:12:07) and deploy-backend (14:12:10–14:16:28) ran against the same live staging environment underneath it — the smoke suite went from 60 failed/108 passed (44.3m wall-clock, vs. a ~10.4m baseline) back to 3 failed/166 passed (11.2m) the moment no concurrent push was mutating the environment during a run. Symptom shape: generic TimeoutError/toBeVisible timeouts spread across the entire authenticated app surface (dashboard, communities, settings, monetization, goals, coach, every feature-flag-gated route) plus a suite duration blowout — never a content or assertion-shape mismatch concentrated in one related spec set, which is the tell separating this from an actual app regression: a code regression concentrates in a related spec set and usually shortens affected runs (fail fast), it doesn't triple total runtime. No local repro is possible — this is a CI-timing-only race, reconstructed from gh run view --log/--json jobs timestamps across the day's pipeline runs, not from any single failing trace. Fix (shipped, OBJ-2751, PR #2496): staging.yml now has a workflow-level concurrency: { group: staging-pipeline, cancel-in-progress: false, queue: max } (staging.yml:13-31), mirroring production.yml:8-21's identical pattern (OBJ-2481) — a new push's entire run (build → migrate → deploy → smoke) now queues behind the previous run's playwright-smoke reaching a terminal state, closing a gap a per-job group structurally cannot close. ci.yml's lint-workflows guard ("Guard smoke-lane concurrency groups against pending-run cancellation") now includes staging.yml:staging-pipeline in its checked pairs, so the barrier can't silently regress to queue: single. A matching guard_stale step was also added to the two Docker build jobs (build-rails-api, build-agent-runner) so a burst of queued pushes still drains at near-zero cost for every superseded commit, instead of paying full build cost per push before the existing migrate/deploy-*/smoke guards kick in. Cost, accepted: each push now waits for the previous push's full pipeline including smoke — nominal ~10–15m, up to ~45m degraded under load — see Deployment Flow (Staging) for the cadence note. Residual gap, deliberately out of scope: preview.yml's smoke lane still points at the staging API (https://staging.api.objectuve.com) at per-PR volume and races a staging deploy with no barrier — serializing every PR preview behind the staging pipeline would wreck PR cadence, so this was left open as a separate, smaller-blast-radius gap. General rule: a per-job (or even a shared smoke-only) concurrency: group only closes races within its own job type — a validation job and a different pipeline's mutation jobs need a workflow-level group to ever see each other. See Staging deploy race for the sibling completion-order gotcha this one is easy to conflate with.

A new Hosting target needs its GCP site provisioned imperatively before the first real deploy

• A new .firebaserc/firebase.json Hosting target is purely repo-side config — it does not create the underlying GCP Hosting Site resource. Adding a targets.<project>.hosting.<target> block to .firebaserc and a matching { "target": "<target>", ... } block to firebase.json (deploy-sites.sh's pattern — see its enkidu-feedback lane) makes firebase-tools deploy --only hosting:<target> accept the target locally, but the GCP Hosting Site resource itself is a separate, imperative, human-gated step: npx firebase-tools hosting:sites:create <target> --project <gcp-project> (same pattern as gcp_setup/05_staging_resources.sh's enkidu-app-staging provisioning), plus mapping the custom domain in the Firebase console. Until that runs, every deploy to the target fails with Error: could not find site "<target>" for project "<gcp-project>" — but only once a run actually reaches the deploy step. The trap: a guard_stale-style staleness guard (see the "One Firebase Hosting target..." gotcha below) can mask the gap behind a green CI run indefinitely — if the first push to reach the deploy step races a second, newer commit, the guard correctly skips the stale run's deploy step, and CI reads all-green with zero real deploy attempts ever made; the gap only surfaces once a commit finally lands without racing a newer one. Fix: whoever adds a new Hosting target must provision the GCP site (hosting:sites:create) and map its domain in the same rollout window the repo-side config ships, not as a follow-up — treat "site created and domain mapped" as a launch blocker, not a nice-to-have. Precedent (2nd occurrence): enkidu-guide (OBJ-1846) was the first — resolved 2026-07-28, see docs/DOCS-AUDIT.md's "Phase 1: Public user-guide surface carve-out" entry; enkidu-feedback (OBJ-3864) is the second, surfaced when feedback.yml's deploy step first survived to execute, weeks after the target's repo-side config had shipped (OBJ-3796).

One Firebase Hosting target needs exactly one owning workflow

• One Firebase Hosting target, one writing workflow — a staleness guard cannot fix a two-lane race — The guard_stale fix above works because it compares against origin/master's tip within one lane. It is powerless when two different workflows publish the same Hosting target, because neither knows the other exists: Firebase has no commit ordering, so whichever firebase-tools deploy finishes last wins outright, even if it built an older commit. This became live exposure when admin_dashboard and marketing_landing moved off production.yml's tagged deploy onto their own master-push lanes (admin.yml / marketing.yml) — leaving hosting:enkidu-admin in production.yml's --only list would have meant a release tag and an admin merge racing each other with no guard capable of arbitrating. Fix (shipped): production.yml's deploy-frontend deploys hosting:enkidu-app only, and ci.yml's lint-workflows job has a "Guard against two workflows deploying one Firebase Hosting target" step that fails CI if any target appears in more than one workflow's deploy command. rollback.yml is the single documented exemption (break-glass, human-triggered instead of the forward lanes, never alongside them) and is excluded by name. If you add a Hosting target, give it exactly one owning workflow. See Independent surface lanes.

Frontend/API deploy skew and migration-model coupling (v4.0.0/.1)

• Frontend/API deploy skew, and migration-model coupling — the two v4.0.0/v4.0.1 release incidents — (1) Deploy skew: production.yml's deploy-frontend (Deploy Frontends to Firebase) and deploy-backend (Deploy Rails API to Cloud Run) used to run in parallel, both gated only on needs: validate. During the v4.0.0 production release (2026-07-12), deploy-backend failed at its enkidu-db-migrate "Run Database Migrations" step, but deploy-frontend had already deployed successfully — leaving the new frontend live against the old API schema for ~2h and producing user-facing GraphQL errors (Sentry OBJECTUVE-CLIENT-AG: Field 'pastAttemptContext' doesn't exist on type 'Goal', culprit: Dashboard). staging.yml's deploy-frontend had the same gap (no needs on deploy-backend at all). Fix (shipped): deploy-frontend now has needs: [validate, deploy-backend] in production.yml and needs: deploy-backend in staging.yml, so a migrate/deploy failure in the Rails API job blocks the frontend rollout in both. If you add a new frontend-ish deploy job, give it the same needs: on deploy-backend. (2) Migration-model coupling (the failure that tripped the migrate step in the first place): db/migrate/20260704000010_seed_team_plans.rb used the live Plan app model via find_or_create_by!. Production runs the full migration chain against the newest deployed code, and by the time this migration ran, Plan had already gained validates :kind, presence: true — but plans.kind isn't added until the next migration, 20260705000001_add_kind_to_plans.rb. Instantiating and validating the record crashed with NoMethodError: undefined method 'kind' for an instance of Plan, blocking the v4.0.0 API deploy. Staging never hit it (migrations ran incrementally, before Plan referenced kind); CI never hit it (test DB is schema-loaded, migrations never execute). Fix (shipped): the migration now uses an inner MigrationPlan < ActiveRecord::Base stub scoped to the migration class instead of the app's Plan model, with its own public_id generation (mirroring Infrastructure::GeneratePublicId without depending on it), and down uses delete_all instead of destroy_all. General rule: migrations must not reference live app models for anything beyond raw SQL-equivalent operations (update_all is fine; create!/find_or_create_by!, which instantiate and validate, are not) — the model's current code, not the code from when the migration was written, is what actually runs. See CHANGELOG.md's [Unreleased]/v4.0.1 "decouple SeedTeamPlans migration from the Plan model" entry for the full write-up. (Root cause: v4.0.0 release 2026-07-12, PR #1487.) Follow-up (shipped): "Run Database Migrations" was later split out of deploy-backend into its own dedicated migrate job in both production.yml and staging.yml (deploy-backend now needs: [..., migrate]), so a migration failure blocks deploy-backend via needs: rather than failing mid-job — the deploy-skew fix above is unaffected since deploy-frontend still transitively depends on migrate through deploy-backend. staging.yml's migrate job also got its own guard_stale step (mirroring deploy-backend's) since its cancel-in-progress: true concurrency group only cancels the local runner, not an in-flight Cloud Run Job execution — without the guard, two commits landing close together could race a concurrent rails db:migrate. Recurrence, a different independent surface lane (2026-09-12, OBJ-3670): the same deploy-skew shape hit docs_site's "Ask the docs" widget. PR #3097 (f58e1ba43, OBJ-3623) merged the widget and the askDocsQuestion GraphQL query field it calls in the same commit; docs.yml deploys docs_site on every path-filtered push to master, so the widget went live within minutes, while the Rails API only ships on a v* tag via the weekly release train — production was still v4.8.3, which predates the field. Every question the widget asked returned Field 'askDocsQuestion' doesn't exist on type 'Query' (extensions.code: undefinedField), but the UI showed a generic "That didn't go through. The answer service didn't respond." because docs_site/.vitepress/theme/askDocsQuestion.ts:89-95 folds any non-RATE_LIMITED GraphQL error into the same { kind: 'error' } outcome as a real network failure. Fix (shipped, PR #3123): docs.yml gained a schema-compat gate mirroring admin's — scripts/check-docs-schema-compat.mjs introspects the live production schema and hard-blocks deploy-docs on master push (advisory on PRs) — see The docs schema-compat gate for the implementation. General rule: any independent surface lane (see Independent surface lanes) that consumes the API needs its own schema-compat gate before it deploys on master push — path-filtered master-push deploy plus API dependency is the shape that produces this incident, not something specific to admin or docs.

__APP_VERSION__ is built from the git tag, not package.json

• __APP_VERSION__ is built from the git tag, not package.json. Production injects ${GITHUB_REF_NAME#v} as VITE_APP_VERSION; staging uses git describe --tags --always | sed 's/^v//'. npm run dev falls back to package.json#version. The frontend's Sentry release reads the same global. See docs/operations/deployment.md#frontend-version-injection.

Firebase Hosting headers rules are last-match-wins, not first

• Firebase Hosting headers rules are last-match-wins per header key — the opposite of rewrites/redirects, which are first-match-wins — and source always matches the pre-rewrite request path — Two independent gotchas compound here. First, a source: "index.html" header rule only ever matches a literal GET /index.html request; real navigations to /, /dashboard, etc. get rewritten to index.html by the catch-all rewrites rule, but Firebase Hosting matches a headers rule's source pattern against the path as originally requested, not the rewrite destination — so the rule silently never fires for any actual page visit. This let firebase.json's Cache-Control: no-cache rule for the HTML shell go completely unenforced: the v4.1.2 incident's trace showed the real GET / response came back Cache-Control: max-age=3600 instead, letting a stale HTML shell (referencing dead asset hashes after a redeploy) get cached for up to an hour. Second, once you fix that by widening the rule to source: "**", remember headers rules apply last-match-wins per key (confirmed against firebase-tools' own issue tracker) — the reverse of how rewrites/redirects resolve (first-match-wins) — so a broad "**" no-cache rule must be listed before the more specific assets/** immutable-cache rule in the same hosting config, or the later, more general rule would win and silently strip long-lived caching from hashed assets. Fix (shipped): for both the enkidu-app and enkidu-app-staging targets, the rule order is now security-headers "**" → no-cache "**" → immutable assets/** → sw.js no-cache → apple-app-site-association, so hashed assets still match the later, more specific assets/** rule and keep their immutable cache, while every other path (including all rewritten navigations) gets no-cache. General rule: when adding or reordering a Firebase Hosting header rule, work out final precedence by scanning the array bottom-to-up for the first matching source, and always test against the pre-rewrite path a real user request would use, not the rewrite target. (Root cause: OBJ-1500, PR #1587.)

Imperative gcloud run deploy --set-env-vars replaces the full set

• Imperative gcloud run deploy --set-env-vars REPLACES the full env-var set rather than merging with it — an omitted var isn't left unset, it's silently dropped from the revision — production.yml deploys the Rails API declaratively via gcloud run services replace against deploy/service.production.yaml (which carries LITELLM_URL/LITELLM_AUTH_MODE), but rollback.yml deploys the same service imperatively via a single gcloud run deploy ... --set-env-vars="..." invocation, and that var list had drifted out of parity with production.yml's — missing LITELLM_URL/LITELLM_AUTH_MODE specifically. This bit a real Tier 2 planned rollback during the OBJ-2145 hotfix incident (2026-08-04): /health's litellm check went degraded ("Failed to open TCP connection to localhost:4000"), breaking AI Coach / LLM features, and Vicki had to work around it with a Tier 1 emergency revision-traffic-switch rollback instead of the planned Tier 2 flow. Fix (shipped): rollback.yml's Rails API deploy step's --set-env-vars now includes LITELLM_URL=${{ secrets.LITELLM_URL_PRODUCTION }},LITELLM_AUTH_MODE=oidc, matching production.yml's equivalent value byte-for-byte. Guardrail (shipped): ci.yml's lint-workflows job has a "Guard against rollback.yml missing LITELLM_URL/LITELLM_AUTH_MODE" step that scans every .github/workflows/*.yml for an imperative gcloud run deploy ${{ env.API_SERVICE_NAME }} step lacking either var in its --set-env-vars list — the detection deliberately keeps the literal strings "gcloud run" and "deploy" in separate awk conditions (never adjacent) so it doesn't also trip check-runtime-service-account.mjs's comment-unaware --service-account scan. If you add a new imperative Rails API deploy step anywhere in .github/workflows/, its --set-env-vars list must carry every var the declarative production.yml path sets, or the guard fails you. (Root cause: OBJ-2145 incident, 2026-08-04. Fix/guard: OBJ-2148, PR #2092.)

Staging health-check Sentry gate doesn't filter by severity level

• Dave's staging health-check autopilot's Check 2 PASS criteria is "Sentry unresolved-issues list must be empty," with no filter on severity level — an existing, intentional info-level breadcrumb re-trips the gate identically to a real unhandled exception, every time it fires. Confirmed instance: Teams::ProcessSubscriptionLifecycleEvent (rails_api/app/interactions/teams/process_subscription_lifecycle_event.rb:20-33) is a deliberate no-op handler for a team-owned Stripe subscription-lifecycle/refund event that would otherwise fall through to Billing::ProcessStripeWebhook and corrupt personal-Supporter state (OBJ-1439) — it logs and calls Sentry.capture_message(..., level: :info) once per occurrence, pinned as expected behavior by rails_api/spec/interactions/teams/process_subscription_lifecycle_event_spec.rb:33-47. Sentry treats every capture_message call as a distinct unresolved issue regardless of level, so this correct, tested code trips the health check's Check 2 FAIL the same as a genuine regression would — and as team billing usage grows, it will fire routinely, with each occurrence looking like a brand-new incident to whoever picks up the resulting FAIL issue until they re-read the controller/interaction/spec chain from scratch. The gap is in the autopilot's gate, not the application code — routing logic in Webhooks::StripeController#create / Teams::ProcessSubscriptionLifecycleEvent is correct and already covered by a passing spec. Fix (not yet shipped): either narrow the health-check's Sentry query to exclude level: info (or level < warning) before evaluating "unresolved issues empty," or give this breadcrumb a stable Sentry fingerprint so repeat occurrences group into one long-lived issue instead of reappearing as "new" each time. (OBJ-2932, OBJ-2933)

Staging health-check Sentry gate doesn't filter ad hoc rails runner noise either

• Check 2's !level:info floor (the entry above, OBJ-2932/2933) excludes an expected info-level breadcrumb, but had no equivalent exclusion for source: "runner"-tagged events — so a sanctioned ad hoc bin/rails runner debugging session against staging trips the gate exactly like a genuine unhandled exception in deployed code, the same gap shape one layer over. Confirmed instance: OBJECTUVE-API-3R ("SystemExit: exit (SystemExit)", culprit bin/rails in <main>, 4 events clustered 2026-09-14T12:01:53–12:09:05Z) traced back to a human running bin/rails runner against staging and referencing the bare constant BadgeCatalog instead of its correct namespace, Achievements::BadgeCatalog. sentry-rails auto-tags any unhandled runner exception source: "runner" (confirmed against the installed sentry-rails 6.5.0 gem's railtie.rb), and Rails::Command::RunnerCommand#perform converts the resulting NameError into exit 1, which Sentry captures as SystemExit. rails_api/script/obj_2830_exposure_check.rb's own header comment documents this exact ad hoc-runner-against-staging workflow as sanctioned, expected practice — this is routine debugging traffic, not a rare edge case. The gap was in the autopilot's gate, not the application code — Achievements::BadgeCatalog is correctly namespaced everywhere it's actually used; no app bug existed. Fix (shipped, OBJ-3736, PR #3160): rails_api/config/initializers/sentry.rb:40-42 adds a before_send hook that downgrades any event tagged source: "runner" to level: :info, routing future ad hoc runner-script noise under Check 2's existing !level:info exclusion instead of needing its own filter clause. General rule: any workflow sanctioned to run against a live environment outside the deployed request path (ad hoc rails runner, a one-off rake task, a console session) needs its own Sentry tag-based exclusion before Check 2 can be trusted to only fire on real regressions — a bare is:unresolved (or even is:unresolved !level:info) gate can't tell "expected side channel" from "the code broke" on its own; each new side channel needs its own carve-out as it's discovered. (OBJ-2932, OBJ-2933, OBJ-3736) See also the entry below for the third recurrence and a timing wrinkle this shape doesn't cover on its own.

Staging health-check Sentry gate: a shipped carve-out doesn't retroactively resolve issues it predates

• Third recurrence of the same gap shape as the two entries above (OBJ-2932/2933, OBJ-3736): a sanctioned k6 load-test webhook payload trips Check 2 exactly like a real regression, because Webhooks::AiWorkforceController#create's "run not found" branch is expected, tested behavior for a synthetic run_id that intentionally misses the AiRun lookup — but on top of that, this instance surfaced a second, distinct wrinkle: a fix already shipped and live on staging does not retroactively resolve the Sentry issue its earlier, pre-fix occurrences created. Confirmed instance: OBJECTUVE-API-41 ("AI Workforce webhook: run not found or not in running status", level: warning, 275 events in a single 60-second burst, 2026-09-15T15:45:22Z–15:46:22Z) traced to a manually-dispatched load-test.yml run (workflow_dispatch, branch phase-3-staging-api-capacity, run 34990172634) whose L4 — AI Workforce webhook step (load_tests/l4_ai_workforce_webhook.js:38-47) fired against POST /webhooks/ai-workforce with a synthetic run_id: k6-load-test-<vu>-<iter>-<epoch> — exactly the payload shape ai_workforce_controller.rb:21-27 is coded to answer with 422 on every iteration (the load test marks 200/422 both expected, l4_ai_workforce_webhook.js:20-26).
  • Timing: the load test fired at 15:45–15:46 UTC against the staging revision deployed ~15:23 UTC (image dd2b7c84e...). The classification fix — rails_api/config/initializers/sentry.rb:46-50's before_send hook, which downgrades any event whose extra[:run_id] starts with k6-load-test- to level: :info so it falls under Check 2's existing !level:info exclusion — didn't even exist yet at that moment: it merged four hours later, at 19:48 UTC (5c1c4bbe5, PR #3250, OBJ-3880). By the time this run's health check inspected Sentry, the fix was merged and deployed (git merge-base --is-ancestor 5c1c4bbe5 <currently-serving-image> → true, confirmed against enkidu-api-staging-02060-rpm) — but OBJECTUVE-API-41 still showed is:unresolved, because shipping a classification fix changes how future events are tagged; it does nothing to the unresolved status Sentry already assigned to events from before the fix existed.
  • The generalized wrinkle: any noise-generating action (a load test, an ad hoc runner session, a one-off rake task) that fires while the serving revision predates its own carve-out — whether the carve-out hasn't merged yet, has merged but not deployed yet, or has deployed but the action ran before that — leaves behind a Sentry issue that will sit unresolved indefinitely, because nothing in this codebase auto-resolves a Sentry issue when the code that classified it changes. Check 2 has no way to distinguish "this will keep re-firing" from "this is the last event from before a fix that's already live" — both read as the identical is:unresolved hit.
  • The gap was in the autopilot's gate and in Sentry issue hygiene, not the application code — ai_workforce_controller.rb's warning-level logging on an unmatched run_id is correct, intentional, and unrelated to the deploy-timing question; no app bug existed here either.
  • Fast-path diagnosis, so a future Dave run doesn't re-derive this from git log/git merge-base each time a Check 2 offender looks like it might match a known carve-out:
    1. Check whether rails_api/config/initializers/sentry.rb's before_send (or an equivalent classification hook) already carves out this signal.
    2. If it does, compare the offending issue's lastSeen to that fix's deploy timestamp (Cloud Run revision Ready time), not its merge timestamp — git merge-base --is-ancestor <fix-sha> <deployed-image-sha> confirms whether the fix is live now, but doesn't tell you whether it was live when the event fired.
    3. If lastSeen predates the fix's deploy and the noisy source is one-off (a single ad hoc debugging session, a one-time rake task): this is a trailing pre-fix artifact, not a live gap — resolve the Sentry issue directly (Check 2 only detects, it doesn't resolve) and don't file a new carve-out.
    4. If lastSeen postdates the fix's deploy: the carve-out isn't actually catching this signal — treat it as a live gap needing its own fix, same as the two entries above.
    5. If the noisy source recurs (a scheduled or repeatable load test, a cron, anything that fires more than once) and Check 2 keeps re-tripping even after a manual resolve: don't keep resolving by hand — that's the fingerprint-tainting shape in the addendum below, and the durable fix is a fingerprint change, not another one-time resolve.
  • General rule: a Sentry before_send carve-out (or any code-level noise classification) is a forward-only fix. It stops future occurrences from re-tripping Check 2; it does not retroactively resolve the Sentry issue any pre-fix occurrence already created. Closing that issue out is a separate, manual step every time a new carve-out ships — Check 2's autopilot only detects, it never resolves. (OBJ-2932, OBJ-2933, OBJ-3736, OBJ-3880, OBJ-3883)
  • Addendum (OBJ-3885/OBJ-3886) — for a recurring noisy source, a level downgrade wasn't durable either, and the fix needed a step further: separating the fingerprint, not just resolving the issue. OBJECTUVE-API-41 is one Sentry group under a single static, message-based fingerprint. Even once the OBJ-3880 downgrade fix was fully live and correctly tagging every new k6 event level: :info, the group kept re-tripping Check 2 on the next load-test dispatch — because the health check's is:unresolved query (same query-shape family as docs/operations/sentry-environment-filter-gotcha.md) evaluates group-level resolution state over its lookback window (~14 days by default), matching if any event in that window satisfies the filter, not just the newest. The group's own pre-fix warning-level events — and each fresh manual resolve's own reopening, since Sentry reopens a resolved issue on its next matching event — stayed inside that window, so the shared group kept failing regardless of how correctly new events classified. A one-time manual resolve (step 3 above) only holds for a source that never fires again; it doesn't hold for a recurring one. Fix (shipped, OBJ-3885, PR #3263): rails_api/config/initializers/sentry.rb's before_send now also sets event.fingerprint = ['k6-load-test-ai-workforce-webhook'] on any k6-load-test--tagged event, alongside the existing :info downgrade — giving k6 traffic its own issue group going forward instead of continuing to write into, and re-taint, OBJECTUVE-API-41's shared one. Covered by the extended rails_api/spec/config/sentry_k6_load_test_downgrade_spec.rb. When this generalizes: any time a single static fingerprint groups traffic you want classified differently (synthetic/sanctioned vs. a genuine regression) and the synthetic source recurs, a level-only downgrade can't outrun the shared group's own history — pair it with an explicit event.fingerprint keyed on something stable to that source (a run-id prefix, a job name) so it stops sharing a group — and therefore a resolution history — with events Check 2 still needs to catch. Verification rule: a passing unit spec on the before_send lambda in isolation (this file's own sentry_k6_load_test_downgrade_spec.rb, which stubs Sentry.init and asserts the lambda's per-event decision) proves the classification logic is correct; it cannot see Sentry's server-side group-history behavior, so it cannot prove Check 2 actually goes green. Any Sentry-classification fix for this health check needs one live re-verification pass — re-run the health check's exact query, or wait for a real load-test dispatch — against the deployed fix before calling it shipped. (Fix: OBJ-3885, PR #3263, Codi. Docs: OBJ-3886, Dori.)

Overwriting a running bash script in place corrupts it mid-execution — cp truncates, mv doesn't

• A plain cp onto a live script path corrupts any process currently executing that script — cp opens the destination O_TRUNC and rewrites the same inode in place, but bash reads a running script by byte offset and re-seeks at loop/function boundaries, so the running process resumes inside different content and silently executes misaligned fragments. scripts/multica-host/disk-guard/install-disk-guard.sh installed both its managed scripts this way, with no check for a running target. This fired for real during the 2026-09-02 host disk recovery: the installer was run while ~/.multica/reap-stale-task-workdirs.sh (pid 82318) was mid-flight under --ttl-days 2 --confirm, and the file changed 16619 → 19952 bytes underneath it — killing an in-flight destructive rm -rf pass over ~1,200 task worktrees. Worst-case blast radius by construction: the file most likely to be running when an installer fires is the one script that deletes things. The self-matching trap: the obvious detection call, pgrep -f <installed_path>, self-matches — the target path also appears in pgrep's own command line, so it always reports a hit. Use ps -eo pid=,command= | grep -F <path> | grep -v grep instead. Fix (shipped): every install now writes to a same-directory temp file (<dest>.tmp.$$) and mvs it onto the destination — a real rename(2), which swaps the directory entry but leaves a running process holding the old inode, intact, until it exits — plus a pre-flight guard (the ps | grep -F | grep -v grep shape above) that refuses to install over a currently-executing target, naming the pid and path, with FORCE_INSTALL=1 as an explicit override that's safe precisely because the replace beneath it is atomic. General rule: any script that reinstalls another script onto a stable, potentially-running path must never cp (or any O_TRUNC write) directly onto that path — cp-to-temp-then-mv in the same directory is the only replace that can't corrupt an in-flight reader. Full incident record: docs/operations/multica-daemon-disk-hygiene.md. (OBJ-3186)

launchctl load/unload/list are domain-blind — use explicit-domain bootstrap/bootout/print

• The legacy launchctl load/unload API registers a job into the CALLER'S SESSION DOMAIN rather than an explicit one, so a job's continued registration depends on the session that happened to run the installer — not on the job itself. launchctl list inherits the same blindness: it can report a job as "registered" with no way to tell which domain, or whether that domain still exists. scripts/multica-host/disk-guard/install-disk-guard.sh used this legacy API to install two LaunchAgents (com.longwoodlabs.multica-disk-guard, com.longwoodlabs.multica-reaper); both silently deregistered on the same host on the same day (OBJ-3258, OBJ-3276) while a third LaunchAgent installed by a different path (com.longwoodlabs.multica-daemon) survived untouched — the failing set exactly matched this installer's target set. OBJ-3258's live remediation read launchctl list as showing the job "registered" right up until it silently vanished, with zero log trail — list gave no signal that anything was wrong. Root cause (OBJ-3283): the manual fix that stuck used launchctl bootstrap gui/501 <plist> — an explicit domain — instead of load. Fix (shipped, OBJ-3282): the installer now runs launchctl bootout gui/$(id -u)/<label> (idempotent, tolerates "not loaded") then launchctl bootstrap gui/$(id -u) <plist>, and asserts launchctl print gui/$(id -u)/<label> finds the job there before declaring success — launchctl print targets a specific domain, so it can't be fooled by the caller's ambient session the way list can. General rule: any LaunchAgent installer/verifier on this host should use bootstrap/bootout/print against an explicit gui/$(id -u) domain, never load/unload/list — the legacy API's ambient-domain behavior is exactly the kind of silent host-state divergence this file exists to catch, and it has now caused two independent silent-deregistration incidents on the same installer. Full incident record: docs/operations/multica-daemon-disk-hygiene.md. (OBJ-3258, OBJ-3276, OBJ-3283, OBJ-3282)

iam.serviceAccounts.getIamPolicy is not a substitute for resourcemanager.projects.getIamPolicy

• iam.serviceAccounts.getIamPolicy answers "who can impersonate this service account," not "what IAM bindings does this service account hold" — the two are easy to conflate when narrowing a project-wide IAM-read permission down to something resource-scoped. resourcemanager.projects.getIamPolicy returns the full project IAM policy: every principal and every role binding on the project. Swapping in iam.serviceAccounts.getIamPolicy when trying to narrow that grant looks like a scoped equivalent but reads a completely different resource — the SA's own IAM policy (who holds roles/iam.serviceAccountUser/.serviceAccountTokenCreator on it), not the project's. Surfaced narrowing dave-autopilot's daveAutopilotDiagnostics custom role (OBJ-3232): neither of the permission's two recorded uses — an ad hoc "is my denial real?" fluke-check, or a hypothetical incident-escalation "who can fix this?" lookup — would be served by the SA-scoped variant. If you're narrowing a resourcemanager.projects.getIamPolicy grant, the resource-scoped analog for "what does this identity hold" doesn't exist as a drop-in swap — reach for testIamPermissions instead (see the entry below) if the real question is "do I hold permission X." See dave-autopilot diagnostics role. (OBJ-3232)

gcloud projects test-iam-permissions is not a real gcloud CLI subcommand

• gcloud projects test-iam-permissions does not exist as a gcloud CLI subcommand (confirmed against SDK 567.0.0 — rejected as an invalid choice under gcloud projects), despite reading like the obvious CLI wrapper for the Resource Manager testIamPermissions REST method. The real replacement — for "does the caller hold permission X," a diagnostic that needs zero IAM grant to run — is the REST method itself, called directly (or via a client library):

  TOKEN=$(gcloud auth print-access-token)
  curl -s -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    "https://cloudresourcemanager.googleapis.com/v3/projects/<PROJECT_ID>:testIamPermissions" \
    -d '{"permissions":["run.services.update","compute.subnetworks.get"]}'

  Returns HTTP 200 with only the subset of the requested permissions the caller actually holds — no PERMISSION_DENIED on the call itself, regardless of what it reveals. Surfaced validating the dave-autopilot getIamPolicy removal (OBJ-3232): the REST call above (or a client library) is the correct instrument, never the nonexistent gcloud form. See dave-autopilot diagnostics role. (OBJ-3232)

An identity denied iam.roles.get on its own custom role can't safely do a full --file= role replace

• gcloud iam roles update <role> --file=<yaml> is a full replace — it overwrites title, description, and stage along with includedPermissions, using whatever the YAML says. An identity that's denied iam.roles.get on its own custom role (expected for a diagnostics-only SA) has no way to read back the role's current title/description to preserve them, so applying a hand-written YAML this way risks silently overwriting real metadata with invented values. Hit narrowing dave-autopilot's daveAutopilotDiagnostics role (OBJ-3232) — dave-autopilot itself can't gcloud iam roles describe its own role. Fix: use targeted --add-permissions=<perm> / --remove-permissions=<perm> instead of --file= when the applying identity can't independently verify the target role's current metadata — a surgical single-permission mutation can't touch fields it never had access to. Keep the full-state YAML as a reviewable IaC record (what the role should contain) separate from the apply mechanism (how it's actually mutated) — see gcp_setup/12_dave_autopilot_diagnostics_role.sh for the pattern. See dave-autopilot diagnostics role. (OBJ-3232)

Staging shared production's Cloud SQL instance and credential until the OBJ-3416 split

• Staging used to run on the same enkidu-postgres Cloud SQL instance as production — sharing both its connection-slot ceiling and its enkidu SQL user/credential — until OBJ-3416 gave it a dedicated instance. Orion traced four consecutive staging deploy failures (2026-09-01, OBJ-3154) to this: enkidu-postgres is a db-f1-micro (25 max_connections), already ~52% utilized at rest by production alone (13/25 via /health) before staging's own load was counted — deploy-time revision overlap (the old revision draining while the new one warms its pool) needs headroom that wasn't there. The same script had also been printing "The staging database shares the 'enkidu' SQL user with production" as an accepted fact, not a flagged gap. Fix (OBJ-3416): a dedicated enkidu-postgres-staging instance (db-f1-micro, us-central1-a, 10GB SSD, zonal — provisioned by Josh, not an agent; no runtime service account can create a Cloud SQL instance under the v4.5 least-privilege model), its own objectuve_staging database, and its own enkidu_staging SQL user backed by a staging-db-password secret version. .github/workflows/staging.yml and all three deploy/*.staging.yaml manifests now point at the new connection name and read DATABASE_PASSWORD/DATABASE_USERNAME from the new user/secret instead of production's. Two things worth remembering if you touch this split again:
  • enkidu_staging holds cloudsqlsuperuser (Cloud SQL's default Postgres user privileges) — this closes the shared-credential gap, but it is not a privilege reduction. Don't describe it as least-privilege.
  • The new instance has a public IP with no private network or authorized networks configured, matching enkidu-postgres's existing posture. That's parity, not a new exposure — but it's also not a network-security improvement. Staging's pre-split data was deliberately abandoned, not migrated (Josh's call) — the new database starts empty and is reseeded via the existing enkidu-demo-reset-staging job. The old objectuve_staging database on the shared enkidu-postgres instance still exists, unused; decommissioning it is a separate, gated ticket, not part of OBJ-3416. Provisioning is documented (not performed) by gcp_setup/11_staging_cloudsql.sh, which reproduces the instance/database/user/secret steps Josh ran by hand. See Cloud SQL for the current two-instance layout and the Connection-Pool Cliff / Database Connection Exhaustion runbook entries for the operational consequence. Two follow-ups spun out rather than folded in: OBJ-3418 (sidekiq-dead-clear.yml/smoke-activity-cleanup.yml still point at the shared instance — resolved by OBJ-3499, see the entry below, which also caught two more missed call sites in preview.yml/preview_teardown.yml) and OBJ-3419 (stale Cloud SQL connection alert + a gcp_setup/11_* script-numbering collision — this entry's own script is the third 11_* file). A third: the never-dropped objectuve_staging database's own residual autovacuum activity is what made a naive Cloud Monitoring "zero connections" check against it unpassable — see the ALIGN_MAX measurement-trap gotcha below and OBJ-3429's revised AC3 ("zero application backends," not the original "zero non-admin backends") for the actual decommission-gate wording. (OBJ-3154, OBJ-3416)

An OBJ-3416 Cloud SQL repoint missed 4 inline gcloud run jobs call sites outside staging.yml

• A migration that repoints one workflow's declared Cloud Run jobs doesn't catch every inline gcloud run jobs update/create call site aimed at the same instance. OBJ-3416 repointed staging.yml's 5 Cloud Run jobs and all three deploy/*.staging.yaml manifests at the new enkidu-postgres-staging instance / enkidu_staging user, but left 4 other inline gcloud run jobs calls hardcoded to the old shared enkidu-postgres instance and enkidu / production-db-password credential: preview.yml's "Ensure PR-scoped preview smoke account exists" bootstrap job, its documented mirror teardown job in preview_teardown.yml, smoke-activity-cleanup.yml's demo:trim_activity job, and sidekiq-dead-clear.yml's sidekiq:dead:* job (the last two were already flagged as a known gap, OBJ-3418, in the entry above, but not fixed until this incident). Each of those jobs kept reporting success while writing its account into a database the live API no longer reads from — so the preview-smoke bootstrap account existed, but User.find_by(email:...) against the new enkidu-postgres-staging database returned nil, and every fallback-auth GraphQL call against that account failed Authentication required (UNAUTHORIZED). The symptom read like a credential/token rotation problem — Capture UI Evidence (Preview) / Ensure Preview Smoke Account failing 3/3 on an unrelated PR's CI runs, plus staging.yml's own Seed Staging Data and Playwright Admin Dashboard smoke tests failing with the identical signature — but FALLBACK_AUTH_SECRET (a GitHub Actions secret) hadn't changed since March; it was a silent write/read split across two databases, not an auth-secret rotation. Fix (OBJ-3499, PR #2985): repointed all 4 remaining call sites at enkidu-postgres-staging / enkidu_staging / staging-db-password:2, matching staging.yml's already-correct pattern; production-context blocks in the two multi-environment workflows were left untouched. When repointing a Cloud SQL instance (or any shared infra target) again: grep the whole repo for every inline call site referencing the old instance/credential name, not just the workflow(s) already inventoried for the migration PR — a full declarative-manifest update and a staging.yml-only sweep both missed these 4. (OBJ-3416, OBJ-3418, OBJ-3499)

A 30-minute ALIGN_MAX Cloud Monitoring bucket can't distinguish a held-open connection from routine sub-minute activity

• A perSeriesAligner=ALIGN_MAX query over a bucket wider than the activity it's watching for reads "nonzero" whether a backend was held open for the whole bucket or touched it for 200ms — the two are indistinguishable at that resolution. OBJ-3512 investigated a num_backends series for objectuve_staging on the production enkidu-postgres instance that looked like a single connection held open continuously for ~20 hours: a flat, unbroken "1" across every 30-minute ALIGN_MAX bucket. Re-pulling the same window at native 60-second resolution showed the opposite shape — only 68 of 360 one-minute samples (19%) were nonzero, scattered irregularly with no run longer than ~2 minutes, against 100% of the 30-minute ALIGN_MAX buckets over that same window reading nonzero. Any activity that recurs faster than the bucket width — which routine PostgreSQL autovacuum-launcher passes against any undropped database will — trips every bucket, producing a flat "held open" signature for something that was never held open. The autovacuum-launcher attribution is supported by the metric shape and by Cloud Logging automatic vacuum/automatic analyze entries against objectuve_staging, but wasn't certified against pg_stat_activity (log_connections is off on enkidu-postgres, and no DB-level session was available in that pass). Fix: before treating a flat metric series as proof of a leak or a "zero activity" claim, sample at native 60s resolution or use ALIGN_MEAN, never ALIGN_MAX over a bucket wider than the expected inter-arrival time of routine background activity on that resource. See the falsifiability rule in docs/operations/dashboards.md and OBJ-3429's revised AC3 for a worked example of an AC written to that rule. (OBJ-3512)

gcloud run jobs update --add-cloudsql-instances never detaches a Cloud SQL instance from a previous migration

• --add-cloudsql-instances is additive — it only ever adds a socket mount, so a gcloud run jobs update call using it after a Cloud SQL instance migration leaves the old instance mounted alongside the new one indefinitely, silently. Every gcloud run jobs update ... || gcloud run jobs create ... pair in .github/workflows/ used to pair an additive --add-cloudsql-instances on the update half with the already-replacing --set-cloudsql-instances on the create fallback beside it — harmless while DATABASE_HOST stayed correctly configured, but it deleted the fail-fast missing-socket error that would otherwise catch a DATABASE_HOST misconfiguration immediately, on the same surface that had already produced four missed-call-site incidents (OBJ-3416 → OBJ-3418 → OBJ-3499 → OBJ-3512). Fix: use --set-cloudsql-instances on the update path too — it's equivalent-or-stricter whenever the site passes only one instance (true of all 16 sites found here). Now CI-guarded: .github/workflows/cloudsql-mount-guard.yml fails the build if --add-cloudsql-instances reappears anywhere in .github/workflows/. (OBJ-3512, OBJ-3513)

An intentionally-empty Cloud SQL cutover needs a post-cutover bootstrap-verification step, not just a stated reseed plan

• A one-time infra cutover that deliberately starts a database empty needs its own day-2 verification that the reseed job it names can actually complete against a genuinely empty instance — a stated intent isn't a check. OBJ-3416 (2026-09-06) cut staging over to a dedicated, empty enkidu-postgres-staging instance, with docs/operations/deployment.md:961 stating the plan plainly: "enkidu-postgres-staging's objectuve_staging database starts empty and is reseeded via the existing enkidu-demo-reset-staging job." Nothing verified that claim post-cutover. demo:reset (rails_api/lib/tasks/demo.rake) requires an admin actor to already exist before it will run DemoData::ReseedJob — and only db/seeds/development/users.rb created that first admin, a file nothing in staging.yml's seed-staging-data job ever loads. The result: a genuinely empty database could never self-seed, and demo:reset raised RuntimeError: no admin user found on every staging deploy from the moment of cutover (OBJECTUVE-API-3C), surfaced two days later by Dave's scheduled health-check autopilot, not by anything in the cutover's own rollout. Two earlier fix attempts (widening the admin lookup, PR #2988; an inline bootstrap in the rake task, rejected by the Shared Kernel Write Guard) treated this as a query-shape bug before Dave's second diagnosis pass traced it to the cutover's unverified bootstrap assumption. Fix (OBJ-3492, PR #2989): UserIdentity::EnsureBootstrapAdmin (see the entry below) gives demo:reset a genuine self-bootstrap path, so a fresh empty database can now seed itself without a separate manual step — but this closed only the admin-user half of the gap; it bootstraps admin@objectuve.com (demo: false) and never touches demo@objectuve.com. General rule: when a migration/cutover PR names another job as the thing that will populate a database it's intentionally leaving empty, that PR (or an immediate follow-up) needs to actually run that job against the empty state once, not just assert that it will work — this is the third OBJ-3416-adjacent gap in three days (the dedicated-instance split itself, then 4 missed inline gcloud run jobs call sites in the entry above, now this bootstrap-order gap), which is worth naming as its own pattern: an infra cutover needs the same day-2 verification pass a schema migration gets from schema.rb drift-checking. Second consequence, same root cause (OBJ-3569): the demo-user half of the gap surfaced two days after that, independently — db/seeds/development/users.rb's demo block was update-only (User.find_by + update!, no create! branch; the only creation branch lived in db/seeds/production.rb, gated to Rails.env.production?), so the nightly demo:trim_activity Cloud Run Job correctly raised RuntimeError: ...no user found for demo@objectuve.com (Sentry OBJECTUVE-API-3F) on a row that had never been created. Fix (OBJ-3569, closed by already-merged PR #3023): db/seeds/development/users.rb:86-99 now self-heals on either failure mode — User.with_deleted.find_by + restore! for a soft-deleted row (acts_as_paranoid default-scopes find_by), User.create! for a genuinely-missing one, using ENV.fetch('DEMO_CLERK_USER_ID_DEV', ...) for the real Clerk auth ID the demo login button depends on. Regression coverage: rails_api/spec/lib/tasks/demo_reset_spec.rb:75-95 exercises the fully-empty-DB case (no admin and no demo user) against the real users.rb domain module, not a stub. Third consequence, a second independently-vulnerable pipeline, same root cause (OBJ-3640): the day-2 verification gap this entry names was never scoped to demo:reset/DemoData::ReseedJob alone — data-seeding migrations are a second, separately-vulnerable pipeline against the identical hazard. db/migrate/20260405200004_seed_supporter_plans.rb and db/migrate/20260704000010_seed_team_plans.rb insert Plan rows via MigrationPlan.create! inside their up blocks (guarded by next if MigrationPlan.exists?(slug: ...)), but the enkidu-db-migrate-staging Cloud Run Job's first execution against the freshly-cutover, structurally-empty instance (enkidu-db-migrate-staging-qv52f, 2026-09-07T18:55:55Z — the day after cutover) took Rails' schema-load path: DROP TABLE IF EXISTS ... CASCADE + CREATE TABLE per table at the current HEAD schema, then one bulk INSERT INTO schema_migrations recording every historical migration — including both Plan-seeding ones — as applied, rather than replaying each migration's Ruby body. The two MigrationPlan.create! calls never executed, leaving plans silently empty on staging (and the supporter tier selector with zero options) for 5 days, invisible to /health and to every db:migrate run since (each correctly sees zero pending migrations and exits clean), until an unrelated deploy (a8a0bad96) happened to re-trigger supporter-overhaul.spec.ts:26 and catch it. Fix (Codi, OBJ-3640): an idempotent one-off backfill of the 5 missing Plan rows on staging, plus a guard step in staging.yml's migrate job that fails loudly when a fresh/near-empty target database takes the schema-load path. The named pattern, restated with all three surfaces in view: a single seed-completeness check — does every row a seed pipeline claims to produce actually exist, whether that pipeline is demo:reset, a data-seeding migration's up block, or any future pipeline shaped the same way — would have caught the admin gap, the demo gap, and this migration gap in one pass instead of three separate discovery cycles; no such check is built yet, this is prose only, not a shipped guard. This is the 4th OBJ-3416-adjacent incident (after OBJ-3492's admin-bootstrap gap and OBJ-3569's demo-user gap above, and the differently-mechanismed OBJ-3606 DELETION_ORDER/paranoid drift entry elsewhere in this doc) — the next intentionally-empty cutover's day-2 checklist needs to verify both the demo:reset pipeline and any data-seeding migrations actually ran their content, not just the former. (OBJ-3416, OBJ-3492, OBJ-3569, OBJ-3606, OBJ-3640)

A three-deep swallowed-exception chain let demo:reset report success on a real partial-seed failure

• A blanket rescue StandardError at each layer of a call chain can swallow a real failure completely, three layers deep, with nothing at the top ever seeing it — "the sub-call didn't raise" isn't the same guarantee as "the sub-call didn't fail." Admin::Logged#call (rails_api/app/interactions/admin/logged.rb:46-49) rescues StandardError, reports to Sentry, and returns a failed result — by design, since it's a generic action wrapper other interactions call directly. DemoData::ReseedJob#finalize (rails_api/app/jobs/demo_data/reseed_job.rb, pre-fix) read that failed outcome, wrote status: 'failed' to the progress cache, and then returned normally instead of propagating it — so perform's own top-level rescue StandardError => e (reseed_job.rb:37-41) never saw anything to catch. rails_api/lib/tasks/demo.rake:33-34 calls ReseedJob.new.perform(...) with no rescue of its own and unconditionally prints [demo:reset] complete. on the next line. With nothing above finalize ever seeing the failure, rake demo:reset exited 0 and printed success on every real mid-seed failure — the actual mechanism behind the OBJ-3569 cascade above: six of the reseed job's nine domain modules (partners.rb:11, gamification.rb:37, mood_logs.rb:15, notifications.rb:15, communities.rb:78, goals.rb:179) use the bang finder User.find_by!(email: 'demo@objectuve.com'), so any run where the demo user row didn't already exist raised ActiveRecord::RecordNotFound at the second module and was swallowed the same way. The only symptom that ever surfaced was unrelated to demo:reset itself — demo:trim_activity's own deliberate raise (a different task) and two Cloud Run job exit(1) failures a human had to notice externally — demo:reset never signaled anything was wrong on its own. Fix (OBJ-3599/OBJ-3568, PR #3045): finalize still writes progress first, unchanged, then re-raises: raise "demo_data reseed failed: #{error}" if outcome.failure? (reseed_job.rb:91). A real failure in any of the 9 modules now propagates out of perform uncaught, so the rake task fails loudly instead of reporting green. Regression coverage: spec/jobs/demo_data/reseed_job_spec.rb asserts the raise and that the cache still records status: 'failed'; spec/db/seeds_spec.rb adds a real (non-stubbed) users.rb → partners.rb cascade assertion — the gap OBJ-3555's own spec block didn't cover, since it never loaded a downstream module. Known side effect, not yet fixed (OBJ-3601, backlog): finalize's new raise also now trips perform's pre-existing top-level rescue StandardError => e (reseed_job.rb:37-41), which previously never fired on a reseed-content failure. write_progress runs twice (harmless — the cache write merges) but the error ends up double-wrapped ("demo_data reseed failed: Admin action failed: <original>"), and Sentry.capture_exception fires twice for one real failure — once inside Admin::Logged#call's own rescue, once again in perform's. Not a correctness bug, but expect two Sentry issues per reseed break until that's addressed. Amplified one layer deeper (OBJ-3602): ClearDemoDataJob#finalize (rails_api/app/jobs/demo_data/clear_demo_data_job.rb:164-182) picked up the identical must-raise-on-failure shape, needed to fix an unrelated bug — a real ActiveRecord::InvalidForeignKey on goals (via integration_activities/habit_integration_mappings, neither cascaded nor demo-scoped) was being swallowed the same way, leaving the outer Postgres connection aborted with no ROLLBACK issued and surfacing as a misleading PG::InFailedSqlTransaction in production.rb's first query (Sentry OBJECTUVE-API-3J/3K; real root cause OBJECTUVE-API-3H). Because ReseedJob#run_reseed! calls ClearDemoDataJob synchronously as a nested sub-step of its own already-open Admin::Logged action, the next FK gap detach_orphaned_goal_dependents doesn't yet cover will now compound through two raise/rescue/Sentry layers instead of one: inner Admin::Logged#call catches the FK violation (1st Sentry capture, error: "Admin action failed: <FK msg>") → ClearDemoDataJob#finalize raises "demo_data clear failed: Admin action failed: <FK msg>" → ClearDemoDataJob#perform's own rescue (2nd Sentry capture, re-raises) → outer Admin::Logged#call (in ReseedJob) catches it (3rd Sentry capture, error: "Admin action failed: demo_data clear failed: Admin action failed: <FK msg>") → ReseedJob#finalize raises the quadruple-wrapped string → ReseedJob#perform's own rescue (4th Sentry capture, re-raises). Net effect for the next uncovered FK gap: 4 Sentry issues and a quadruple-nested error string, instead of the double-wrapped shape above. Dormant on the OBJ-3602 fix itself — the golden path now succeeds cleanly and never reaches either finalize's raise — but the OBJ-3601 backlog item now needs to account for two compounding layers, not one, before it's closed. Trade-off, accepted as intended: this also raises staging's CI noise floor — any transient failure in one of the 9 modules now hard-fails seed-staging-data and cascade-skips smoke_tests on every deploy, with no rollback path since the deploy has already happened by then. That's the fix working as designed, not a regression. (OBJ-3568, OBJ-3599, OBJ-3602, PR #3045)

One User.demo flag carries opposite intent across two systems — "findable by smoke" vs. "safe to hard-delete" (OBJ-3597)

• Two independent systems read the same User.demo flag and assume opposite things about what true means, with nothing naming either assumption as a contract. db/seeds/development/users.rb:21 (inside find_or_create_demo_user!) deliberately stamps admin@objectuve.com demo: true on every reseed — intentional, so admin_dashboard smoke can find her via a User.demo scope. DemoData::ClearDemoDataJob's demo-scoped User hard-delete purge (pre-fix: entry[:model].with_deleted.demo.to_a in its DELETION_ORDER loop) reads that same flag as "safe to wipe," with no exception for the actor attributing the very reseed it's running inside of. From the second reseed onward, UserIdentity::EnsureBootstrapAdmin#existing_admin (rails_api/app/interactions/user_identity/ensure_bootstrap_admin.rb:23-25) hands ReseedJob that same demo-tagged admin row as its actor, ClearDemoDataJob hard-deletes her mid-transaction, and Admin::Logged's AdminAction.create!(actor: ...) immediately after raises PG::ForeignKeyViolation on admin_actions.fk_rails_66b5257ce2 — a self-referential delete-then-log inside one transaction. Deterministic on every run once the first reseed flips her demo flag, which is why staging's Playwright smoke stayed red on every executed run for 24h+ (22 failures, 0 passes across 40 staging.yml runs) until diagnosed.
• Fix (OBJ-3597, PR #3050 — merged): ClearDemoDataJob now excludes the acting admin (actor_id:) from both the dependent-detach sweep and the User purge step, regardless of her demo flag — other_demo_user_ids = User.with_deleted.demo.where.not(id: actor&.id).pluck(:id) is computed before calling detach_orphaned_user_dependents (rails_api/app/jobs/demo_data/clear_demo_data_job.rb:78), and the DELETION_ORDER loop's paranoid branch rejects any User record matching actor.id before hard-deleting (clear_demo_data_job.rb:92). This closes the self-referential FK gap without touching either system's underlying assumption — the seed still tags her demo: true on purpose, and the purge still treats demo: true as its scope; only the actor herself is carved out of that scope, and only for the User row and its own direct dependents (her other demo-tagged data — goals, XP ledger, etc. — still purges normally each run, since the other DELETION_ORDER entries and the residue-goal computation above the exclusion aren't actor-scoped).
• The flag still means two different things depending on which system reads it — this is a targeted exception, not a resolution. Any future demo-scoped system added against User.demo inherits the same ambiguity: does true mean "discoverable test fixture" or "eligible for hard-delete"? Check both existing readers (db/seeds/development/users.rb, clear_demo_data_job.rb) before assuming either meaning generalizes to a new one.
• Correction to the fix's own commit message: the spared admin is not "purged and reseeded normally on a later pass by a different actor." UserIdentity::EnsureBootstrapAdmin#existing_admin (ensure_bootstrap_admin.rb:23-25) resolves via User.where(admin: true).first || User.joins(:admin_roles).distinct.first — the same row every time there's exactly one admin, which is the normal steady state. She's retained indefinitely across every future reseed as the actor, never purged on some later pass by a different one. (OBJ-3597, PR #3050)

A hand-maintained DELETION_ORDER paranoid: flag drifted from its model's own acts_as_paranoid, and a plain unique index turned that drift into a self-perpetuating demo:reset blocker (OBJ-3606, OBJ-3635)

• A hand-maintained metadata array whose per-entry flag duplicates a fact already declared on the model will silently drift — and when the affected table also carries a plain, non-partial unique index, that drift becomes a production blocker that regenerates itself on every failed run. DemoData::ClearDemoDataJob::DELETION_ORDER (rails_api/app/jobs/demo_data/clear_demo_data_job.rb:25-38) marks each purge target paranoid: true/false by hand to choose between really_destroy_all (hard-delete) and destroy_all (soft-delete) — see the file's own header comment. Its AiEmployee entry read paranoid: false even though AiEmployee (rails_api/app/models/ai_employee.rb:7) declares acts_as_paranoid, so every demo:reset purge soft-deleted AiEmployee rows instead of hard-deleting them. Because index_ai_employees_on_role_key (schema.rb:145) is a plain, non-partial unique index — no where: deleted_at IS NULL — a soft-deleted leftover still occupied its role_key slot at the DB level. The next reseed's db/seeds/development/ai_workforce.rb find_or_create_by!(name: 'Gym') (paranoia-scoped, so it can't see the soft-deleted row) issued an INSERT that collided with that row's role_key in the unique index, and Rails' post-failure find_by! retry — also paranoia-scoped — couldn't find the row it had just failed to insert over, raising ActiveRecord::RecordNotFound inside DemoData::ReseedJob#finalize. Confirmed live on Cloud Run execution enkidu-demo-reset-staging-ctcv5 (2026-09-10) by Dave, deterministic across three consecutive runs. Every failed purge left a fresh soft-deleted row behind, so once the first orphan landed, staging's demo:reset — and downstream Playwright Smoke Tests (Staging) — could never recover on its own. This is the same DELETION_ORDER mechanism the User.demo entry above turns on, hitting a different table.
• Fix (PR #3082): DELETION_ORDER's AiEmployee entry corrected to paranoid: true, mirroring the Community/Goal/User branches already in the array. A one-time data migration (db/migrate/20260911170000_hard_delete_orphaned_soft_deleted_demo_ai_employees.rb) hard-deletes the rows already stranded on staging — raw SQL only, scoped to demo = true AND deleted_at IS NOT NULL, manually replicating the ai_employee_memories → ai_artifacts (via ai_runs) → ai_runs → ai_employees cascade really_destroy! would perform, never instantiating the live AiEmployee model in the migration — see migration-model coupling above.
• Why OBJ-3597's own regression specs didn't catch this: clear_demo_data_job_spec.rb and demo_reset_spec.rb ran against a fresh Postgres test DB with no pre-existing soft-deleted rows to collide with — this only manifests against accumulated state from every past demo:reset since the mismatch was introduced. The original regression spec pinned only AiEmployee's DELETION_ORDER entry against AiEmployee.paranoid?, not a blanket assertion over every entry.
• The blast radius was six entries, not one (OBJ-3635, PR #3166): AiEmployee was never the only drifted entry, just the first one a live collision surfaced. OBJ-3635 started as a single fix — AiRun (clear_demo_data_job.rb:31) had the identical paranoid: false-vs-acts_as_paranoid mismatch, dormant at the time (no non-public_id unique index on ai_runs to collide against) — bundled with turning the OBJ-3606 spec into a blanket assertion over all 12 DELETION_ORDER entries. That new guard failed CI on its first run against a live third drift, FeedbackComment; fixing it and re-checking every remaining entry by hand turned up two more previously-undiscovered instances, FeedbackPost and AiArtifact, plus GoalEvent in the same pass. 6 of the array's 12 entries had drifted — AiEmployee, AiRun, FeedbackComment, FeedbackPost, AiArtifact, GoalEvent — half the array, not the single case this entry originally described. Roy's review independently re-verified all 12 entries against each model's own acts_as_paranoid declaration by reading the model files directly, and checked schema.rb for the same role_key-style collision risk on the four newly-fixed tables: feedback_comments, feedback_posts, and ai_artifacts each carry only a public_id unique index (unrelated); goal_events' idx_goal_events_client_event_dedup is a partial unique index scoped where: deleted_at IS NULL, so a soft-deleted leftover falls outside it and can't collide. No live or latent collision risk on any of the four. All 12 entries are now covered by the blanket spec (clear_demo_data_job_spec.rb:231).
• General rule: before trusting a hand-maintained per-model flag in an array like DELETION_ORDER, check whether the model itself already exposes the fact that flag is supposed to mirror (here, Model.paranoid? from the paranoia gem) — and if the affected table has any plain unique index, a paranoid: false mismatch on a model that's actually acts_as_paranoid won't just misbehave once, it will compound on every subsequent run. A single fixed instance doesn't mean the array's other entries are clean — a blanket assertion over the whole array, not a per-entry pin, is what actually closes this drift class; the per-entry pin found one drift and missed five others sitting right next to it. (OBJ-3606, PR #3082; OBJ-3635, PR #3166)

gcp_setup/02_create_resources.sh creates a database named enkidu_production — the real production database is objectuve_production

• gcp_setup/02_create_resources.sh's Cloud SQL step has always created a database literally named enkidu_production — that name has never matched the actual production database. rails_api/config/database.yml's production block defaults to database: objectuve_production whenever DATABASE_NAME is unset, and no deploy/*.yaml manifest or .github/workflows/*.yml job ever sets DATABASE_NAME — so objectuve_production is what every real deploy actually connects to. That name is independently corroborated by docs/operations/deployment.md and by docs/operations/disaster-recovery.md's live restore drill (cloned the real production backup and confirmed objectuve_production, 71 tables, 120 migrations, intact). git log --follow on the script shows the enkidu_production line has read that way since the file was first committed — it was never renamed to track the real name. Re-running this script against a fresh project as written would create a database Rails never looks at. Flagged by Orion (OBJ-3154 task package) as a pre-existing discrepancy; this entry and the corrected deployment.md doc are the resolution for the documentation side — fixing the script's literal string is a separate follow-up, not done here. (OBJ-3154, OBJ-3416)

08_runtime_service_accounts.sh's awk parser only stripped a literal :latest, silently mis-extracting any pinned secret version

• extract_job_secrets() and extract_litellm_secrets() in gcp_setup/08_runtime_service_accounts.sh stripped a secret's version suffix with sub(/:latest$/, "", secret) — a pattern that only matches the literal string :latest. .github/workflows/staging.yml pins DATABASE_PASSWORD=staging-db-password:2, a real version number, not :latest. Hand-executed against that line, the function emitted staging-db-password:2 — not a valid Secret Manager resource name — instead of staging-db-password. Re-running 08 as-is would not grant enkidu-run-jobs-staging access to the secret it needs; it would keep emitting an invalid name, so the grant would keep silently failing to apply. Surfaced while investigating OBJ-3425 (a missing roles/secretmanager.secretAccessor grant on staging-db-password for four staging runtime SAs, applied by hand). extract_manifest_secrets() (used by api/worker via deploy/*.yaml, and by 09_crono_service_account.sh via deploy/crono.*.yaml) was never affected — those manifests carry the version in a separate key: field, never appended to the secret name. Fix: both sub(/:latest$/, ...) sites now use sub(/:.*$/, ...), which strips any version suffix (:2, :latest) and is a no-op on a bare name with no : at all. Net effect before the fix: a missing jobs-class grant fails silently at deploy time (PERMISSION_DENIED on Secret Manager access) instead of loudly at provisioning time, since 08's own assert_nonempty check only verifies the extracted list isn't empty — it doesn't validate the extracted names are well-formed. A structural assertion that no extracted secret name may contain : would catch this class of bug non-tautologically; that's tracked as a separate follow-up, not folded into this fix. (OBJ-3454)

gcloud artifacts repositories set-cleanup-policies leaves cleanupPolicyDryRun sticky

• set-cleanup-policies --policy=<file> only writes the cleanupPolicyDryRun field to its update mask when --dry-run/--no-dry-run is named explicitly on the CLI line — the bare absence of --dry-run does not clear a previously-set dry-run flag. A repository that was ever registered with --dry-run stays in dry-run mode forever, silently, no matter how many subsequent set-cleanup-policies calls omit the flag — only a call that passes --no-dry-run explicitly flips it. gcp_setup/18_artifact_registry_cleanup_policy.sh (OBJ-3466, v4.54 Phase 2) hits this directly: its default invocation calls set-cleanup-policies ... --dry-run (safe, simulates only), and its --enable path must call set-cleanup-policies ... --no-dry-run — not just omit --dry-run — to actually register the policy for GCP's cleanup pipeline. Verified live against enkidu-registry on OBJ-3466: a policy applied with a bare set-cleanup-policies --policy=... (no dry-run flag at all) left the repository's existing cleanupPolicyDryRun: true untouched. Documented inline in the script's --enable branch so the next person editing it doesn't drop the --no-dry-run on a refactor. See Artifact Registry cleanup policy — APPLIED for what the policy that depends on this actually keeps and prunes. (OBJ-3466)

gcloud auth activate-service-account and ADC's GOOGLE_APPLICATION_CREDENTIALS are two independent credential stores

• Rotating a service account's key file alone does not resync the gcloud CLI's own cached credential — every gcloud subcommand keeps signing with the retired key until gcloud auth activate-service-account is re-run. A host running any service account has two separate credential stores that both claim to hold "the" key: the key file GOOGLE_APPLICATION_CREDENTIALS points at (e.g. ~/.config/dave-autopilot/key.json) — read fresh, directly, on every call by Application Default Credentials (ADC) and any Google client library — and ~/.config/gcloud/credentials.db, gcloud's own internal store, populated once by gcloud auth activate-service-account --key-file=... and then reused indefinitely; it does not re-read GOOGLE_APPLICATION_CREDENTIALS on each invocation. Rotating the key file (writing a new key to the ADC path) is sufficient for any ADC/client-library caller by itself, but leaves gcloud's cached credential pointed at a private_key_id Google has now retired — every gcloud subcommand run under that account (gcloud run services describe, gcloud run revisions list, gcloud logging read, etc.) then fails 100% of the time with invalid_grant: Invalid JWT Signature., not intermittently. Symptoms that rule this out are clock skew (a genuine skew reads as "Token used too early/late", not "Invalid Signature") and a corrupt key file (the file still parses as valid JSON with all expected fields). Fix: re-run gcloud auth activate-service-account <account> --key-file=<path> after every key rotation — one command, re-syncs the CLI's cached credential to the current key. Nothing in this repo's tooling re-runs this automatically on a schedule or on rotation; a script that shells out to gcloud under a rotatable service account should either re-activate the key idempotently at the top of its own run, or preflight-compare the key file's private_key_id against gcloud's currently-activated credential for that account and fail loud on mismatch, rather than assume the two stores are still in sync. Structurally the same shape as the source-of-truth-vs-cached-copy drift class this repo has hit before on other surfaces (OBJ-3559/OBJ-3563) — a live source of truth changed, a cached/derived copy didn't, and nothing diffed the two — just at the credential layer instead of a docs-mirror layer. See docs/operations/deployment.md's dave-autopilot diagnostics role section for this service account's key details (non-expiring USER_MANAGED key, OBJ-3218 tracks migrating it to Workload Identity Federation). Fix (OBJ-3564): scripts/multica-host/ensure-gcloud-sa-credential.sh implements the idempotent-reactivation option above — it's the "gcloud credential preflight" step now called at the top of every gcloud-shelling Dave/Vicki runbook in docs/guides/multica-autopilots.md: Staging health check, Crono liveness check, Email canary liveness check, Email suppression liveness check (OBJ-3560), and Vicki release readiness preflight. (Root cause: OBJ-3564.)

A Cloud SQL instance's reserves budget is fully claimed — it is not deploy-overlap headroom

• The 8-connection reserves entry in deploy/capacity-budget.json for both Cloud SQL instances (enkidu-postgres, enkidu-postgres-staging) is not spare capacity — it's a named allocation, fully claimed before a single deploy-time connection is counted. deploy/capacity-budget.json's reserves object breaks the 8 down explicitly: superuser: 3, cloudsqladmin: 3, migrate_job: 2 — 3+3+2 = 8, zero left over. Nothing in that accounting sets anything aside for the connection spike a deploy-time revision overlap produces (the old Cloud Run revision still holding its pool while the new revision boots and opens its own). deploy/service.staging.yaml's comment used to claim the opposite — that this reserve absorbed deploy-overlap headroom — until OBJ-3956/OBJ-3957 corrected it (deploy/service.staging.yaml:56-68).
• What the conflation hid: modeling the actual worst-case deploy-time overlap draw (reserves + Σ_{other consumers}(pool×maxScale) + 2×(redeploying consumer's pool×maxScale) — advisory rule 5 in scripts/check-deploy-capacity.mjs, added OBJ-3957) puts both instances over their 25-connection ceiling today, with zero margin in either sizing: enkidu-postgres-staging draws 33 against 25 (worst case: enkidu-api-staging redeploying), enkidu-postgres draws 37 against 25 (worst case: enkidu-worker-production redeploying) — run node scripts/check-deploy-capacity.mjs to reproduce. This overlap gap is what produced the staging ActiveRecord::DatabaseConnectionError that traced back to it (Sentry OBJECTUVE-API-2Z, OBJ-3956).
• Fix, scope, and what's still open: OBJ-3957 path-gated .github/workflows/staging.yml's backend deploy chain to cut overlap frequency (a docs-only push no longer triggers a redundant migrate/deploy-backend run) and added rule 5 as advisory-only — it reports the draw, it does not fail the build, since every instance already fails it today by design. Raising maxScale, pool sizes, or the Cloud SQL tier to close the overlap gap itself is an open capacity decision for Josh, tracked on OBJ-3956 — not something this reserve, or this fix, resolves. See rails-concurrency-capacity.md § Known limitation for the worst-case bounds recorded as an ongoing capacity gap, not a one-time incident writeup. (OBJ-3956, OBJ-3957)

Backend (Rails/GraphQL)

User#details is plural, not user.detail

• User#details is plural — User declares has_one :details, class_name: 'UserDetail'. Calling user.detail (singular) raises NoMethodError. Always use user.details. Check spec expectations too — the singular form compiles but blows up at runtime.

Migration without regenerated schema.rb breaks CI

• Adding a migration without regenerating schema.rb silently breaks CI — Rails 8.1's maintain_test_schema! detects schema SHA mismatches and reloads schema.rb before running tests. If a new migration file exists but schema.rb hasn't been regenerated, the schema reload drops the new table, and the migration check finds it pending and raises ActiveRecord::PendingMigrationError. RSpec attributes the load error to the first alphabetical spec file, masking the real cause. Fix: After committing a migration, always run bin/rails db:migrate and commit the updated schema.rb in the same CI cycle.

Custom Interaction gem, not ActiveInteraction

• Custom Interaction gem — Use Interaction::Base, NOT ActiveInteraction. Current patterns (v3.3.0): input, guard, call, result.details, fail_with(error:, code:). delegate_input and result.fail(error:) are pre-v3 and still work in older interactions (19 and 58 files respectively) but shouldn't be used in new code — see CLAUDE.md's Interaction Pattern section for the current example.

A nested Admin::Logged call shares one physical Postgres transaction with no savepoint

• Admin::Logged#call (rails_api/app/interactions/admin/logged.rb:13-15) wraps its action_block in a bare ActiveRecord::Base.transaction do...end with no requires_new: true — when one Admin::Logged-wrapped action calls another synchronously from inside its own action_block, both share one physical Postgres connection/transaction, not a nested savepoint. Admin::Logged#call's own rescue StandardError => e (admin/logged.rb:46-49) catches whatever the inner call raises and returns a failed result instead of re-raising — by design, since it's a generic wrapper other interactions call directly. But because the shared transaction was never a real savepoint, no ROLLBACK is issued at that point; only the outermost transaction call issues a real ROLLBACK/COMMIT. The physical connection is left in Postgres's "aborted" state while Ruby-land believes nothing has gone wrong, so the next unrelated statement anywhere else in the outer transaction throws the misleading PG::InFailedSqlTransaction: current transaction is aborted instead of the real error. First hit by DemoData::ReseedJob calling DemoData::ClearDemoDataJob synchronously as a nested sub-step of its own Admin::Logged action (reseed_job.rb:59) — a real ActiveRecord::InvalidForeignKey inside the inner call was swallowed this way, and the only visible symptom was production.rb's first, unrelated GoalType query blowing up several statements later (OBJ-3602; see the three-deep swallowed-exception entry above for the full incident and its fix). General rule: any future Admin::Logged-wrapped action that synchronously calls another Admin::Logged-wrapped action will reproduce this exact misdiagnosis shape — a swallowed inner failure surfacing as a confusing downstream PG::InFailedSqlTransaction instead of its real cause. ClearDemoDataJob/ReseedJob closed the one FK gap that tripped this on OBJ-3602, but did not change Admin::Logged itself — a requires_new: true savepoint on the inner call (or requiring nested callers to check outcome.failure? and re-raise, as ClearDemoDataJob#finalize and ReseedJob#finalize now both do) is still open, unticketed structural work, not something this entry's incident fixed. (OBJ-3602)

A migration landing while an older, still-warm revision keeps serving traffic can leave GraphqlController#execute holding a stale prepared-statement plan

• Signature to grep for: ActiveRecord::PreparedStatementCacheExpired immediately followed, seconds later, by ActiveRecord::StatementInvalid: PG::InFailedSqlTransaction — same culprit (GraphqlController#execute), single occurrence each, right after a deploy. First seen 2026-09-14 (OBJ-3779, Sentry OBJECTUVE-API-3W/OBJECTUVE-API-3X, 15:26:43–44Z).

• What it is not (two corrections worth stating explicitly, since both are easy first guesses):

  1. Not a guard_stale/cross-run concurrency race, and not a third sibling of Staging deploy race (OBJ-1144) or its OBJ-1992 follow-up. staging.yml's migrate and deploy-backend jobs are already hard-sequenced (deploy-backend: needs: [..., migrate], staging.yml:270) inside the same workflow run, on the same commit — there is no staleness divergence between two different runs to close here. Don't file this under deployment.md's "Gotchas: GitHub Actions Concurrency" heading; it doesn't belong there.
  2. Not the nested Admin::Logged gotcha above, even though both eventually surface as PG::InFailedSqlTransaction. The distinguishing tell: here, InFailedSqlTransaction is preceded by a PreparedStatementCacheExpired on the same connection, seconds apart — a Postgres plan-cache invalidation, not a swallowed inner exception leaving a transaction aborted. There, InFailedSqlTransaction appears alone, with no preceding PreparedStatementCacheExpired, because its root cause is a rescued-and-swallowed error inside a shared, non-savepointed transaction.

• What it actually is: a structural exposure window inherent to any migrate-before-deploy pipeline, not a bug in guard_stale's sequencing. rails_api/config/database.yml sets no prepared_statements: false override, and both staging and production connect directly to Cloud SQL over the Unix socket (no PgBouncer transaction-pooling in the path) — so Rails' postgresql adapter uses server-side prepared statements everywhere by default. Even inside one hard-sequenced run, db:migrate finishing (schema changed) and deploy-backend finishing (new revision at 100% traffic, old Puma↔Postgres connections recycled) are two different timestamps — on OBJ-3779's run (34859540214) that gap was migrate at 15:25:57Z → deploy-backend at 15:29:30Z. Any request served by the still-warm previous revision during that gap, on a connection holding a server-side plan for a table the migration just altered, can hit Postgres's "cached plan must not change result type" guard: PreparedStatementCacheExpired if the statement is inside a transaction (Rails auto-rolls-back and clears the local statement-name cache before re-raising — but only for the outermost transaction, and only that cache, not any Postgres-side aborted-transaction state); every subsequent statement on that same now-poisoned connection then raises ActiveRecord::StatementInvalid wrapping PG::InFailedSqlTransaction instead, until something actually rolls the physical transaction back. Production carries the identical window — .github/workflows/production.yml:169-217 has the same deploy-backend: needs: [..., migrate] shape — with real users on it; shrinking the window (e.g. the staging-only job-graph reorder below) helps, it cannot close it, because some gap between "schema changed" and "every serving connection recycled" always exists in a migrate-then-deploy pipeline.

• Fix (shipped, rails_api/app/controllers/graphql_controller.rb#execute_schema/#stale_prepared_statement?, lines 76–101): the app survives it instead. execute_schema rescues the stale-plan signature exactly once, calls ActiveRecord::Base.lease_connection.reset! (a real ROLLBACK if the connection isn't idle, then DISCARD ALL, then a statement-cache clear — not clear_cache! alone, which only clears the local Ruby-side statement-name cache and cannot recover an already-aborted Postgres transaction), and retries the query on the now-clean connection. A second failure, or any unrelated error, propagates unchanged through the existing Sentry.capture_exception path in process_query. stale_prepared_statement? walks the full #cause chain (not just one hop) matching either ActiveRecord::PreparedStatementCacheExpired or PG::InFailedSqlTransaction anywhere in it — RailsServerSchema's GraphQL::Backtrace wraps resolver-raised errors in GraphQL::Backtrace::TracedError, and PG::InFailedSqlTransaction itself always arrives wrapped as ActiveRecord::StatementInvalid via Ruby's implicit exception chaining, so a single-hop #cause check would miss both wrapped shapes. A successful retry emits an info-level Sentry.capture_message breadcrumb so recurrence stays visible rather than silently disappearing. No transaction is open at the rescue point — GraphqlController/Interaction::Base never wrap a request in their own transaction, so the only transaction in play is whichever individual interaction opens itself (verified against GoalTracking::CheckInHabit#persist_completion!'s single, unnested ActiveRecord::Base.transaction do — the realistic trigger, since it touches the migrated table — has no requires_new: anywhere in its call chain and genuinely is the outermost transaction); Rails has already unwound it before the exception reaches the controller, so the retry re-executes from a clean slate and cannot double-apply a mutation's side effects. Covers both the single-query and _json batch paths (rails_api/spec/requests/graphql_prepared_statement_retry_spec.rb). Rollback lever, documented but not shipped: prepared_statements: false in rails_api/config/database.yml would eliminate the failure class outright, at the cost of losing server-side plan caching on hot read paths (goal/habit-completion queries) for a failure that occurs a few times a year — keep as the fallback if the retry proves fragile, don't ship it first.

• Fix (shipped, staging-only optimization, .github/workflows/staging.yml): Verify Migration-Seeded Data used to run serially inside the migrate job (contributing ~3m45s of the exposure window for no benefit to blocking the deploy) — it's now its own verify-migration-seed-data job (staging.yml:565) gated on needs: deploy-backend, collapsing the staging window from ~5m19s to roughly the deploy itself. deploy-backend's needs: migrate is unchanged. This shrinks the staging-only window; it does not touch production's, and does not by itself make the retry above unnecessary — the retry is what actually closes the failure for users, on both environments.

Admin/User provisioning belongs in a User Identity interaction, never an ops rake task or seed file

• Creating or promoting an admin user is a User Identity write, not something an ops rake task or seed file may do inline — scripts/check-shared-kernel-writes.mjs enforces this and fails the build on a direct user.update!/update/update_columns call site outside it. User is this codebase's shared kernel (docs/architecture/ddd/03-context-map.md:89); the guard greps rails_api/app/rails_api/lib for direct write call sites and requires every one outside rails_api/app/interactions/user_identity/ to be in its hardcoded ALLOWLIST — and that allowlist is "a permissive freeze of today's surface, not a shrink" (the script's own docblock), so admitting a new write site by adding to it is the wrong move, not just an inconvenient one. This tripped OBJ-3492's round 2 fix: an inline bootstrap_admin! helper added directly to rails_api/lib/tasks/demo.rake called user.update!(admin: true) from outside User Identity and failed CI's Verify no unallowlisted direct writes to User job. The sanctioned move: USER_IDENTITY_HOME_DIR (rails_api/app/interactions/user_identity/) is the guard's own path-based exclusion — a write there needs no allowlist entry at all, because it's User Identity's own sanctioned mediator, not an exception to the rule. Fix (shipped): the bootstrap logic moved into UserIdentity::EnsureBootstrapAdmin (rails_api/app/interactions/user_identity/ensure_bootstrap_admin.rb), following the same Interaction::Base .call/result.details shape as its user_identity/ siblings; demo.rake calls it and is back to three lines (resolve an actor, raise if still nil, run the reseed job). The interaction also demonstrates a second locked invariant worth citing alongside this one: the bootstrapped admin is created with demo: false, never demo: true — a demo: true actor would sit inside DemoData::ClearDemoDataJob's hard-delete scope and could be wiped mid-reseed before Admin::Logged records the attributing AdminAction against it. General rule: if a change needs to create or mutate a User record and the change doesn't live under rails_api/app/interactions/user_identity/, either route it through an existing User Identity interaction/event, or write a new one there — don't reach for a direct user.update!/update_columns from a rake task, seed file, job, or any other context, and don't "fix" a guard failure by adding your new site to the ALLOWLIST. (OBJ-3492, PR #2989)

Never expose integer IDs — always use public_id

• Never expose integer IDs — Always use public_id in GraphQL types. Lookups: Model.public_find(id).

An uncoded GraphQL::ExecutionError breaks the frontend's own error-classification contract

• An uncoded GraphQL::ExecutionError breaks the frontend's own error-classification contract silently — The codebase has a well-established convention for tagging GraphQL errors machine-readably: the Authorization concern (rails_api/app/graphql/concerns/authorization.rb) raises every auth failure with extensions: { code: 'UNAUTHORIZED' } / 'FORBIDDEN', and the frontend's useApiError#extractCode (ionic_frontend/src/composables/useApiError.ts:9-26) is built specifically to read error.graphQLErrors[0].extensions.code and route to the right toast/redirect/template. Nothing enforces that every hand-rolled GraphQL::ExecutionError.new(message) actually sets extensions:, though — a resolver can raise a bare, uncoded error and it silently falls through extractCode to null, landing on the generic "Something went wrong" branch instead of the code-specific one, no matter how specific the underlying condition actually was. This is invisible in review because the error still fires, still gets caught, still reports to Sentry — it just can't be classified once it gets there. Confirmed instance (OBJ-2801): Resolvers::GoalQueries#goal/#public_goal/#goal_event/#is_following_goal (rails_api/app/graphql/resolvers/goal_queries.rb) raised bare 'NOT FOUND' errors for the ordinary, correctly-enforced case of a viewer requesting a private goal they don't own or partner on. Because the error carried no extensions.code, PublicGoal.vue couldn't distinguish it from a real network failure and rendered the "Couldn't reach the server. Tap to retry." state instead of its own existing "This goal isn't public" copy — and each retry tap re-fired the same deterministically-failing query, manufacturing a fresh Sentry event every time (271 events/73 users and 114 events/39 users over 7d, all commingled into one Sentry issue pending OBJ-2348's fingerprinting work). Fixed by tagging the four call sites with extensions: { code: 'NOT_FOUND' } (PR #2542) and branching PublicGoal.vue's template on extractCode(error) === 'NOT_FOUND'. Not fully swept: goal_queries.rb still has four more bare GraphQL::ExecutionError.new(...) calls with no extensions.code — goal_kinds, goal_categories, goal_summary (forwards summary.details[:error] verbatim), goal_progress_data (forwards progress.details[:error] verbatim) — and a repo-wide grep confirms 60 files under rails_api/app raise GraphQL::ExecutionError.new at all — not every call site in every one of those files sets extensions:. Scoping and fixing the rest is an open follow-up, not yet a ticket. When adding or reviewing a resolver error: if the condition has a name more specific than "something went wrong" (not found, forbidden, validation failure), give it an extensions: { code: '...' } matching an existing code the frontend already understands (UNAUTHORIZED, FORBIDDEN, NOT_FOUND), or useApiError#extractCode has nothing to key off. (OBJ-2801, PR #2542)

SessionToken header, not Authorization: Bearer

• SessionToken header — PascalCase, NOT session-token or Authorization: Bearer. Carries Clerk RS256 JWT.

Clerk is the auth provider — no custom auth

• Clerk is auth provider — No custom sign-in/sign-up mutations, password hashing, or JWT generation. All auth via Clerk. Profile updates (name, username) must sync to Clerk.

/health degrades on Sidekiq dead-queue threshold

• /health returns degraded + HTTP 503 when Sidekiq dead-queue size exceeds SIDEKIQ_DEAD_THRESHOLD (default 25). The dead queue contains jobs stuck permanently; this is the correct signal for a health check. Never use stats.failed (lifetime cumulative counter) for health gates — it will eventually fire false positives as the counter grows on busy services.

check_litellm gated /health against a dependency deployed to legitimately cold-start (OBJ-3164)

• A gating /health check against a deliberately-scale-to-zero dependency produced false "degraded" responses at ~42% of requests, with a fully healthy API/DB/Redis/Sidekiq underneath. enkidu-litellm-staging runs with --min-instances=0 (.github/workflows/staging.yml:688, deliberate cost control) and is redeployed on every master push regardless of whether infra/litellm/ changed. Its legitimate cold-start latency (observed up to ~97s across retries) regularly exceeded check_litellm's 8s combined timeout budget (health_controller.rb:116-136). Until this fix, litellm was a gating check — absent from HealthController::INFORMATIONAL_CHECKS — so any /health request landing during a cold-start window flipped the entire response to degraded/HTTP 503, even though nothing was actually broken. This is architecturally the same shape already solved for crono (dead-boot detection, OBJ-1290) and email_canary — both fail-open at the HTTP layer on purpose, with the reasoning documented at health_controller.rb:6-20 — but litellm had never been given the same treatment. Fix (shipped): added litellm to HealthController::INFORMATIONAL_CHECKS (rails_api/app/controllers/health_controller.rb:28). Cloud Run's startupProbe already targets /health/live (checks nothing, the OBJ-2479 fix), so this doesn't weaken deploy-readiness gating — it only stops a legitimate cold LiteLLM from flipping the status-page/monitoring surface to degraded. Regression guard (shipped): rails_api/spec/requests/health_spec.rb asserts /health stays 200/ok with checks['litellm']['status'] == 'error' when check_litellm times out; a pre-existing rails_api/spec/controllers/health_controller_spec.rb example that still asserted the old degraded contract was updated to match. If a genuinely-down (not just cold) LiteLLM should still page someone, that needs its own fail-closed layer — same pattern as the "Dave — Crono liveness check" Multica autopilot — rather than folding it back into /health's gating status; not yet built. (Root cause + diagnosis: OBJ-3164, Dave. Fix: PR #2783, Codi.)

Demo login via /sign-in?demo

• Demo login — Visit /sign-in?demo to reveal demo button. See docs/architecture/authentication.md.

User migration via rake clerk:migrate_users

• User migration — Use rake clerk:migrate_users to import existing users to Clerk (supports bcrypt hash import). Check status: rake clerk:migration_status.

Day streak tolerates one missed day per 7 (soft-grace)

• User-level day streak tolerates one missed day per 7 streak-days (soft-grace) — User#current_streak forgives single-day gaps via User#calculate_contiguous_streak (and the new shared walk_streak helper), but the forgiven day does NOT count toward the streak number. Forgiven dates are surfaced via User#streak_grace_days_used and the User.graceDays GraphQL field (non-null array of ISO date strings, defaults to []). Two adjacent missed days always break. Grace budget = STREAK_GRACE_INITIAL (1) + floor(streak / STREAK_GRACE_INTERVAL_DAYS (7)). No retroactive migration — update_streak! recomputes naturally on next trigger, which may cause current_streak to jump for users with a recent gap and re-fire consistency_is_key / habit_former / dedicated badge triggers as a one-time side effect. The dashboard's StreakDetailsModal renders forgiven dates with a muted-warning border (no solid fill) and shows the welcome-back copy "Rest day taken — you're back on it." when a grace day occurred within the last 7 days.

StreakGapChecker computes gaps from habit_completions, never Goal-level streak fields

• Gamification::StreakGapChecker determines whether a goal has a repairable gap purely from its HabitCompletion rows — it never reads the Goal-level streak fields that look like they should carry the same information. StreakGapChecker#compute_missed_dates (rails_api/app/services/gamification/streak_gap_checker.rb:26-38) derives last_real from goal.habit_completions.where(streak_freeze_used: false).maximum(:completed_date); with zero completions, last_real is nil, missed_dates is [], and any_missed_days? is unconditionally false — no matter what pre_break_habit_streak, streak_repair_eligible_until, or streak_repaired_count say on the Goal record itself. Gamification::StreakRepairOfferResolver#find_repair_target (rails_api/app/services/gamification/streak_repair_offer_resolver.rb:70) filters its candidate scope through any_missed_days?, so a goal can look fully eligible by every Goal-level field and still be silently excluded. This fails silently wrong, not obviously wrong — a fixture, admin tool, or rake task that seeds only the Goal-level fields will pass code review and reproduce this exact bug in some other seeding/debug context. Confirmed instance (OBJ-3121): the smoke:ensure_preview_account rake task's repair-eligible habit provisioning set only Goal-level fields, so spec/lib/tasks/smoke_accounts_spec.rb:214 failed on master with expect(result[:eligible]).to be true returning false — a real regression, not CI flake, blocking the required check on unrelated PRs. Fixed (PR #2775) by seeding a real HabitCompletion gap in rails_api/lib/tasks/smoke_accounts.rake, mirrored in .github/workflows/preview.yml per the three-source preview-provisioning gotcha below: repair_goal.habit_completions.delete_all followed by an unconditional create!(completed_date: 3.days.ago.to_date, streak_freeze_used: false) — delete-and-recreate every run, deliberately not a find-or-create, so a fixed completed_date can't drift into "already covered" the longer a fixture-provisioning PR stays open. If eligibility needs to reflect intent without a real completion history, StreakGapChecker itself needs to read the Goal-level fields — seeding around it just reproduces this trap elsewhere.
• Second recurrence, same shape, different source of truth (OBJ-3704): the same preview smoke seed carried a single habit goal purpose-built for streak-repair eligibility (habit_streak: 0, pre_break_habit_streak: 5) that also happened to satisfy every Goal-level field a naive reading of streak-freeze eligibility would check. But StreakFreezeCard.vue's state precedence (as of PR #3129's not-yet-merged phase-3-spend-a-freeze-affordance branch, confirmed by Roy's review) evaluates afterBreak = habitStreak === 0 && preBreakHabitStreak > 0 before it ever considers ready/ready-placed — so this goal's repair-eligible values (habitStreak: 0, preBreakHabitStreak: 5) locked it to the 'after-break' state and correctly suppressed the freeze CTA the smoke/UI-evidence tests needed. The real gate is streak_freezes_available, a persisted integer column (default 0) recalculated by exactly one method, Goal#update_habit_streak! (rails_api/app/models/goal.rb:251-267) — nothing in the smoke bootstrap ever called it, so no Preview smoke goal had ever gotten a nonzero value. Fix (PR #3138, open at time of writing): adds a second, dedicated "Smoke Habit Freeze Eligible" goal (habit_streak: 7, pre_break_habit_streak: 0, streak_freezes_available: 1, streak_freezes_used: 0, no habit_completions) to both rails_api/lib/tasks/smoke_accounts.rake (freeze_goal) and .github/workflows/preview.yml's inline seed script, rather than trying to make the repair-eligible goal serve both purposes — same pattern as the OBJ-3121 fix: a dedicated fixture per feature's real eligibility gate, not one goal overloaded across gates that read different sources of truth. streak_freezes_available: 1 was verified against what update_habit_streak! would actually compute for these inputs (earned_freezes = (7 / 7) + starter_freezes_granted(0) - streak_freezes_used(0) = 1) rather than picked as an arbitrary nonzero value. Regression guards on both sides, added by PR #3138 (not yet merged as of writing): rails_api/spec/lib/tasks/smoke_accounts_spec.rb's 'freeze-eligible habit provisioning (OBJ-3704)' context (creation, idempotency, self-healing refresh) and a new ionic_frontend/tests/unit/smoke-streak-freeze-provisioning-parity.spec.ts (rake task and preview.yml's inline script stay in sync on the goal name and every field).
• The general trap, stated once: when a smoke/fixture seed sets "the fields that look relevant" on a Goal record instead of driving it through the real code path (a check-in, update_habit_streak!, a completion write) or a feature-specific dedicated fixture, it will pass review and still fail silently for any feature whose actual eligibility gate reads a different column or association than the one seeded. Before adding a new eligibility-gated feature's smoke seed: find that feature's actual determining logic first (the service/component method that computes eligibility, not the Goal columns that look like they should), and either drive the seed through the real recalculation path or give it its own dedicated fixture goal — don't extend an existing goal that was purpose-built for a different feature's gate.

Production Clerk auth-config changes go through clerk-auth-config.expected.json, never the dashboard alone

• Production Clerk auth-config changes must go through a pull request that edits scripts/clerk-auth-config.expected.json, never a bare Clerk-dashboard edit. That file is the checked-in statement of what production should hold (_meta.warning), not what it currently does — .github/workflows/clerk-auth-config-drift.yml runs scripts/check-clerk-auth-config.mjs daily (0 13 * * * UTC) against both instances' live GET /v1/environment and flags any mismatch against the snapshot, or against staging (modulo staging_accepted_divergences, each with an owner issue, and staging_temporary_windows, a second, time-boxed exemption for an authorized divergence expected to resolve by a given date — an expired window fails loud exactly like an unrecorded one, so it can't silently extend grace). There is no legitimate-false-positive class here — a red run is always either a real out-of-band dashboard change or a stale snapshot. Never hand-edit the snapshot to match a live incident state just to turn a red check green — that defeats the guard entirely. See docs/architecture/authentication.md#clerk-dashboard-configuration. (OBJ-2609, OBJ-2755)

A stuck /factor-one with no password field isn't necessarily a lockout — Clerk auto-selects email_link as first factor

• A stuck /factor-one screen with no password field is not a lockout by default — it's Clerk's hosted <SignIn> auto-selecting email_link as the first factor for any account that has a password, on both staging and production. user_settings.attributes.password.used_for_first_factor is false on both instances (confirmed via GET /v1/environment, OBJ-2769) — attributes.password.enabled is true, but a password is never eligible for auto-selection as the first factor. Given only an identifier (no strategy), supported_first_factors comes back password, email_code (primary: true), email_link (primary: true), reset_password_email_code (primary: true) — identical shape on both instances — and the hosted <SignIn> component's own client-side logic picks email_link from among the primary: true options. No password field renders; the screen polls indefinitely for a magic-link click that may never come, which reads exactly like an account lockout. This is not specific to one account: OBJ-2729 first hit it against a disposable, username-less test subject and nearly misread it as a lockout; OBJ-2769 confirmed it generalizes to a normal account (has username, has password) on both instances.
  • How to check before assuming a lockout: read supported_first_factors off the sign-in attempt response, or probe GET /v1/environment → user_settings.attributes.password.used_for_first_factor directly — this is still the right immediate diagnostic, since the drift guard below only runs once a day. The daily drift guard does now track this field (PR #2510, same OBJ-2769 investigation) — scripts/check-clerk-auth-config.mjs's extractComparableFacts() reads attributes.password.used_for_first_factor alongside the .enabled/.required pairs (asserted in scripts/check-clerk-auth-config.test.mjs, pinned in scripts/clerk-auth-config.expected.json), so a live divergence on either instance surfaces on clerk-auth-config-drift.yml's next daily run — it just isn't a substitute for checking supported_first_factors in the moment.
  • Recovery: "Use another method" → "Sign in with your password" — the password path is available the whole time, just one non-obvious click behind the auto-selected magic-link screen.
  • Related: password.required stays false on production is a separate, deliberate setting — used_for_first_factor is a different attribute and was not touched by that decision. (OBJ-2729, OBJ-2769)

GoalType/GoalKind is intentionally integer-only (not a bug)

• GoalType/GoalKind is intentionally integer-only — it is the documented exception to the "never expose integer IDs" / PublicRecord rule — GoalType is a small fixed reference table (Habit / Milestone / Quantity) seeded at setup; it does not inherit PublicRecord, has no public_id column, and its GraphQL type (Types::GoalTypeType) exposes the integer primary key as id on purpose. The frontend queries (GOAL_KINDS_QUERY in constants/graphql/goals.js) and all callers (GoalForm.vue, GoalCreate.vue, CreateCommunityChallengeModal.vue) correctly key on kind.id — this is not a bug to "fix" by adding publicId. (CHALLENGE_GOAL_KINDS_QUERY was removed in #1004; both surfaces now use GOAL_KINDS_QUERY.) Audited under OBJ-785.

An if ENV['X'] truthiness guard silently accepts an empty string

• if ENV['VAR'] is truthy for "" — a wired-but-empty env var passes the guard and silently disables whatever it configures, with zero signal. In Ruby, only nil and false are falsy — an empty string is truthy. config.dsn = ENV['SENTRY_DSN'] if ENV['SENTRY_DSN'] (the pre-fix form of rails_api/config/initializers/sentry.rb) let a Cloud Run env var that was present in the container but never assigned a value (confirmed via gcloud run services describe ... --format="value(spec.template.spec.containers[0].env)" returning {'name': 'SENTRY_DSN'} with no value key) pass straight through. Sentry.init then "succeeded" with an empty DSN and dropped every event it tried to send, for 90+ days, with no error anywhere. This is a general trap, not specific to Sentry — any if ENV['X']/ENV['X'] && guard has the same hole; use .present? (or ENV.fetch('X', nil)&.strip&.presence) whenever blank-but-set must be treated the same as unset. scripts/validate-secrets.sh cannot catch this class of bug — it validates that a var is wired (present in the deploy manifest) and that a referenced Secret Manager secret exists; it has no notion of "wired but empty," and SENTRY_DSN is additionally listed in its ALLOWLIST array, which exempts it from every check the script does run. If a subsystem's health depends on a blank-vs-unset distinction, that check has to live in the app's own initializer, not in the deploy-time secrets validator. A known live second instance of this exact bug class exists, out of this fix's scope: rails_api/config/initializers/security_headers.rb:40 (policy += "; report-uri #{ENV['CSP_REPORT_URI']}" if ENV['CSP_REPORT_URI']) has the identical bare-truthiness hole — a wired-but-empty CSP_REPORT_URI would append a malformed empty ; report-uri directive to the CSP header on every response. Not fixed here; tracked separately. (OBJ-2821)

config/puma.rb is evaluated before ActiveSupport loads — its "safety" fix broke every boot

• rails_api/config/puma.rb runs under the raw puma binstub, before preload_app!'s deferred Rails boot ever loads ActiveSupport — any core-extension method (.present?, .blank?, etc.) reached for in this file raises NoMethodError on every single boot, 100% reproducible. rails_api/Dockerfile:56 (CMD ["bundle", "exec", "puma", "-C", "config/puma.rb"]) and the equivalent bin/start-worker.sh / bin/start-crono.sh invocations hand this file straight to Puma's own DSL parser — Rails, and everything it monkey-patches onto core Ruby classes, hasn't booted yet at that point. PR #2947 (ec33980f) swapped this file's two bare truthiness checks (if ENV['SOLID_QUEUE_IN_PUMA'], if ENV['PIDFILE']) for .present?, correctly recognizing the general "empty string passes a bare truthiness guard" bug class documented in the entry above — but .present? is an ActiveSupport core extension, unavailable in this file's execution context. Every enkidu-api-staging / enkidu-worker-staging / enkidu-crono-staging instance crashed on boot from the moment this merged (2026-09-06 17:48:35 UTC):

  config/puma.rb:41:in 'Puma::DSL#_load_from': undefined method 'present?' for nil (NoMethodError)
  bundler: failed to load command: puma (/usr/local/bundle/ruby/4.0.0/bin/puma)

  Cloud Run correctly refused to route traffic to the crashing revision — no live regression, but every staging deploy was blocked until fixed. This is the mirror image of the ENV-truthiness gotcha above, not a restatement of it: there, the bare check was the bug and .present? was the fix; here, in this one file only, the bare check was already correct for its execution context and .present? is what breaks it. The right fix for this specific file is a non-ActiveSupport-dependent blank check, e.g. ENV['SOLID_QUEUE_IN_PUMA'].to_s != '' / !ENV['PIDFILE'].to_s.empty? — never .present?/.blank?, no matter how idiomatic they look elsewhere in this codebase. General rule: before reaching for ActiveSupport sugar in any file, confirm it actually runs inside a booted Rails process — config/puma.rb (and, by the same reasoning, config/boot.rb if it's ever edited the same way) does not. Fixed: 68a2fef13 (PR #2955, merged 2026-09-06) replaced both .present? calls with the plain-Ruby form this entry recommends (rails_api/config/puma.rb:43,47); OBJ-3424 is closed. Guarded: 1f85ec11f (PR #2957, merged 2026-09-07, OBJ-3427) added a boot-smoke gate to .github/workflows/staging.yml:110-132 — before pushing the built image, it runs bundle exec puma -C config/puma.rb config/ci/puma_boot_smoke.ru -e staging against that exact image and blocks the push if the container isn't running 3 seconds later, so this class of bug (anything in config/puma.rb/config/boot.rb assuming ActiveSupport or any Rails-loaded state) is now caught before a broken image ever reaches the registry. (Root cause: OBJ-3424, PR #2947. Fix: PR #2955. Guard: OBJ-3427, PR #2957.)

One env var drives both Puma's thread count and ActiveRecord's pool size, with two different defaults, neither ever set

• PUMA_MAX_THREADS and DB_POOL_SIZE resolve through the same two-step fallback chain, but land on different hardcoded defaults if nothing is ever set — 3 for Puma, 5 for ActiveRecord. rails_api/config/puma.rb:30 (ENV.fetch('PUMA_MAX_THREADS') { ENV.fetch('RAILS_MAX_THREADS', 3) }) and rails_api/config/database.yml:22 (ENV.fetch("DB_POOL_SIZE") { ENV.fetch("RAILS_MAX_THREADS") { 5 } }) both check a specific var, then a shared RAILS_MAX_THREADS, then a hardcoded default — but the two files were never designed to converge on the same number in the fully-unset case. Every deploy/*.yaml manifest declares both vars explicitly today (v4.59 Phases 3–4), so this doesn't fire in a deployed environment — it's live wherever a manifest omits either var: local dev, a new environment, or a future service added without following docs/operations/rails-concurrency-capacity.md.
  • The compounding half: containerConcurrency: 80 (Cloud Run's own default) against a real thread count of 1–3 meant Cloud Run's autoscaler wouldn't add an instance until requests were already queued dozens deep behind the one thread actually available. deploy/service.production.yaml:64-70's comment states the mechanism directly: Cloud Run targets routing ~60% of containerConcurrency to an instance before starting a new one — at containerConcurrency: 80 with 1 real Puma thread, that threshold (~48 concurrent requests) is never reached by a thread pool that saturates at 1, so scale-out never triggers on thread exhaustion alone; requests just queue behind the single thread instead. Fixed in v4.59 Phases 3–4 by setting containerConcurrency explicitly to each service's resolved PUMA_MAX_THREADS in every deploy/*.yaml manifest, and guarded going forward by scripts/check-deploy-capacity.mjs (rule 2) — see docs/operations/rails-concurrency-capacity.md for the full knob reference, the guard's four rules, and the disclosed known limitation (production is still capped well below its measured connection peak — see that doc's Known limitation section, cross-linked from gcp-costs.md).
  • Related: ActiveRecord::DatabaseConnectionError's "username/password" message doesn't mean a credential problem below — same connection-pool-ceiling neighborhood, different failure shape (a capacity-exhaustion error misread as a credential error, rather than a sizing mismatch). (v4.59 Phase 5, OBJ-3753)

ActiveRecord::DatabaseConnectionError's "username/password" message doesn't mean a credential problem

• Rails/pg's ActiveRecord::DatabaseConnectionError renders the same generic "username/password" text for any connect-time PG::ConnectionBad — including Postgres refusing a connection for capacity reasons that have nothing to do with credentials. The exception's friendly message ("There is an issue connecting to your database with your username/password, username: enkidu.") is a blanket template, not evidence of a real auth failure — and Sentry's issue title surfaces only that template, never the underlying Postgres FATAL text. Confirmed on staging (OBJECTUVE-API-2Z/30/31/32, 2026-08-31): four of these fired in a 16-second window that fell entirely inside a Cloud Run revision cutover (enkidu-api-staging-01801-w22). First-pass triage read the title at face value and diagnosed a credential-propagation race — wrong. The real cause was one Caused by: line into Cloud Logging:

  Caused by: PG::ConnectionBad (connection to server on socket
  "/cloudsql/enkidu-488723:us-central1:enkidu-postgres/.s.PGSQL.5432" failed:
  FATAL:  remaining connection slots are reserved for non-replication superuser connections

  i.e. enkidu-postgres's connection-slot ceiling (at the time, shared across production, staging, every preview environment, and Cloud Run jobs) was briefly exhausted by the old revision draining and the new revision warming its pool at the same time — an ordinary deploy-cutover doubling, not a credential issue. Before assuming "username/password" in a Sentry title means a credential problem, pull the Cloud Logging entry for that timestamp and read the Caused by: line — that's where the real Postgres FATAL text lives; the Sentry title never carries it. See Connection-Pool Cliff in the incident-response runbook for the capacity angle. Update (OBJ-3416, 2026-09-06): staging no longer shares this ceiling — it moved to its own dedicated enkidu-postgres-staging instance, see the split entry above and Cloud SQL. enkidu-postgres today is shared across production, previews, and Cloud Run jobs only. (Diagnosed by Dave, OBJ-3029.) Related: one env var drives both Puma's thread count and ActiveRecord's pool size, with two different defaults, neither ever set above — same connection-pool-ceiling neighborhood, different failure shape.

Mailtrap's sandbox and sending APIs are separate products with no shared message store

• Mailtrap's Email Testing (sandbox) API and Email Sending API are two distinct products with no shared message store — a mailer-routing change on one side can silently invalidate a delivery check built against the other. GET /api/email_logs (domain-scoped, "sending-logs") only ever sees mail that actually left Mailtrap's live relay; GET /api/accounts/{account_id}/inboxes/{inbox_id}/messages (inbox-scoped, "sandbox") only ever sees mail captured by the Email Testing sandbox, which is never relayed anywhere. Staging routes its SMTP through the sandbox (config/environments/staging.rb, OBJ-2822); production routes through the live relay (config/environments/production.rb). Mailtrap::EmailLogs (rails_api/app/services/mailtrap/email_logs.rb) reads whichever product matches the current environment's SMTP target via an explicit sandbox_mode? split (Rails.env.staging?) — see docs/architecture/email.md § Verify path: sandbox vs. sending API for the full split. The trap: these two Mailtrap products also return differently-shaped message objects — sandbox messages have no message_id or status field, only subject — so a delivery-verification client written against one product's response shape silently fails against the other, total rather than intermittent, even with real data present. This is exactly how the v4.43 delivery canary's verify stage shipped unable to ever succeed on staging (OBJ-2939): it read the sending-logs API (wrong product for staging's sandbox-routed mail) with a parser hardened to the sending-logs Hash envelope ({"messages": [...], "total_count": N, "next_page_cursor": ...}) — which, per the OBJ-2958 addendum below, is not the shape the sandbox inbox returns. If you change which Mailtrap product an environment's SMTP relay points at, check every downstream API-based delivery check for the same assumption.

  • Addendum (OBJ-2951): the two products' list entries don't just differ in shape — they differ in the id field name itself. Sandbox-inbox entries carry id; sending-logs entries carry message_id. Mailtrap::EmailLogs#single_match_id_from (rails_api/app/services/mailtrap/email_logs.rb) read entries.first['id'] unconditionally on both paths, which worked by construction for sandbox (the only path OBJ-2939 live-verified) and was silently wrong for sending-logs — a gap invisible until production's Email::DeliveryCanaryVerifyJob executed for the first time ever in v4.7.0 and immediately regressed Sentry issue OBJECTUVE-API-2W. The same commit also called an unfilterable server-side param (find_by_rfc_message_id's bare message_id filter) that Mailtrap's own docs confirm was never a real filterable field — it silently ran unfiltered on every call. Both are fixed as of OBJ-2951: single_match_id_from now reads the field via a message_id_field helper keyed on sandbox_mode?, and the unfilterable route was removed entirely rather than "fixed," since no server-side filter for this field exists to fall back to.

  • The general pattern, not just this one bug: OBJ-2939's own stated acceptance bar — "a spec against a real recorded Mailtrap response body, since a hand-written fixture is what let defect 2 ship" — was satisfied for the sandbox fixture and silently not satisfied for the sending-logs fixture, in the same PR that named the failure mode. Nothing gated that PR on "does every code path this touches get the same live-verification treatment," so the exact defect class it was written to prevent reshipped one file over, inside the commit meant to close it out. When a fix's regression spec covers multiple structurally-parallel code paths (e.g. two products behind one environment split), every path needs the same live-or-documented-schema verification — verifying one and leaving a sibling on a hand-authored fixture is not a smaller version of the fix, it's an unfixed copy of the original bug. (Diagnosed by Dave, fixed by Codi, reviewed by Roy — full hypothesis log on OBJ-2951.)

  • Addendum (OBJ-2956, "H_page"): a third failure mode, distinct from shape and id-field-name — an unfiltered page-1 read against a shared, unpurged inbox silently misses a message that has fallen off the first page. Mailtrap::EmailLogs#find_by_subject_scan (rails_api/app/services/mailtrap/email_logs.rb) originally listed the target inbox/log with no query filter at all, trusting that a given day's canary message would still be on page 1 by the time the verify job's cron-triggered read ran. Nothing purges either the sandbox inbox or the sending-logs endpoint, so as entries accumulate across days, a given day's canary message can fall off page 1 entirely — producing a clean "zero subject matches" miss with the exact same signature as the id-field-name bug above (no exception, all retries exhausted, OBJECTUVE-API-2W regressed on staging 2026-08-25 through 2026-08-28). PR #2655 (commit 8f9b58a98, OBJ-2956) shipped two changes: the list request now passes Mailtrap's documented search query param (the canary token) on both the sandbox and sending-logs paths, and #parsed_list surfaces total_count alongside the message array so a zero-match result logs a diagnosable Rails.logger.warn instead of failing silently (spec coverage: rails_api/spec/services/mailtrap/email_logs_spec.rb). At the time, both were verified against WebMock fixtures only — no environment in the automation tier had Mailtrap API or GCP Secret Manager read access to run a real live-fire check against the actual inbox (tracked separately: OBJ-2958, held for human credential provisioning). Live-probed 2026-08-29 under OBJ-2958: on the sandbox inbox (staging) search does filter server-side — ?search=<token> returned exactly 1 result and a nonsense token returned 0 — so the search half of this fix is correct on that path. That probe was a one-off manual observation against a live inbox; nothing in the repo re-runs it, so its provenance lives in this prose and not in a green test — same caveat as the envelope finding below. It was still never what kept staging red; see the OBJ-2958 addendum below for what actually was. The sending-logs endpoint (production) behaves the opposite way — see the OBJ-2967 correction immediately below. The two findings are product-scoped, not contradictory: search filters the sandbox inbox and does not filter sending logs.

    • Correction (OBJ-2967, verified empirically against the live production Mailtrap account 2026-08-29 — the live-fire check this addendum's WebMock caveat above was waiting on, whether or not it ran through OBJ-2958 specifically; it overturns half of the "fixed" claim, not all of it): the search query param does not narrow the lookup on the sending-logs endpoint (production). This was measured on the sending-logs path only, and does not carry to the sandbox inbox, where OBJ-2958's live probe on the same day found the opposite (see the OBJ-2956 addendum above and the OBJ-2958 addendum below) — the two Mailtrap products differ here as they do in shape, id-field name, and envelope. ?search= and ?message_id= both return byte-identical, unfiltered result sets to a bare call — same total_count — so passing search buys no server-side filtering at all. What's still real: #parsed_list surfacing total_count and the zero-subject-match Rails.logger.warn are unaffected by this finding — that diagnosability half of the fix is correct and still working. What actually narrows results today: only client-side filtering on the returned subject field (find_by_subject_scan's own matches = messages.select { ... } line), and only within whatever single page that call already fetched. next_page_cursor is present in the response shape (rails_api/spec/services/mailtrap/email_logs_spec.rb) but nothing in this class reads it — no pagination is implemented, so a busy log can still push a canary's row off page 1 with no mitigation in place today. The H_page gap this addendum believed it had closed is still open on the sending-logs/production path: search never filtered server-side there to begin with, and no pagination-based fix was ticketed at the time this addendum was written (it named "OBJ-2986, v4.7.1 release-prep" as the tracking pointer — that issue was never actually created, and the gap sat as prose until Dave re-diagnosed it from scratch on OBJ-3051, which is now the real tracking issue for the pagination fix; see the "diagnosed-but-untracked follow-up" gotcha below for the general pattern).

  • Addendum (OBJ-2958): the list envelope differs by product too — and a hardening pass written off one product's live response can turn the other product's path into a guaranteed nil. Sending logs (production) return a Hash: {"messages": [...], "total_count": N, "next_page_cursor": ...}. The sandbox inbox (staging) returns a bare JSON array of message hashes, with no envelope at all. Both shapes were live-probed against the real API on 2026-08-29. OBJ-2939 read the sending-logs shape off the live API and hardened #parsed_list to body.is_a?(Hash) only — which made every staging lookup return nil at that type guard, before the subject scan and before the total_count warning OBJ-2956 added to diagnose exactly this could ever run. That, not pagination and not the missing search param, is the real root cause of OBJECTUVE-API-2W on staging. Three consecutive diagnostic passes missed it for the same structural reason: the only live-fire task in the repo, email_canary:live_fire, invents a synthetic token with no email_canary:send:<token> cache record, so DeliveryCanaryVerifyJob#perform returns at its record.nil? guard and never reaches Mailtrap at all — it proves the alert path and nothing about the lookup path, but reads like end-to-end coverage. The general rule: a "live-fire" task that cannot reach the subsystem under investigation is worse than no task, because it launders fixture-level confidence as live confidence. email_canary:live_fire_lookup is the missing half; #parsed_list now accepts both envelopes and labels its count total_count=N (server-reported grand total) vs returned_count=N (this page only) rather than conflating them into one misleading number.

  • A fix that resolves one issue's explicitly-flagged follow-up needs an explicit link back onto that issue's own thread, even when the fix itself ships through its own separate, fully-tracked issue. OBJ-2956 (the H_page fix above) ran through the normal Multica crew chain start to finish — Dave diagnosed it, Codi shipped it, Riley confirmed CI, Roy reviewed it, Vicki merged it (PR #2655, 2026-08-28T16:39:51Z) — so this was never work that escaped the tracked-issue system. The gap is narrower: OBJ-2956 directly answers a follow-up ("H4") that OBJ-2951 — a different, already-done issue — had explicitly flagged as open (Vicki's OBJ-2951 ship note deliberately withheld a Fixes trailer specifically so the issue would stay open until a live canary cycle confirmed it), and nothing ever commented that resolution back onto OBJ-2951's own thread — confirmed by reading OBJ-2951's full comment history, which ends 2026-08-27T21:04Z, a full day before OBJ-2956 was even created. The next scheduled staging-health-check autopilot cycle (OBJ-2959, 2026-08-28) came within one git log --all --grep archaeology pass of re-diagnosing H_page from scratch and dispatching a second, conflicting fix for work already done — purely because OBJ-2951's thread had no pointer to OBJ-2956. General rule: when a fix answers another issue's explicitly-flagged follow-up, comment the resolution back onto that other issue's own thread — not just the new issue that tracks the fix — even when the new issue is itself fully and correctly tracked.

  • Addendum (OBJ-3204) — the production-side sibling of the staging gap above: the weekly release-train cadence, not the Sentry dedup window, is what leaves a fix invisible this time. The OBJ-2956/OBJ-2958/OBJ-2962 material above is all staging-side — a fix landing between two once-daily cron cycles reads as still-broken because Sentry hasn't seen a confirmed-good run yet. This is a different mechanism, not another instance of the same one: master deploys to staging continuously on every merge, but production only deploys on a v* tag, cut by the weekly, human-gated Vicki — Weekly release train autopilot (Wednesdays 08:00 America/Chicago). A fix can sit merged, live on staging, and fully proven for up to 7 days before the next tag ships it to production — and for that whole window, production's own daily canary alert keeps firing on a bug that's already fixed, just not yet released.

    Worked example (OBJ-3147, diagnosed 2026-09-02). Production was running v4.7.0 (86a224f55, deployed 2026-08-26T19:50:48Z) — the commit before OBJ-2951's message_id_field fix landed. On that deployed code, Mailtrap::EmailLogs#single_match_id_from read entries.first['id'] unconditionally; production's sending-logs API rows carry the id under message_id, never id (the OBJ-2951 addendum above), so find_message_id returned nil on every single production run — 100% deterministic, not intermittent. Sentry OBJECTUVE-API-2W ("EMAIL CANARY - CANNOT VERIFY (production)") fired daily from first-seen 2026-08-25 through the 2026-09-02 diagnosis — 12 occurrences, zero confirmed deliveries ever recorded — while six fixes touching this exact file (1ca7e54af/OBJ-2951, 8f9b58a98/OBJ-2956, b452d611f/OBJ-2958, 7f78302a3/OBJ-2971, 67e0517d9+48eb955a8/OBJ-3051+OBJ-3093) sat merged and live on staging the entire time. Staging read healthy throughout, because sandbox_mode? routes it through the corrected code path — the exact split that makes this gap invisible to staging-only monitoring.

    The diagnostic move. When a production-only Sentry issue looks chronic — regressed/unresolved across many consecutive cron cycles, with staging quiet on the same signal — don't re-diagnose from scratch first. Diff the deployed production SHA against master for the issue's culprit file:

    gcloud run services describe enkidu-api-production --region us-central1 --project enkidu-488723 \
      --format 'value(spec.template.spec.containers[0].image)'   # image tag = deployed SHA
    git diff --name-only <deployed-sha>..origin/master

    If the culprit file shows up in that diff, the fix likely already shipped to staging and is waiting on the next release train — not a fresh bug.

    Automated check (OBJ-3204 Task 1, in progress as of this writing). The Vicki — Release readiness roundup autopilot's Tuesday preflight is gaining exactly this diff-and-intersect as a readiness gate — for every unresolved production Sentry issue, cross-referencing its culprit against files changed since the deployed SHA, and reporting matches under an "At risk — fix already merged, unreleased" section (advisory only: it raises urgency to ship, it does not block the train). Once landed, check the autopilot's live description (multica autopilot get c49cad3b-02c4-4d94-ab0d-8c21d72b3c8c) for the exact gate wording rather than assuming this paragraph's summary is current.

    Cross-references: OBJ-3147 (this worked example), OBJ-2951 (the root-cause fix cited above).

A Mailtrap hard bounce on any address suppresses every future send to it, from any sender, with no application-visible signal

• One hard bounce puts an address on Mailtrap's account-wide suppression list — every subsequent send to that address, from any sender, is silently dropped before any SMTP attempt. This is a distinct failure mode from the sandbox-vs-sending-API gotcha above: that one is about reading the wrong product's delivery record; this one is about Mailtrap refusing to attempt delivery at all, and it isn't scoped to whatever sender/token caused the original bounce — it blocks the recipient address account-wide. There is no error, no retry, and nothing visible in Rails, Sentry, or Mailtrap's own send logs for the suppressed message — the send call returns success-shaped output regardless. Nothing in this codebase reads Mailtrap's suppression list. Mailtrap::EmailLogs (rails_api/app/services/mailtrap/email_logs.rb) only ever resolves per-message delivery status via GET /api/email_logs, scoped to whichever token made that specific send — it has no code path that queries https://mailtrap.io/suppressions or equivalent, for any address, on any schedule. The only way this has been caught so far is a human manually opening the Mailtrap dashboard.
  • Two production incidents, same mechanism, found the same way, seven weeks apart. OBJ-2967 (canary@objectuve.com): the address was never a real mailbox, so the daily delivery canary's first send (2026-08-27) genuinely hard-bounced, which suppressed the address — the next two canary sends (08-28, 08-29) never made an SMTP attempt at all. Not three independent failures; one real bounce plus two local rejects. OBJ-2981 (demo@objectuve.com, the shared production demo account): unrelated address, same mechanism — hard-bounced 2026-07-12, silently suppressed for roughly 7 weeks before anyone noticed. Both were found only because Josh happened to open Mailtrap's suppression-list UI directly, not through any automated check.
  • The compounding tradeoff: Mutations::RequestMagicCode always returns success: true, by design, which makes a suppressed send structurally invisible to the end user. RequestMagicCode (rails_api/app/graphql/mutations/request_magic_code.rb:12-26) is the passwordless "email me a code" path wired into the live sign-in UI (ionic_frontend/src/views/SignIn.vue — openMagicCode, the "Code sent — use the newest one in your inbox." confirmation text at line 337). It looks up User.find_by(email:), calls UserMailer.magic_code(...).deliver_later, and unconditionally returns success: true — line 24's comment states the reason: "don't reveal whether the email exists," an anti-enumeration measure that is a legitimate tradeoff on its own. But when that always-succeed response intersects with Mailtrap's suppression behavior, the result is a UI that shows "code sent" to a real, existing user whose email is on the suppression list, and then delivers nothing — forever, with zero signal to the user or to any operator not manually checking Mailtrap. No feature flag or rollout gates this path; it's live and unconditional. This exact intersection was root-caused on OBJ-2981, investigating whether demo@objectuve.com's suppression had any live blast radius — it does, via this path, independent of the weekly-digest job (which turned out to be a separate, unrelated no-op for all production users at the time).
  • If you add a periodic suppression-list sweep or an allowlist-based watchdog, this is the gap it needs to close — extending the existing email_canary health-check pattern (docs/operations/alerting.md) from "verify our own canary's delivery" to "verify no watched address is suppressed" would have caught both incidents without a human opening the dashboard.

first_sign_in is a real badge every user has, and it hid three empty-state surfaces for months

• first_sign_in ("Day One") is not internal bookkeeping — it's a real, designed, common-rarity badge (Achievements::BadgeCatalog::BADGES, rails_api/app/services/achievements/badge_catalog.rb:10, its own name and icon, present since the catalog's creation, 3ff0e6260, 2026-04-29) that Mutations::SyncUser#resolve awards unconditionally on every user's first sign-in — before they've taken any other action. Achievements::ComputeUserRankService#unlocked_count (compute_user_rank_service.rb:33-38) counts it alongside all 21 BadgeCatalog::BADGE_KEYS, so unlockedCount >= 1 for every user who can physically reach the Achievements page — it can never be a valid "has this user done anything yet" signal.
• The empty-state predicate used unlockedCount === 0 anyway, so it never fired for a real user, for months. AchievementsHero.vue's isEmpty branch ("A quiet shelf, ready to fill."), Achievements.vue's Inner Drawer ("first six" preview), and NextOnTheShelfCard's "Begin here" call-to-action were all designed, written, and shipped — and all three were dead code in production, with no error, no failing test, and no signal anywhere, because nothing could ever reach unlockedCount === 0. The only thing in the whole system still asserting these surfaces work was a QA smoke fixture (ACHIEVEMENTS_EMPTY_EMAIL) — and by the time its own drift got fixed (below), the underlying product bug was still there, because the fixture's assertion was correct and the predicate it was testing was wrong. The fix (OBJ-3879): a new earnedCount — badges earned through action, i.e. all badge keys except first_sign_in — gates the empty state instead. unlockedCount stays byte-for-byte unchanged everywhere else (rank/XP, the badge grid, SideMenu.vue, share cards); Day One still counts toward it and still renders unlocked in the grid even when earnedCount === 0. See docs/features/achievements.md for the full contract.
• General lesson: a smoke fixture asserting a UI state renders is not evidence the state is reachable by a real user — it only proves the fixture's own account can reach it. Here the fixture was itself drifting (see the badge-key-list entry immediately below) for two rounds before anyone asked why a QA-only account needed special handling to see a state every new user should see by default. When a "make the fixture pass" fix and the underlying product behavior disagree about what "empty" means, that disagreement is usually the real bug.

Three disagreeing badge-key lists — GAMEPLAY_EXEMPT_BADGE_ACTIONS is the shared source for "all badges except Day One"

• Three separate lists have, at different times, tried to represent "the set of badge keys" and disagreed with each other: Achievements::BadgeCatalog::BADGE_KEYS (badge_catalog.rb:37, 21 entries — the actual catalog, source of truth), UserAction::GAMEPLAY_EXEMPT_BADGE_ACTIONS (user_action.rb:100, 20 entries — BADGE_KEYS minus first_sign_in, by design), and a hardcoded array in SideMenu.vue:366-371 (20 entries — missing founding_member, not first_sign_in; a separate, pre-existing, unrelated bug tracked as its own issue, not part of this fix). A user holding the Founding Member badge sees a different unlocked count in the side menu than on the Achievements page as a direct result.
• UserAction::GAMEPLAY_EXEMPT_BADGE_ACTIONS is the shared, derived source going forward for "every badge except Day One." It was introduced (OBJ-3879 round 2, PR #3304) to fix the QA fixture's purge scope, and is guarded at every badge-granting call site — UserAction.generate and the founding_member path that bypasses generate — so a gameplay_exempt account can never re-acquire a badge no matter which call site grants it. The earnedCount predicate (see the entry above) derives from this same constant rather than hand-writing a fourth list. If you need "all badges except Day One" anywhere else, reuse GAMEPLAY_EXEMPT_BADGE_ACTIONS — do not write a fifth list. Drift between exactly these kinds of parallel lists is what caused two prior rounds of this same incident (OBJ-3879 rounds 1 and 2) before round 3 found the real predicate bug.
• ONCE_PER_USER_ACTIONS (user_action.rb:84, 21 entries) is a fourth, intentionally different list — don't conflate it with the other three. It answers "which actions are DB-uniqueness-constrained," not "which are badges": it includes streak_comeback (not a badge in BadgeCatalog::BADGE_KEYS at all) and excludes founding_member (a badge, but community-scoped and re-grantable per qualifying community, so it isn't once-per-user). The first (round 1) fix to the QA fixture purge scope used this list by mistake, which is what round 2 corrected.

AI Workforce & Agent Runner

Agent Runner is a separate standalone service

• Agent Runner is a separate service — agent_runner/ is a standalone TypeScript Express app, NOT part of the Rails API. It communicates via webhook (HMAC-signed POST to /webhooks/ai-workforce). No database access.

Claude Agent SDK package name changed

• Claude Agent SDK package name — Use @anthropic-ai/claude-agent-sdk (renamed from @anthropic-ai/claude-code). Do NOT use the old package name.

AI Workforce access is admin-or-marketing, then crew-scoped for marketing

• AI Workforce access is not admin-only — AI Workforce GraphQL queries and mutations require require_ai_workforce_access! (rails_api/app/graphql/concerns/authorization.rb), not require_admin!. That check passes for admin users and for any user with the marketing role (AdminRole), because admin: true on the user row is only set for super_admin — require_admin! would reject a marketing-role user before the role check ever ran.
• Marketing-role callers are further scoped to their own crew — require_ai_workforce_crew_access!(employee) is called after require_ai_workforce_access! at every AI Workforce query/mutation call site that resolves a specific AiEmployee (or an AiRun/AiArtifact belonging to one). Admins pass automatically; a marketing-role user only passes when employee.crew == 'marketing' — any other crew (or no crew) raises FORBIDDEN. List queries (aiEmployees, pendingArtifacts, recentArtifacts) apply the same scoping as a where(crew: 'marketing')/where(ai_employees: { crew: 'marketing' }) clause for non-admins rather than a per-record check.
• crew is a free-text string, not an enum — AiEmployee#crew (added alongside the AI Workforce Ops Board) has no VALID_KINDS-style validation; 'marketing' is only meaningful because the authorization concern hardcodes it. See the Crew entry in docs/architecture/ddd/02-ubiquitous-language.md.

Brand-voice filter is a separate query() call, not a skill

• Brand-voice filter is NOT a loaded skill — It runs as a separate query() call in the Agent Runner, not as part of the main run's skill context. This prevents context dilution.

MCP scoping via employee's mcp_servers column

• MCP scoping — Each employee's mcp_servers column controls which tools they can access. An employee without 'github' in their mcp_servers array cannot invoke GitHub tools.

Autonomy promotion is a DB validation, not a UI warning

• Autonomy promotion is a DB validation — Not a UI warning. validate :promotion_requirements_met on AiEmployee enforces promotion criteria at the model level.

AiArtifact delivery is kind-routed in DeliverArtifactJob

• AiArtifact delivery is kind-routed in DeliverArtifactJob — draft_post/code_patch → GitHub PR, email_template → Mailtrap sandbox, report → Slack + docs, recommendation → docs, social_post → Buffer scheduled drafts (Ai::SocialPublisherClient, X, LinkedIn, and Facebook — Facebook ships as link-preview posts only, Route A, no generated image; Instagram and the generated-image Route B remain deferred, see docs/product/og-image-facebook-scope.md). facebook in channels requires a non-blank link_url or the artifact fails permanently before ever reaching Buffer. Buffer drafts are scheduled, never immediate-published — a human still confirms in the Buffer UI. delivered_at is set iff every requested channel was genuinely posted; a channel already succeeded on a prior attempt is never re-submitted on retry (no Buffer idempotency key), and malformed input (no channels, or all channel content blank) fails permanently rather than retrying.

Shadow-locked roles capped below full autonomy

• Shadow-locked roles cannot be promoted above shadow autonomy — AiEmployee::SHADOW_LOCKED_ROLES (customer_support_drafter, social_publisher — i.e. Ally and Remy) is enforced both on update (promotion_requirements_met) and on create (ally_block_check), so the cap can't be bypassed by creating a new record at a higher autonomy level.

Frontend (Vue/Ionic)

Frontend GraphQL uses graphql-tag, not .graphql files

• GraphQL query syntax — Frontend uses graphql-tag (gql templates), NOT .graphql files.

Component templates: Pug legacy, HTML new, both valid

• Component templates — Legacy uses Pug, new uses HTML. Both valid — match existing file's style.

Don't use Date.now() in client-side insight IDs

• Don't use Date.now() in client-side insight IDs — Dismissal state is keyed to the ID; a timestamp-based ID resets on every module reload, so the dismiss record in localStorage never matches the regenerated ID. Use stable deterministic strings encoding content branch, not generation time (e.g., 'achievements-tip-first-badge' not `achievements-tip-${Date.now()}`). The 24-hour TTL in useAiInsights.ts handles expiry; the ID doesn't need to encode time.

Headless UI fragment components throw on class without as=

• Headless UI components render as a fragment/template by default — passing class (or any attr) to one that wraps multiple children throws Passing props on "template"! — @headlessui/vue components like <TabGroup>, <Disclosure>, <RadioGroup>, <Listbox>, and <Combobox> default to as="template" (no wrapping DOM element). When you write <TabGroup class="mt-6"> and the component renders multiple children (e.g. TabList + TabPanels), Headless UI has no single root element to forward class/style/listeners onto and throws at render time: Error: Passing props on "template"! The current component <TabGroup /> is rendering a "template". However we need to passthrough the following props: - class. In Vue this is a render-time throw, not a console warning — it trips the nearest error boundary (admin dashboard's ErrorBoundary.vue → "Something broke."), so the whole page fails to render, deterministically, for every visit. Fix: add an explicit as="..." so the component renders a real element that receives the attrs — e.g. <TabGroup as="div" class="mt-6">. (TabList/Tab/TabPanels already default to real elements — div/button — so they take class fine; the trap is specifically the fragment-rendering wrapper components.) This was OBJECTUVE-ADMIN-2 (EmployeeDetailView.vue, fixed in #1028) — and it was invisible until the admin dashboard's Sentry DSN was wired up (#1020), because admin crashes weren't being captured at all.

TypeScript ?? isn't falsy-safe — empty strings pass through

• TypeScript ?? (nullish coalescing) is NOT falsy-safe — it only gates on null/undefined, so empty strings and the literal string "null" pass straight through — Env vars sourced from a shell pipeline frequently arrive as "" (unset-but-present) or the string "null" (a jq artifact, see above). process.env.SMOKE_BASE_URL ?? 'http://localhost:5173' keeps the empty string and the app boots against an empty base URL. Fix: use || when the value may be an empty string or "null": process.env.SMOKE_BASE_URL || 'http://localhost:5173'. Reserve ?? for values where 0/false/'' are legitimately distinct from "unset" — which env-var strings almost never are. (Source: OBJ-651, Roy's review.)

@vitejs/plugin-legacy removed — nomodule bundle is dead code

• @vitejs/plugin-legacy was removed in OBJ-983 (PR #1195) — the nomodule bundle was dead code — The current browserslist floor (Chrome 79+ / Firefox 70+ / Edge 79+ / Safari 14+ / iOS 14+) is uniformly ESM-capable; every supported browser ignores <script nomodule>. If a future browser-matrix change reintroduces a pre-2021 cohort, re-add the plugin AND add esbuild: { supported: { destructuring: true } } to both the top-level esbuild block and optimizeDeps.esbuildOptions — esbuild ≥ 0.27.7's compat table incorrectly reports Chrome 64 / Safari 12 / Firefox 67 / Edge 79 as not supporting destructuring, causing the build to abort with Transforming destructuring to the configured target environment is not supported yet. See vite/vite#22225 and esbuild/esbuild#3743. The esbuild.supported.destructuring = true flag is the documented escape valve from esbuild's author — not a freeze — and is safe to apply even when not strictly required (as it is today, with the legacy plugin absent).

A hung dynamic-import during boot bypasses the error UI

• A hung (not rejected) dynamic-import Promise during SPA boot silently bypasses even an existing, working error UI — main.ts's bootstrap() wraps await import('./App.vue') in a try/catch whose catch shows a "Something interrupted the launch" recovery screen with a Reload button — but that only fires when the import rejects. During the v4.1.2 production rollback, the post-deploy Playwright smoke gate caught desktop-chromium users stuck on a permanently blank <div id="app"></div>: every other page asset returned 200, except assets/acquisitionAnalytics-Ct_QAe_c.js, which came back status: -1 — a request that never got a response at all, not a 404. Because the import Promise never settled, the try block never finished and the catch block's recovery UI never got a chance to run either. Root cause was most likely CDN edge propagation lag: Firebase Hosting finished deploying at 18:58:20 UTC, and chromium — the first of the three device smoke projects to run — hit the hang at 19:00:48 UTC (~2.5 min later); mobile-safari (6.6 min later) and mobile-android (12 min later) fetched the identical asset successfully. Fix (shipped): importWithTimeout() races import('./App.vue') against a 20s timeout via Promise.race — a hang now rejects and reaches the existing recovery UI instead of freezing forever. A window.addEventListener('vite:preloadError', ...) listener was added alongside it for the related-but-distinct "stale chunk hash after a redeploy" failure mode — that listener's own unconditional-reload hazard is now a separate, fixed entry below (vite:preloadError's reload could strand the app on Chromium's offline interstitial), not covered by this entry's fix. General rule: any try/catch around a boot-time or route-time dynamic import() only handles the rejected case by default — if the import can hang instead (flaky network, CDN lag, a proxy that swallows the response), race it against an explicit timeout, or the catch block is dead code for that failure mode. (Root cause: OBJ-1500, PR #1587; diagnosed via gh run view/Playwright trace inspection, not live repro.)

vite:preloadError's reload could strand the app on Chromium's offline interstitial

• A vite:preloadError handler that calls window.location.reload() unconditionally can fire while the browser is genuinely offline — the reload's own document fetch then fails the same way the triggering import did, and Chromium substitutes its internal chrome-error://chromewebdata/ interstitial for the app, with no code path (app-side or Chromium-side) that ever re-navigates once connectivity returns. Root-caused via coach-reconnect-grace.spec.ts (ionic_frontend/tests/smoke-playwright/specs/coach-enneagram/): a forced /cable close + page.context().setOffline(true) → setOffline(false) toggle reproduced identically across five independent CI dispatches (chromium + mobile-android, initial attempt and retry #1) as a "total miss" — no online transition observed, no /cable socket reconstructed, toBeVisible() on the resumed line timing out. The diagnostic tell that broke the case open: page.evaluate() reads of session-storage-backed lifecycle diagnostics threw SecurityError: Failed to read the 'sessionStorage' property from 'Window': Access is denied for this document during the offline hold, with page.url() reading chrome-error://chromewebdata/ at the moment of the throw — that specific URL is Chromium's own internal "no internet" interstitial (an opaque-origin document, hence the SecurityError on any storage read), proof a real top-level navigation was attempted and failed while offline, not a hung/silent app. Fix (shipped): the handler was extracted to handleVitePreloadError() (ionic_frontend/src/lib/vitePreloadRecovery.ts), registered from main.ts, which now checks navigator.onLine and only reloads immediately when true; when offline, it defers the reload to the next online event instead of firing blind. Regression-guarded by ionic_frontend/tests/unit/lib/vitePreloadRecovery.test.ts (reload-when-online, no-synchronous-reload-when-offline, reload-on-subsequent-online). Diagnostic tell to reuse: page.url() === 'chrome-error://chromewebdata/' (or a SecurityError on a storage/DOM read) during any Playwright setOffline(true) window is the signature of a real failed navigation, not an app-code hang — distinguish it from the hung-dynamic-import entry above, which never leaves the app's own document at all. Known nit, not yet fixed: repeated vite:preloadError firings while still offline each register their own one-shot online listener (vitePreloadRecovery.ts); harmless today (the first reload() call wins), but a future edit to this file should add an "already armed" guard. (Root cause: Dave's diagnosis on OBJ-4047, PR #3397; instrumentation by Tess, fix by Codi, reviewed by Roy.) Follow-on collision found on the very next real exercise of this fix (OBJ-4006, PR #3404): deferring the reload to the next online event stops the offline-interstitial hazard above, but that same event also drives any reconnect-dependent UI flow — see the entry below.

OBJ-4047's deferred vite:preloadError reload can win a race against an in-progress reconnect flow

• OBJ-4047's fix correctly stopped vite:preloadError's reload from firing while genuinely offline — but deferring it to the next online event puts it in a race against any other online-triggered recovery, and the reload wins. useNetworkStatus.ts's online listener calls reconnectCable() to reopen ActionCable the instant connectivity returns (ionic_frontend/src/composables/useNetworkStatus.ts:38, imported from @/apollo-client) — the same event vitePreloadRecovery.ts's deferred branch was waiting on. Root-caused on coach-reconnect-grace.spec.ts's first genuinely unconfounded R4 run after OBJ-4047 shipped: the spec's own sessionStorage-backed [reload probe] diagnostic caught vite:preloadError firing while offline (expected — a lazy chunk import failing during the test's fault injection), then beforeunload/pagehide firing 69–228ms after that attempt's online event — a full-page reload, not an app-logic bug. probeInstallCount going 1→2 between the pre-drop and post-reopen checkpoints confirmed a fresh document, not a hung script: the reload tore down the live ActionCable consumer and CoachChatPanel.vue's disconnectedSinceMount tracking before reconnectCable()'s reopen (or the resumed-line UI depending on it) ever got a chance to complete, so the freshly-loaded page's reconnect gate never saw the original disconnect and could never pass. Fix (shipped): the deferred branch now waits RELOAD_DEBOUNCE_MS (5s, exported for tests) after online fires before reloading, giving an in-progress online-triggered flow a window to finish first — a heuristic, not a guarantee, per the code's own comment (ionic_frontend/src/lib/vitePreloadRecovery.ts). Roy's review round added a second layer: at the moment the debounce elapses, armDeferredReload() re-checks navigator.onLine — still online → reload (plus a Sentry.captureMessage('Deferred vite:preloadError reload firing after online', 'info'), closing a prior signal gap where a real production occurrence was indistinguishable from an ordinary page load); offline again → re-arm a fresh { once: true } listener rather than reloading blind or giving up silently, keeping OBJ-4047's original guarantee (only ever reload while genuinely online) intact through a flaky-connectivity window. Regression-guarded by ionic_frontend/tests/unit/lib/vitePreloadRecovery.test.ts: debounce-before-reload, no re-trigger/extension on a rapid second online, no reload if offline again at the debounce boundary, and re-arm-then-reload on the next online after that. General rule: a "defer this side effect to the next online event" fix needs to be checked against every other listener already keyed off that same event, not just the failure mode the fix itself targets — the fix's own success (the reload no longer fires while offline) is what let it collide with a second online-driven flow that could only race it once it started working correctly. (Root cause: Dave's diagnosis on OBJ-4006, PR #3404; instrumentation from OBJ-4047's own reload probe, fix by Codi, reviewed by Roy.)

Fallback/magic-link sessions can inherit a stale cached public_id

• Fallback-token/magic-link sessions can silently inherit a stale public_id from storageState/localStorage — syncClerkUser() (ionic_frontend/src/router/index.ts) has a fast path that skips re-syncing with the backend if Session.decoded()?.id is already present in localStorage, on the assumption that a cached id means "already synced this session." That assumption breaks for fallback-token and magic-link sessions: they authenticate via a SessionToken header, not Clerk's own session state, so a cached objectuve_user_public_id can be left over from a different account's session (e.g. Playwright's storageState reuse across test runs, or any real device that switches accounts without clearing localStorage). Downstream, AchievementWatcher.vue (mounted globally in App.vue, not gated by route.meta.onboarding) polls USER_QUERY with that stale id every 10s (pollInterval: 10000); since rails_api/app/graphql/resolvers/user_queries.rb#user checks current_user.public_id == id and raises FORBIDDEN otherwise, every poll fails for the full duration of the session — continuous GraphQL errors, Sentry noise, and wasted round-trips, not a one-time glitch. Fix (shipped): the fast path now also checks isTokenAuth = !!(Session.getFallbackToken() || Session.getMagicLinkToken()) and only short-circuits when existingId && !isTokenAuth; token-authenticated sessions always re-run SYNC_USER_MUTATION, which resolves the real current_user and overwrites the id via Session.setPublicId() before AchievementWatcher ever queries. General rule: a "sync once per session, trust the cache after that" fast path is only safe when the cache key (here, public_id) is scoped to the same identity as the active auth mechanism — always audit fallback/alternate auth paths for the same assumption, not just the primary (Clerk) one. (Root cause: Dave's diagnosis on OBJ-1605, split out as OBJ-1607, PR #1667.) Recurred on the primary Clerk path too (OBJ-2889): the identical hazard shape, but on syncClerkUser()'s other short-circuit — the module-level let userSynced = false, set true once on the first successful sync and never reset anywhere. On a same-bundle-lifetime Account A → Account B switch (sign-out via SPA router push, sign-in as B, no reload), if (userSynced) return fires immediately on B's first protected navigation, before existingId/isTokenAuth are even read — setSynced() never runs again, and currentUserId stays pinned at A's id for the rest of the session, so every currentUserId-gated query (not just AchievementWatcher.vue's poll) hits the same FORBIDDEN public_id mismatch this entry already documents. Roy's review of PR #2616 traced it from two independent pre-existing doc citations (syncClerkUser.test.ts's own note that the flag "only resets on module re-instantiation," and v4.37-flow-integrity-interstitial-arbitration-ROADMAP.md); Dave confirmed it via static trace, no reset path existed anywhere in production code. Fix (shipped): resetUserSyncedFlag(), exported from router/index.ts, resets both userSynced and a new diagnostic-only lastSyncedClerkUserId; called from Session.logout() and Session.destroy() alongside the Phase 2a subscription resets above. The previously-silent early return also gained a console.warn + Sentry breadcrumb (sync-clerk-user-skipped-different-identity), but only when the live Clerk identity actually disagrees with lastSyncedClerkUserId — the common same-identity-skip case stays silent by design. General rule (restated): any module-level "already did this" flag that gates a per-session sync — not just a cached id — needs to be checked against the same question: does anything reset it on sign-out, or does it survive an SPA-router account switch with no page reload? Search for = false / = true assignments with no corresponding reset before trusting a "sync once" guard is session-scoped rather than bundle-lifetime-scoped. (Root cause: Roy's review + Dave's diagnosis on OBJ-2889, PR #2616.)

Absent USER_QUERY user rendered identical to a real zero streak

• An absent USER_QUERY user used to render byte-identical to a genuine zero streak — a confident 0/0/blank 30-cell heatmap, no loading or error affordance, for either a failed query or one that never fired — Dashboard.vue's const user = computed(() => userResult.value?.user ?? {}) collapsed "we don't have data" and "the user really has a 0-day streak" into the same shape, and the header pill + StreakDetailsModal both rendered from user unconditionally, outside the dashboard's own loading/hasError gate. Two independent silences fed this: hasError (Dashboard.vue) only tripped when both USER_QUERY and GOALS_QUERY failed, so a lone USER_QUERY failure with a healthy GOALS_QUERY (goal cards rendering fine) never surfaced; and the query is gated on enabled: userSynced.value && !!currentUserId.value, so a null/stale currentUserId meant the query never fired at all — no error, no loading, nothing. A production probe on this exact account (current_streak=3, credited entirely from GoalEvent/MoodLog activity with zero habits) proved the backend was correct the whole time; the UI was lying about a state it never actually observed. This is the render-layer half of the OBJ-1607 stale-public_id family above — PR #1667 fixed one upstream trigger (a cached id causing USER_QUERY to raise FORBIDDEN), but the null-user render path stayed unguarded, so any future cause of an empty USER_QUERY would have reproduced this identically. Fix (shipped): a new streakQueryState/streakState computed (Dashboard.vue, deliberately separate from the existing dashboard-wide hasError, which is untouched — preserving the OBJ-740/765 loading-gates-hasError guard that keeps a mid-flight errorPolicy: 'all' error from cycling GoalsSection's fade animation) derives an honest loading / unavailable / ready tri-state: ready the instant real data exists (so a background refetch never flickers the pill), unavailable on a query error or the never-fired case, and a loading stuck past 8s degrades to unavailable instead of hanging forever. The streak pill and StreakDetailsModal (new state prop, default 'ready') now render a skeleton or an honest "Couldn't load your streak" + retry instead of a fabricated zero; a blank heatmap is reserved for a confirmed zero, never "we don't know." Known gap (not fixed here, filed as OBJ-1730): the one-silent-auto-retry on first entering unavailable calls refetchUser(), which is a no-op for the query-never-fired case — that query was never enabled in the first place, so refetching it doesn't re-establish currentUserId. A real fix needs a useClerkSync.ts-level resync export that avoids importing the router module directly (that import broke 35 existing tests, since the app router isn't mocked in the test suite). General rule: a computed that unconditionally falls back (?? {}, || [], || 0) on query data is only safe if every consumer sits behind a gate that distinguishes "no data yet" from "confirmed empty" — check both the error path and the query's own enabled gate, not just one. (Root cause: Josh's production probe + Orion's re-scope on OBJ-1711, PR #1767.) Recurred twice more (OBJ-1968, PR #1946): the same shape shipped independently in two more views. Coach.vue's USER_QUERY rendered result.value?.user ?? {} before binding loading/error — a user who had already finished the Enneagram assessment was told to start one. EnneagramTypeDetail.vue's clampType(dominantType ?? 1) rendered Type 1 as the reader's own result whenever the type couldn't be resolved (bare /coaching/type, or a malformed ?type= param). Both are fixed the same way as Dashboard.vue above — an honest loading/error/ready (Coach.vue) or loading/error/empty/ready (EnneagramTypeDetail.vue) tri-state gating all render, using the new shared QueryStateCard component for the loading/error/empty affordance instead of a bespoke skeleton per view. EnneagramTypeDetail.vue also added an own-vs-reference split, since ?type= can point at someone else's type: ownership only ever upgrades from the conservative reference default to own, never back. Guardrail (shipped): ESLint rule no-fabricated-query-state (ionic_frontend/eslint-rules/no-fabricated-query-state.js, internally QSTATE-GUARD-1) is wired into ionic_frontend/eslint.config.js as local/no-fabricated-query-state: 'warn' and runs in CI via the existing npm run lint frontend-lint job — no separate workflow wiring needed. It flags two shapes: a useQuery destructure that binds result without loading or error, and a computed() ?? {} / ?? <literal> fallback on query data (.value access on the left side). ?? null and ?? [] are deliberately exempt — neither renders as a confident non-empty result on its own, and ?? [] gates a component with its own loading/error props in sanctioned cases like EnneagramHistoryList.vue; a ?? <literal> feeding straight into a comparison (AnalyticsView.vue's cardState() isEmpty predicates) is exempt too, since the fallback never escapes as a displayed value. Calibration is pinned by an ESLint RuleTester spec (no-fabricated-query-state.test.js): fires on the exact pre-fix Coach.vue/EnneagramTypeDetail.vue shapes, clean on their post-fix equivalents and on AnalyticsView.vue. Warn-level, not a hard fail — it already flags Dashboard.vue's own separate, out-of-phase-scope user = computed(() => userResult.value?.user ?? {}) (the original occurrence documented above, left untouched by this phase) as expected, not a false positive: real pre-existing debt the rule now makes visible going forward.

Fabricated progress fallback (|| 100) recurred across 8 sites

• The fabricated-fallback family recurred a third time, in goal progress, across eight independent sites, before anyone connected it to the two entries above — targetAmount || 100 (frontend) and the byte-identical target_amount.presence || 100 (backend) both coerced an absent target amount into a known denominator of 100, so a goal with no target — but a completed 1/1 roadmap — rendered a confident 0% on its dashboard card and detail hero while its own roadmap bar read 100% · 1/1 steps on the same screen (production goal 2aua0oA, reported by Josh). Confirmed sites spanned both frontend and backend, including: Dashboard.vue, Goals.vue, Goal.vue (the detail hero and the AI Coach's setContext watcher — the fabricated number didn't just render wrong, it got spoken back to the user), GoalListItem.vue (dead code, deleted rather than fixed), goal_queries.rb#compute_avg_progress (the visible "Avg Progress" KPI tile), and calculate_completion_by_category.rb#progress_fraction (the Analytics "Completion by category" bars, which also feeds CalculateMonthInReview's leading-category pick) — eight independent copies by Josh's count in the OBJ-1978 issue thread. Fix (shipped): one shared derivation per side — useGoalProgress.ts / GoalProgressCalculator — implementing a single precedence (habit completionRate → roadmap milestone % → currentAmount / targetAmount → not tracked) that returns percent: null/nil for not-tracked, never a fabricated 0. GoalSummaryType#avg_progress changed from Float, null: false to nullable to carry the same contract onto the GraphQL surface. null is this codebase's not-tracked signal — UrgentGoalCard.vue:20's pre-existing v-if="progress !== null" was the template this fix generalized, not a new invention. Known gap, not fixed by this PR: GoalAtAGlance.vue's "Progress" tile (ionic_frontend/src/views/Goal.vue:242-243) still reads raw targetAmount/currentAmount directly instead of useGoalProgress's output, so it can still disagree with the card/hero/roadmap bar on a habit or roadmap-only goal. General rule: the same ?? {} / || 0 / || N audit applies to any derived metric, not just user/streak state (the first two instances of this family) — before adding a fallback to a query field, ask whether the fallback value is a genuine default or a stand-in for "we don't know," and if the latter, propagate null instead. (Root cause: OBJ-1978, PR #1949.)

Supporter theme overrides need auditing against fixed-color consumers

• CSS custom properties overridden by a supporter theme must be audited against every non-theme-chrome consumer before shipping a new theme override — --accent/--accent-light (design_system/css/tokens.css) were dual-purposed: the generic, intentionally-themeable UI accent variables that ionic_frontend/src/theme/supporter-themes.css overrides per supporter color theme (e.g. html[data-theme="ocean"] sets --accent: 200 80% 50%), but also reused directly by the streak/gamification "stones" standard/rare tier (StoneBlock.vue:101-109) and 10 sibling files (FlameIcon.vue, WeekRibbon.vue, StreakHalo.vue, PathToNextBar.vue, XpChip.vue, RankChip.vue, streakAnimations.ts, CelebrationCinematic.vue, CelebrationToast.vue, CelebrationBalanced.vue, SparkLayer.vue) as their fixed brand color, meant to stay constant regardless of the active app theme. Supporter themes never touched --accent before that feature existed, so the collision was latent — enabling theme overrides wired up the cascade without auditing which non-chrome components already consumed the token. Fix (shipped): dedicated --streak-accent/--streak-accent-light tokens added to design_system/css/tokens.css, deliberately not referenced anywhere in supporter-themes.css, with the streak/gamification standard/rare references swapped onto them. General rule: before adding a new supporter-theme (or any theme) CSS override for an existing token, grep every consumer of that token for uses outside generic UI chrome — a token that's safe to theme in one place can be silently load-bearing as a "fixed brand color" somewhere else. Follow-up (shipped): StreakHalo.vue:26's epic-tier secondary halo — a lower-severity partial bleed on the same root cause, left out of this fix's scope and tracked separately — also read hsl(var(--accent) / 0.35) for its secondary glow (primary glow uses --gold, unaffected); swapped onto hsl(var(--streak-accent) / 0.35) in OBJ-1760, PR #1793. (Root cause: OBJ-1755, PR #1788.)

Global tab-bar padding reserves space on tab-bar-less routes

• App.vue's global tab-bar padding reserve applies to every ion-content under 768px — including meta.public routes that never render BottomTabBar — App.vue:238-242's non-scoped rule sets ion-content { --padding-bottom: calc(60px + env(safe-area-inset-bottom, 0px)) } for every ion-content on a mobile viewport. But BottomTabBar.vue:34's showTabBar = computed(() => !route.meta.public) means the tab bar never renders on a meta.public route (src/router/index.ts), so any such route reserves ~60px + bottom-safe-area (94px on a device with a home indicator, per the shipped fix's own comment) of dead scroll space for a nav bar that isn't there. Currently affected (renders ion-content, no local override): /sso-callback (SsoCallback.vue), /public_goal/:goalId (PublicGoal.vue), /u/:username (PublicProfile.vue), /design-system (DesignSystem.vue), /join-team/:code (views/teams/JoinTeamView.vue). Not affected despite being meta.public: /invite/:token (PartnerInvitePage.vue) doesn't render ion-content at all, so the global selector never matches it — don't assume every meta.public route needs the fix below. Fix: a scoped local override, ion-content { --padding-bottom: 0; } in a <style scoped> block, as shipped on /sign-in//sign-up (SignIn.vue, SignUp.vue). General rule: correcting the global rule at the source (e.g. scoping it to routes that actually render the tab bar) would fix all of these at once but touches every authenticated screen's bottom spacing — treat as a separate, deliberately-scoped follow-up, not a drive-by fix. (Root cause/fix: OBJ-2020, PR #1969.)

min-h-screen inside ion-content overflows under viewport-fit=cover

• min-h-screen (100vh) inside ion-content overflows under viewport-fit=cover — index.html:11-12 sets viewport-fit=cover, so 100vh (min-h-screen) is the full physical screen including the status bar and home-indicator strip, while ion-content's inner scroll viewport is smaller than that by construction — a 100vh child inside a sub-100vh scroll container overflows. min-h-full (100%) is the correct unit; that's what shipped on /sign-in//sign-up (SignIn.vue:4, SignUp.vue:4). Confirmed remaining instances actually inside ion-content (reproduce the identical overflow — same fix applies): SsoCallback.vue:4, MeetCoach.vue:75. Also use min-h-screen but sit outside any ion-content (this specific overflow mechanism doesn't apply the same way — verify before assuming the fix carries over unchanged): GlobalErrorBoundary.vue:2 (as of OBJ-2775, App.vue nests two instances — an outer one wrapping the full chrome block (SideMenu, ion-router-outlet, AiCoachModal, BottomTabBar, AchievementWatcher, LegendaryCelebration) and an inner one wrapping only ion-router-outlet, so a routed-content error still tears down just the page while a chrome-sibling error now also gets caught; rendered only as the error fallback in either instance) and PartnerInvitePage.vue:127 (its own root <div> — no ion-content anywhere in the file). General rule: inside ion-content, always use min-h-full, never min-h-screen/h-screen/100dvh — the percentage unit is the one that matches the shrunk scroll viewport. (Root cause/fix: OBJ-2020, PR #1969.)

JSON.parse('null') returns null without throwing

• JSON.parse(x) returns null for the input "null" without throwing — a try/catch around only the parse does not protect the property access on the next line — NewHabitToast.vue's notification watcher did try { details = JSON.parse(n.detailsJson) } catch { continue } followed by details.action_name outside the try block. UserNotification#details_json (rails_api/app/models/user_notification.rb) returned the JSON string "null" for any notification created with details: nil — mood_reminder was the one live caller — and JSON.parse("null") doesn't throw, it successfully returns the value null, so the catch block never fired and the very next line crashed on null.action_name for every logged-in user, regardless of any feature flag. Fix: filter by a discriminant (the notification's kind) before parsing, not a wider try/catch — NewHabitToast.vue now skips any notification whose kind isn't integration_auto_checked_in/integration_proposed before touching detailsJson at all, matching the kind-first pattern already used in CoachCheckInSection.vue/CheckInPromptCard.vue. The root cause was closed at the source too: UserNotification#details_json changed from details.to_json to (details || {}).to_json, so it can never emit the string "null" again for any caller, current or future. Pair this with: a component mounted in global chrome (e.g. App.vue) is not "behind a feature flag" just because the feature it belongs to has one — the mount site itself needs its own v-if. NewHabitToast shipped with no gate at all on its App.vue:59 mount despite connected_apps_enabled existing; App.vue's AllyRequestToast at line 56 (v-if="allyRequestFlowEnabled") is the established precedent to match, but it wasn't followed on the first attempt. A second call site with the identical unguarded shape (useNotifications.ts, live-push path) was found and closed in the same fix rather than left as a deferred, unverified risk. (Root cause: OBJ-1982 attempt 2 rollback, live crash OBJECTUVE-CLIENT-B8, PR #2024; fixed in attempt 3, PR #2034, commit 506d03e39.)

Tailwind's calc() needs its own class string, not raw CSS

• calc(100%-24px) (no space around the minus) only resolves correctly through Tailwind's own class-string normalization — copy it into a raw CSS rule or a scoped <style> block and the browser silently produces mask-image: none, no error. Tailwind's JIT compiler splits and reassembles an arbitrary-value expression like [mask-image:linear-gradient(to_right,#000_calc(100%-24px),transparent)] before emitting the declaration, so the un-spaced - parses fine there. Plain CSS requires whitespace around a calc() operator (calc(100% - 24px)); without it, the browser's CSS parser treats the whole calc() as invalid and drops the declaration — and mask-image: none renders as "no mask, fully opaque," not a visible error. Verified directly: pasting the unspaced form into a <style> block produces exactly this silent no-op. Fix: keep the mask expression as a Tailwind arbitrary-value class ([mask-image:linear-gradient(...)]), never move it into raw CSS or a scoped <style> block — and if it ever needs to leave a class attribute, add the spaces first. Reference implementation: CoachingAccordion.vue:85. (Source: Desi's UI-SPEC addendum on OBJ-2451.)

An unenumerated third-party auth status falling through to the connectivity message

• An unenumerated third-party auth status silently fell through to a "check your connection" message — telling the user the wrong thing with full confidence, on a working password, account, and network. useClerkNative.signInWithPassword() (ionic_frontend/src/composables/useClerkNative.ts:136-182) only special-cased Clerk's needs_first_factor status; any other non-complete result threw a plain string-only Error, and SignIn.vue's catch block had no way to tell "Clerk returned a real, unhandled status" apart from "the request never reached Clerk at all" — both landed on the same generic branch: "Can't reach the sign-in service. Check your connection and try again." A live staging simulator run surfaced needs_client_trust (Clerk's device-verification challenge) hitting exactly that branch — password, account, and connection were all fine; the message told the user to check the one thing that wasn't broken. This was latent on production (device verification is off there today), and would have become a total native-password sign-in outage with a misleading message the moment device verification is turned on — a standard Clerk security recommendation (OBJ-2532).
• Two status names, one underlying cause: needs_second_factor and needs_client_trust aren't independent failure modes to separately design for — per Clerk's own docs, needs_second_factor is the legacy response name for the same Device Trust challenge needs_client_trust names in newer custom flows; which name a given call sees depends on the requesting app's Clerk dashboard config, not on whether the account actually has a second factor enrolled. Confirmed against Clerk's docs plus two live probes on the same day against the same staging instance — the app itself got needs_second_factor, a hand-built FAPI probe got needs_client_trust with an identical email_code challenge shape (OBJ-2532 Task 4 investigation).
• Fix (shipped): signInWithPassword now throws a structured ClerkSignInIncompleteError carrying signInStatus for any non-complete result (useClerkNative.ts:163-176) instead of a plain string-only Error. SignIn.vue's catch block branches on that structured property first, through a lookup with an explicit default arm (copyForSignInStatus(), SignIn.vue:856-859) — every status, including one never seen before, renders an honest fail-closed message with a recovery affordance (the existing "Sign in with an email code instead" flow). Updated (OBJ-2690): the connectivity branch is no longer reserved for "no HTTP response at all" — it's now also the deliberate fail-open destination for an unretryable 5xx (a real HTTP response Clerk did send), per the entry below. A genuine 4xx rejection never reaches it either way. See Authentication § Native Password Sign-In Status Contract for the full contract.
• General rule: when branching on a third-party status string, a lookup/switch with an explicit default arm that fails closed (not open) is the only shape that's correct for statuses the vendor hasn't invented yet. An if/else if chain that ends in "assume it's the most common failure" silently misclassifies every future status as whatever the last else branch says — here, a real API response read as a network outage. (Root cause: .planning/quick/20260814-auth-path-audit/, OBJ-2532, PR #2348.)
• The native Apple/Google sign-up path had the identical trap, closed separately (OBJ-3033): this entry's fix (above) was scoped to signInWithPassword only. useClerkNative.signInWithApple()/signInWithGoogle() (useClerkNative.ts:456-541, :547-628) had their own unhandled-status branch — any non-complete sign-up result other than the specially-handled legal_accepted gap threw a plain, untyped Error, and useNativeOAuth.handleNativeOAuth()'s catch (useNativeOAuth.ts:333-377) had no .status property to tell that apart from a dropped connection, so it rendered the same "check your connection" message. Apple's App Store review of build 4.4.1 (116) hit this on an iPad Air 11" (M3): the Apple plugin response only carries givenName/familyName (appleSignIn.ts:15-19) on a user's first authorization with the app, clerkFrontendApi.createSignUpWithToken() never forwarded either field to Clerk, and production Clerk requires first_name/last_name (scripts/clerk-auth-config.expected.json) — so a repeat-reviewer Apple ID's sign-up hit missing_requirements and read as a network failure. (Field-forwarding itself is unfixed and separately tracked as OBJ-3053, backlog — narrows how often this fires, doesn't replace the fix below.) Fix (shipped): the throw now carries a structured ClerkSignUpIncompleteError.signUpStatus (useClerkNative.ts:46-48, thrown at :528-529 Apple / :615-616 Google), tagged to Sentry at the throw site (clerk.native.sign_up_status, closing the same Traceable-but-not-Actionable signal gap this family has hit before), and useNativeOAuth's catch branches on it first via copyForSignUpStatus() (useNativeOAuth.ts:103-106) with a fail-closed default arm — same shape as copyForSignInStatus() above, currently reusing the existing approved OAUTH_INCOMPLETE_MSG copy pending dedicated Desi copy (OBJ-3052, backlog). See Authentication § Username Collection — the "Do not extend Clerk's instance-level 'Username' requirement" paragraph — for the doc that predicted this exact failure shape as a hypothetical before it fired for real.

A 400 that means the operation succeeded — native session_exists reported as a connection failure

• A Clerk 400 { code: 'session_exists' } on the native Apple/Google in-process sign-in path was reported to the user as "Apple didn't connect. Try again." — the opposite of what the response means. session_exists is Clerk saying a live session already exists on this client — i.e. the sign-in already succeeded, most commonly via a double-tap retry, a slow-network retry landing after the first attempt completed, or an app-resume race. useClerkNative.signInWithApple()/signInWithGoogle() let the error propagate uncaught to useNativeOAuth.handleNativeOAuth()'s generic catch, which mapped every non-cancellation throw to ERROR_MSGS[strategy]. The user saw a fabricated failure on a successful sign-in and, having no other signal, retried — reproducing the identical 400 every time, an unbreakable loop from the user's side. Captured on-device (iPad12,1): attempt 1 → 200, session persisted; attempt 2 (user retry) → 400 session_exists — independent proof attempt 1 had already succeeded server-side.
• Fix (shipped): both in-process paths now call a shared recoverExistingSession(currentToken, strategy) (useClerkNative.ts) on session_exists instead of throwing: reload GET /v1/client, persist the rotated token first, resolve a candidate session id (last_active_session_id → in-memory sessionId → secureStorage), and adopt only when clerkFrontendApi.isSessionActiveOnClient() reports status === 'active' — never from id-membership in client.sessions alone, since ended/replaced/expired sessions remain in that array too. On adopt, the function returns normally and the caller reaches the same success path as an ordinary sign-in. On any fail branch, a typed ClerkSessionRecoveryError.recoveryOutcome (no_candidate | no_active_session | reload_failed) drives a tagged Sentry.captureException ('clerk.native': 'sessionExistsRecovery', 'clerk.native.recovery': <outcome>) and the user-facing string becomes "You're already signed in — close and reopen the app to continue." — never the connectivity message.
• This is a distinct producer from the OBJ-2077 sign-out-teardown session_exists (clerk-native-api-gotchas.md's "Write side: mutating the session" section) — same status code, unrelated root cause; don't assume one fix covers the other.
• Device-verification gap: Apple was device-verified (the on-device double-tap trace above). Google's in-process recovery is unit-test-only — blocked on OBJ-2528 Task 0.
• General rule: a non-2xx status from a third-party auth provider is not automatically an error condition from the user's perspective — check what the specific code actually means before mapping it to generic failure copy. Same family as An unenumerated third-party auth status falling through to the connectivity message just above, but the inverse failure mode: that entry is about an unhandled status defaulting to the wrong message; this one is about a handled status whose meaning was read backwards. Full mechanism: Clerk Frontend API gotchas § A session_exists 400 on the in-process Apple/Google exchange means the sign-in already succeeded. (OBJ-2659, PR #2504.)
• Naming update (OBJ-3473): ERROR_MSGS above no longer exists under that name. The same terminal-4xx trap this entry documents — a definitive Clerk rejection reported as a connectivity failure — recurred a second time via google_one_tap_token_invalid, and the fix generalized past the one session_exists special-case: useNativeOAuth.handleNativeOAuth()'s catch now branches on isTerminalClerkError(e) / retryExhausted / a positively-tagged transport failure / a fail-closed default, in that order, and the renamed TERMINAL_REJECT_MSGS map supplies only the first arm's copy. See An unenumerated third-party auth status falling through to the connectivity message for the sibling status-axis fix and useNativeOAuth.ts for the current four-arm shape.

A retry-exhausted 5xx and a terminal 4xx throw the same error shape

• clerkFetch()'s single throw site (ionic_frontend/src/services/clerkFrontendApi.ts:298-315) throws the identical ClerkApiError shape for a terminal 4xx and for a 5xx that already exhausted MAX_RETRIES — catching a different error class won't tell them apart, only reading status will. The retry guard immediately above the throw (response.status >= 500 && retries > 0 recurses instead of falling through, clerkFrontendApi.ts:283-285) means control only reaches the throw with a 5xx once the retry budget is spent — err.retryExhausted = response.status >= 500 (clerkFrontendApi.ts:304) records that redundantly so a caller doesn't have to re-derive it from status alone.
• isTerminalClerkError(err) (clerkFrontendApi.ts:130-134) is the discriminator — true only for 400 <= status < 500. It's the only correct way to tell "Clerk evaluated this request and rejected it" (terminal, fail closed) apart from "Clerk, or the network, failed before evaluation was possible" (transient, fail open). A network-level failure (clerkFetch's outer try/catch rethrows unmodified) carries no status at all and also reads as non-terminal — the same fail-open branch has to handle both a used-up 5xx and a connection that never reached Clerk.
• Three call sites currently branch on this: useClerkNative.init() clears a stale session on a 4xx but stays signed in through a 5xx (useClerkNative.ts:51-83); useClerkNative.signInWithPassword() throws a separate, structured ClerkSignInIncompleteError for a non-complete 200 response — a different failure shape entirely, not an HTTP error (useClerkNative.ts:136-182); and SignIn.vue's passwordLogin() catch block chains incomplete-status, then isTerminalClerkError (429, then any other 4xx), then fail-open, in that fixed order (SignIn.vue:898-936).
• General rule: don't add a new Clerk Frontend API caller that regexes the thrown Error's message or assumes a 4xx from string content — read err.status via isTerminalClerkError() instead. The message string's shape (`Clerk API ${method} ${path} → ${status}${suffix}`) is identical across every status band. See Clerk Frontend API gotchas § A retry-exhausted 5xx and a terminal 4xx throw the same error shape and Authentication § Native Password Sign-In Status Contract for the full contract. (OBJ-2690, PR #2442.)

A || fallback on the object doesn't guard its nullable fields

• someObject || { firstName: 'User', ... } only fires when someObject itself is absent — it does nothing when the object exists but its individual fields are null. SideMenu.vue's user = computed(() => userResult.value?.user || { firstName: 'User', lastName: '', admin: false }) guarded the object, so it never ran for the far more common case: a real, synced user whose firstName/lastName are null because their Clerk profile carries no given/family name — every email/magic-code signup, since SyncUser#backfill_from_clerk (rails_api/app/graphql/mutations/sync_user.rb) only sets each field if data['first_name'].present?. The very next line, `${user.value.firstName} ${user.value.lastName}`, stringified those nulls straight into the UI as the literal text "null null," and the same string fed the avatar URL, rendering initials "NN." Confirmed across nine independent render sites (SideMenu.vue, GoalEventCard.vue, CommunityGoals.vue, CommunityMembers.vue, BetterTogetherInterstitial.vue, CommunityAlliesSection.vue, Community.vue), each with its own inline template literal — no shared helper existed. One site (CommunityMembers.vue) also had a dead || 'Anonymous' fallback that could never fire, since a template literal is never falsy. Fix (shipped): a single displayName() helper (ionic_frontend/src/helpers/displayName.ts) that checks each nullable field individually — both names → "First Last"; one → that one, no stray space; neither but username present → username; otherwise a generic fallback ("Friend", overridable per call site) — with all nine sites routed through it and the dead fallback removed. Avatar-URL call sites additionally wrap the resolved name in encodeURIComponent (an unescaped & previously corrupted the query string). General rule: an ||/?? fallback on a container only protects against the container being absent — checking the container's individual fields for null needs its own guard, one level down. Same family as the ?? {} and JSON.parse('null') entries above; this is the third recurrence of "the fallback guards the wrong thing." (Root cause/fix: OBJ-2533 addendum, PR #2350.)

Router navigation before Ionic's page-transition settles corrupts the router outlet

• Calling router.replace() from an onMounted async callback the instant an awaited call resolves can race Ionic's own page-transition bookkeeping — if the transition hasn't finished settling the view, the navigation corrupts the router outlet's visibility state and leaves the page frozen, with no exception, no console error, and no Sentry event. SsoCallback.vue's onMounted (pre-fix) awaited authStore.completeOAuthRedirect() and called router.replace(redirectTo) as soon as it resolved — nothing gated the navigation on Ionic having actually finished settling the view's transition. When the async auth call resolved before that transition settled (most reproducible on the Android Custom-Tab → app foreground return), the user was left staring at the "Signing you in…" spinner permanently, with force-quit as the only exit. This failed silently wrong, not empty: a 3,548-line device logcat capture had zero matches for Exception in commit, printIonError, Uncaught, handlePageTransition, or ion-router-outlet, against a positive control proving the console channel was recording — the corruption happened below any layer that logs. Fix (shipped): the navigation is now gated on onIonViewDidEnter having fired (SsoCallback.vue:54-56, the entered promise resolved from the onIonViewDidEnter callback), with a bounded fail-open — DID_ENTER_TIMEOUT_MS = 2_000 (SsoCallback.vue:63) — that proceeds anyway and reports a Sentry captureMessage (level warning, tag clerk.native: ssoCallback.didEnterTimeout) if the timeout elapses (SsoCallback.vue:74-89); onMounted now awaits waitForViewEntered() before router.replace() (SsoCallback.vue:100-101). The fail-open exists because onIonViewDidEnter isn't guaranteed to fire — Ionic's transition() can fail to settle, and the WebView can be suspended mid-handoff during the same Custom-Tab return that triggers the underlying race. Known residual risk (accepted, not fixed): the fail-open only helps when the transition is merely slow. If it's genuinely stuck, navigating at the 2s timeout races the same in-flight transition as navigating at t=0 and can still reproduce the frozen-spinner symptom — accepted because it's now rare, bounded, and observable via the Sentry report, versus the prior guaranteed, silent hang. General rule: any router.replace()/router.push() fired from an onMounted callback on an Ionic page — especially one reached via an external redirect (OAuth callback, deep link, Custom Tab return) — needs to be gated on the page's own transition having settled (onIonViewDidEnter), not just on whatever async work preceded it resolving; a bounded fail-open with Sentry visibility is the accepted mitigation for the case where the lifecycle hook never fires. (Root cause/fix: OBJ-2578, PR #2359.)

A component used in a template but never imported rendered nothing, with no lint or build error

• A Vue SFC referencing a component that was never imported silently renders nothing — no lint error, no build error, no console warning. OBJ-2026 shipped a <Plus> icon reference with no matching import; Vue's runtime component resolver found no match and rendered no icon, and nothing in the pipeline (lint, build, or console) surfaced it. The same shape recurred on the very first run after the guardrail below went in: Settings.vue's desktop header used <ion-button> without IonButton in its @ionic/vue import — caught immediately, fixed in the same change (OBJ-2032).
• Guardrail (shipped): vue/no-undef-components (eslint-plugin-vue) is enabled at error level in ionic_frontend/eslint.config.js, scoped to src/**/*.vue, and runs in CI via the existing npm run lint step in ci.yml's frontend-lint job — no new workflow wiring. Error level, deliberately unlike local/no-fabricated-query-state and local/no-unawaited-feature-flag above (both 'warn' because of 18+ pre-existing violations): this rule had zero violations repo-wide after the one-line Settings.vue fix, so error-level cost nothing, and it's the only level that works here — the lint run already emits ~1036 warnings, so a warn-level rule would have been functionally invisible and would not have caught the original <Plus> bug.
• When to add to ignorePatterns: only for a component registered globally by a plugin, never imported per-file by convention — the shipped config's ignorePatterns: ['router-link', 'router-view', 'RouterLink', 'RouterView'] covers Vue Router's own globally-registered components (~38 <router-link> usages per Orion's OBJ-2032 scoping measurement, none importing RouterLink per-file). Never add a component to ignorePatterns to silence a genuinely missing import — that defeats the guard's purpose.
• Two known blind spots — coverage does not extend past these:
  • <component :is="..."> dynamic-resolution sites are invisible to this rule (~32 per Orion's OBJ-2032 scoping measurement) — the referenced component can still be undefined at runtime with no lint signal.
  • lang="pug" templates are invisible to vue-eslint-parser, which this rule depends on to resolve template references — it cannot check inside a Pug template at all. ionic_frontend/src has zero Pug templates today, but Pug remains valid for legacy files (see "Component templates: Pug legacy, HTML new, both valid" above) — a new Pug template would silently drop out of this guard's coverage.
• Gate-strength note (resolved): frontend-lint (the job that runs ESLint) is a required status check — its display-name context, "Frontend Lint + Guards (ionic_frontend)", is present in ruleset 17927598's required-status-checks list alongside vitest's (confirmed via gh api repos/objectuve-softworks/enkidu/rulesets/17927598 on 2026-09-07; see "Splitting a required-check job silently drops its gate strength" in the CI & GitHub Actions section above for the history — this was a real gap after OBJ-1942, closed some time before 2026-09-07). A vue/no-undef-components failure blocks merge. (Root cause/fix: OBJ-2032, commit 64836b175.)

A resolved NavigationFailure is invisible to try/catch — check the return value too

• Vue Router resolves rather than throws on a failed navigation — router.replace()/router.push() resolve to a NavigationFailure object for aborted (type 4), redirected (2), duplicated (8), and cancelled (16) navigations, so a try/catch around the call only ever sees a genuine rejection and is structurally blind to all four resolved-failure cases. A caller that does try { await router.replace(target) } catch { report() } and nothing else reads as fully guarded — it awaits, it catches — but a resolved failure produces no thrown error, so the catch block never runs and the call site looks identical to a real success. App.vue's stale-consent recovery path (router.replace('/sign-in') in handleConsentAccept's staleConsent branch) was exactly this shape until OBJ-2661: awaited-and-caught, but never inspecting what await actually returned. Fix (shipped): App.vue's replaceOrReport(target, condition) helper (App.vue:274-298) awaits router.replace(), checks the resolved value for truthiness — router.replace() resolves undefined on success, so any truthy resolution is a NavigationFailure — and reports each failure mode separately per the repo's exception-vs-observation convention: a rejection gets Sentry.captureException(navErr, { tags: { 'clerk.native': ${condition}Rejected } }), a resolved failure gets Sentry.captureMessage('[App] navigation did not complete', { level: 'warning', tags: { 'clerk.native': ${condition}Failed } }) — captureMessage/warning because a returned NavigationFailure is a non-exception state observation, not a caught exception. All three router.replace() call sites in App.vue (stale-consent recovery, handleConsentAccept, handleConsentDecline) route through this one helper, tagged handleConsentAccept.staleConsentRouterReplace{Rejected,Failed}, handleConsentAccept.routerReplaceRejected, and handleConsentDecline.routerReplaceRejected respectively. Don't reach for isNavigationFailure without checking the installed version first: in the pinned vue-router@^5.2.0 (ionic_frontend/package.json:105), isNavigationFailure brands failures with a module-private Symbol() not publicly exported, which makes it impossible to construct a fixture failure in a spec that mocks vue-router — a plain truthiness check on the resolved value is complete and correct here, and is what ships. General rule: an awaited router.replace()/router.push() needs its resolved return value inspected, not just a surrounding try/catch — a catch-only guard is not a substitute for that check, and both failure modes should be reported since a "handled" branch that silently absorbs one of them is not actually handled. Known gap (not fixed here): useNativeOAuth.ts's post-sign-in router.replace() (useNativeOAuth.ts:297-310) is awaited and try/caught but still doesn't inspect the resolved value — same blind spot, different file/review surface, tracked separately as OBJ-2693 (backlog). (Root cause/fix: OBJ-2661, PR #2443.)

Ionic canDismiss attribute order can permanently strand a modal on success

• Vue patches v-bind-forwarded attribute object keys in insertion order, so the order attributes appear in the template determines the order Ionic's Stencil @Watch('isOpen') observes their values — a canDismiss binding placed after is-open can permanently strand a modal that just closed successfully. AppModal.vue sets inheritAttrs: false and forwards everything via v-bind="$attrs" onto <ion-modal>. App.vue's handleConsentAccept() holds consentSubmitting = true across the await, waitForNextFrame(), and the navigation (replaceOrReport() → router.replace(), since OBJ-2661), releasing it only in its finally. The old binding was :can-dismiss="!submitting" placed after :is-open — so on a successful Apple sign-up consent accept, isOpen flipped to false while canDismiss was still false. Ionic's canDismiss gates programmatic isOpen=false dismissal and does not retry — isOpen has no second transition, so the overlay stayed rendered on top of /dashboard, and tapping the modal's button again produced a misleading "Couldn't finish. Check your connection and try again." on a working network, account, and an already-successful request. The unit suite stubs IonModal as a plain <div>, so canDismiss enforcement is structurally untestable there — this shipped with an all-green suite. Original fix (shipped, order-dependent, now superseded): the first fix bound :can-dismiss="!submitting || !open" before :is-open in the template — the || !open clause kept the real guard (no backdrop dismiss mid-request, i.e. while open && submitting) while letting the programmatic close land. That made the whole guard order-fragile: can-dismiss had to stay literally the first attribute in template source, enforced only by a source-order regression test in this one file, nothing enforcing it anywhere else. Fix (shipped): AgreeToTermsModal.vue now binds :can-dismiss to a canDismiss() function ((data?, role?) => boolean) instead of a boolean. Ionic's checkCanDismiss() (@ionic/core/.../modal.js) calls it live — typeof canDismiss === 'function' ? canDismiss(data, role) : canDismiss — at its own dismiss-check time inside dismiss(), which itself awaits an internal lock and dismissNestedModals() first, so the call lands well after any Vue prop/DOM patch for that render has settled. That reads current props.submitting/props.open off the closure rather than a value snapshotted onto the element earlier, which removes the attribute-order dependency entirely — :can-dismiss and :is-open can now appear in either order. Verified against two things in node_modules: @ionic/vue's defineOverlayContainer (the inline-component path used here renders h('ion-modal', {...props}), and Vue sets canDismiss as a real DOM property rather than a stringified attribute, because the Stencil-generated custom element exposes it as one) and @ionic/core's modal-interface.d.ts, which types canDismiss as boolean | ((data?, role?) => Promise<boolean>) — the function form is first-class API, not an undocumented escape hatch. The regression test that used to assert template SOURCE ORDER by regex (anchored as /^\s*:can-dismiss=/m, since a bare indexOf(':can-dismiss') also matches the explanatory comment above the binding and would pass against known-bad code) is gone — order no longer matters — replaced by a test asserting the bound value IS a function and that invoking it reproduces the same guard (false while open && submitting, true once open is false). General rule: any guard+trigger prop pair (canDismiss/isOpen and friends) forwarded through v-bind="$attrs" is order-fragile as a boolean binding — prefer the function form so the guard is evaluated fresh at dismiss time instead of patched in a hopeful order. (Root cause/fix: OBJ-2602, PR #2385.)
• The parent side of this coupling, not just the component side: a composable or view that owns both an open ref and an in-flight ref must clear the in-flight flag before (or atomically with) the open flag — a component-side || !props.isOpen-style disjunct is belt-and-braces tolerance for a transient contradictory pair, not a licence to keep emitting it. useCollectiveGoalOptIn.ts's onConfirmOptIn and onAcknowledgePrivacyContract briefly held (isOpen=false, loading=true) on their success paths — optInOpen.value = false (resp. privacyOpen.value = false) landed before optInLoading.value = false (resp. privacyLoading.value = false) in the finally — which CollectiveGoalOptInModal.vue's canDismiss() (and TeamPrivacyContractView.vue's equivalent) only survived via the same || !props.isOpen disjunct pattern described above. Two things sink a naive "component test driving the transition" as a way to catch this class at the parent: (1) the contradictory pair is synchronous-only — no await separates the two writes, so nothing observable renders in between for a component-level assertion to catch; and (2) the unit suite stubs IonModal as a plain <div> (same fact as above), so real canDismiss enforcement is structurally untestable at that layer regardless of timing. The pattern that does work, since the transition itself is unobservable: a { flush: 'sync' } watcher registered on the ref pair before driving the action, asserting the pair is never observed as (open=false, loading=true) — see the invariant tests in useCollectiveGoalOptIn.spec.ts, added alongside the composable's ordering fix. Fix (shipped): both success paths now clear their loading flag before their open flag; pause() owns no open ref and needed no change. (Root cause/fix: OBJ-2738.)

Shared in-flight flag causes cross-blocking between unrelated actions

• A composable or component that reuses one in-flight ref (loading, saving) across multiple independent actions makes a naive re-entry-guard fix cross-block them — guarding one action on the shared flag makes an unrelated action refuse to start while the first is in flight. composables/useCommunity.ts used one loading ref across six functions (joinCommunity, leaveCommunity, followCommunity, unfollowCommunity, createCommunity, createCommunityPost) — a literal if (loading.value) return guard on any one of them would have made creating a post refuse to start while an unrelated follow was still in flight. views/MeetCoach.vue had the same shape at smaller scale: savePreferences() and resetToDefaults() shared one saving ref, so resetting to defaults would have refused to start while a preference save was still resolving, and vice versa. Found and fixed during v4.39's mutation re-entry guard census (OBJ-2671/2673) while adding the synchronous early-return guard this bug class needs (see Mutation Re-Entry Guard § Why a reactive :disabled binding is not sufficient for why the guard itself has to be a synchronous check, not a template binding) — the census script itself doesn't detect this trap; it was caught by inspection while triaging the flagged sites. Fix (shipped): useCommunity.ts split loading into six independent per-action refs (joiningCommunity/leavingCommunity/followingCommunity/unfollowingCommunity/creatingCommunity/creatingCommunityPost, useCommunity.ts:10-15), keeping loading as a computed OR of all six for any consumer that only needs "is anything in flight" (useCommunity.ts:16-24). MeetCoach.vue split saving into savingPreferences/resettingToDefaults (MeetCoach.vue:561-562), each independently guarded (:725, :767) — a Vitest test asserts resetToDefaults still starts while savePreferences is in flight. General rule: before adding a re-entry guard to a function that shares an in-flight ref with other functions in the same file, check whether those other functions are actually the same action (fine to share) or independent actions that happen to reuse a flag out of convenience (give each its own ref, first). Full pattern and worked examples: Mutation Re-Entry Guard § The shared in-flight flag — cross-blocking trap. (Root cause/fix: OBJ-2673, v4.39 Phase 3.)

Apollo cache replaces (not merges) a non-normalized nested User field written by operations with different field subsets

• A GraphQL nested object with no id/publicId of its own doesn't normalize in Apollo's InMemoryCache — it's stored inline on its normalized parent, so two operations selecting different field subsets on it make Apollo replace the cached object wholesale instead of merging, silently dropping whichever fields the narrower write didn't ask for. User.achievementStats, User.coachingPreferences, and User.goalMotivationProfile all hit this: UserQuery selects the wide shape (10/8/5 fields respectively), while StreakRepairOfferQuery/RepairStreakMutation, WelcomeBackOfferQuery, and SyncUserMutation each select a 1-field subset of the same field on the same normalized User:{publicId} entity — SyncUserMutation fires on every sign-in, so goalMotivationProfile was being clobbered on a hot path. Apollo logs Cache data may be lost when replacing the <field> field of a <Type> object… when this happens; the fallback silently wins over erroring, so nothing in the pipeline surfaces it beyond that console warning. Fix (shipped): ionic_frontend/src/apollo-type-policies.ts adds { merge: true } (Apollo's built-in shallow merge) to all three fields, extracted out of apollo-client.ts into its own side-effect-free module so a Vitest unit test can import it without booting the live ActionCable consumer / Sentry / auth store apollo-client.ts creates at module scope. REPAIR_STREAK_MUTATION was also widened to select the derived rank fields (currentRankName, currentLevel, xpToNextRank, last7DaysXp) alongside currentXp — merge: true alone stops the blanking but would otherwise leave those fields stale after a repair, since the narrow mutation write never refreshes them. Two escape hatches already in the codebase: a nested field with its own id (latestEnneagramAssessment) normalizes and merges by default, no policy needed; a field where every writer shares one fragment (onboardingStatus via UserFragment) never has a subset mismatch to collide on. PublicProfile.achievementStats was checked and left without a policy — PublicProfile isn't normalized today and has one writer, but it inherits this exact exposure the moment anyone adds PublicProfile: { keyFields: [...] }. Full write-up, the audited-and-cleared field list, and the regression-test negative control: Apollo Cache Merge Policies — Non-Normalized Nested Fields. (Root cause/fix: OBJ-2707, PR #2476.)

A GraphQL mutation's { errors: [String] } payload is not a promise rejection — a success-path-only handler drops it silently

• A GraphQL mutation that resolves with { data: { goal, user, errors } } on a business-rule rejection is not distinguishable from a genuine no-op unless the errors branch is handled explicitly — Apollo's useMutation does not reject/throw for this shape, so a try { ... } catch { ... } around the call alone cannot see it. useStreakRepair.ts#repair() and useWelcomeBack.ts#accept() (REPAIR_STREAK_MUTATION/ACCEPT_STREAK_MERCY_MUTATION, both following this repo's standard { goal, user, errors } mutation shape — see Backend: GraphQL Structure) each had if (!data?.errors?.length && data?.goal) { ...success... } with no accompanying else — a real backend rejection (XP balance too low, repair window lapsed, already repaired, enforced by Gamification::RepairStreak's sufficient_xp?/within_repair_window?/not_already_repaired? guards) left success/dismissed both false and repairing reset in finally, so the confirmation modal silently reverted to idle — a legitimate "no" from the backend was indistinguishable from a tap that did nothing. Duplicated verbatim across both composables, same copy-paste lineage. This co-occurred with an unrelated visual bug (RepairConfirmationModal.vue missing a --backdrop-opacity override that its sibling modals both set, plus Dashboard.vue never hiding StreakRepairCard while the modal was open) — same screen, two independent root causes, worth not conflating when triaging a similar report. Fix (shipped, OBJ-2725): an else if (data?.errors?.length) branch in both repair() (useStreakRepair.ts:98-105) and accept() (useWelcomeBack.ts:66-73) that reports Sentry.captureMessage(...) (extra: { errors: data.errors }) and surfaces toast.error(data.errors[0] ?? "Couldn't restore your streak") instead of falling through to nothing. RepairConfirmationModal.vue's CTA also gained an insufficientXp disable guard (mirroring StreakRepairCard.vue's existing one) to close the most likely trigger path before the user ever taps.

Fix (shipped, OBJ-3576, PR #3043, commit d4c49e6db): the two failure classes this entry originally left open — a thrown/network error (previously an empty catch {}) and a resolved-but-unusable falsy data response (previously fell through both branches with no else at all) — are now both handled explicitly in both composables, closing every gap this entry has ever tracked. Both get parallel treatment to each other, and to the data.errors path above, but stay fail-open rather than fail-closed: Sentry.captureException(e, { extra: { goalId, composable } }) on the throw path (useStreakRepair.ts:115, useWelcomeBack.ts:83) or Sentry.captureMessage(...) on the falsy-data path (useStreakRepair.ts:109, useWelcomeBack.ts:77), plus a transientError ref set to a user-visible message — the offer/interstitial stays live (no dismissed, no refetch()), unlike the terminal data.errors branch. transientError renders as an inline role="alert" line on both aria-modal="true" surfaces — RepairConfirmationModal.vue:101 ("That didn't go through. Your XP wasn't spent — try again.") and WelcomeBackInterstitial.vue:90 ("That didn't go through. Nothing's lost — try again.") — and as toast.error(...) (no inline alert) on the non-modal WelcomeBackBanner surface, fired from Dashboard.vue#handleWelcomeBackAccept (Dashboard.vue:669-670) when isReducedCoach is true. StreakRepairCard.vue is untouched by design — it's guaranteed unmounted before a repair failure can resolve. transientError clears on retry-dispatch, success, the terminal data.errors path, and dismiss — never on a timer.

General rule: an Apollo mutation call site has three distinct failure classes, not two — (1) a thrown/network error, catchable only via try/catch; (2) a business-rule rejection returned as data.errors on a resolved promise (this repo's standard { goal, user, errors } shape, per CLAUDE.md's GraphQL mutation example — a try/catch alone never sees this one); and (3) a resolved-but-unusable falsy/malformed data (no thrown error, no errors, no expected payload key — the shape a try/catch and a bare data?.errors?.length check both miss). Each needs its own explicit branch — an if (success) {...} else if (data?.errors?.length) {...} with no final else, or a try/catch with an empty catch, both silently drop one or two of the three. Same family as A retry-exhausted 5xx and a terminal 4xx throw the same error shape above — different transport (GraphQL payload vs. HTTP status) and different layer (Clerk's REST API vs. this repo's own mutations), but the same root shape: a single failure-detection mechanism only ever covers one class, and every other class needs its own explicit check. (Root cause: Dave's diagnosis, OBJ-2725, PR #2473. Residual throw/falsy-data gap closed: OBJ-3576, PR #3043.)

An unguarded async @click handler escalates a scoped failure into a whole-app crash

• A template @click handler that awaits a mutation with no try/catch lets any rejection — including a transient one like a rate-limit response — propagate through Vue's async error handling straight into GlobalErrorBoundary, crashing the entire app for what should be a recoverable failure of one UI action. NotificationsPanel.vue's handleMarkAllRead() called acknowledgeAllGlobal() (via useNotifications.ts#acknowledgeAll(), which itself fired one ACKNOWLEDGE_NOTIFICATION_MUTATION per unread notification in parallel via Promise.all) with no try/catch. A user with a non-trivial unread backlog (29 unread in the reporter's screenshot) fired 29 simultaneous mutation POSTs from one tap; if any single one came back HTTP 429, Promise.all rejected on that first failure, the rejection had no catch between acknowledgeAll() and the click handler, and Vue's async error propagation routed it to GlobalErrorBoundary.vue's onErrorCaptured (ionic_frontend/src/components/ui/GlobalErrorBoundary.vue:35-40) — rendering the full-page "Something went wrong" crash screen for the whole app instead of a scoped, in-panel error. Fix (shipped): two independent layers, both required. (1) Replaced the N-parallel-mutations pattern with a single batch AcknowledgeAllNotifications GraphQL mutation (rails_api/app/graphql/mutations/acknowledge_all_notifications.rb, backed by Gamification::AcknowledgeAllNotifications's one update_all scoped to current_user) — one request instead of N removes the request-storm shape that makes a rate limit likely in the first place. (2) handleMarkAllRead() (NotificationsPanel.vue:239-266) now wraps both its global and team-scoped paths in try/catch, surfacing a toast.error('Could not mark all as read. Try again in a moment.') on rejection instead of letting it propagate — comment at :262-263 names the invariant directly: "A transient failure (e.g. a rate limit) here must stay scoped to this action — never let it propagate to GlobalErrorBoundary." General rule: this shape is generic beyond notifications — any template @click="asyncHandler" that awaits a mutation with no surrounding try/catch is one transient network blip away from escalating a one-action failure into GlobalErrorBoundary's full-page crash. Guard the handler, not just the mutation call site; a toast/inline error is the right failure mode for a UI action, a global crash never is. (Root cause: Dave's diagnosis, OBJ-2948, PR #2647.)

Cold-entry teamId resolution needs an unscoped bridge query + enabled guard

• Any team-scoped view on a route with no :teamId param can be a cold-entry point, and useTeamScopedView()'s activeTeamId fallback is not guaranteed populated on one. useTeamScopedView() (ionic_frontend/src/composables/useTeamScopedView.ts) resolves teamId from the route param first, then falls back to a module-level activeTeamId ref seeded from localStorage (key objectuve_active_team_id, OBJ-1821) — but that key is only written by setActiveTeamId(), which only runs after some earlier view resolved a team. /team-home and /team-leaderboard both carry no :teamId route param (single-team beta), so a cold direct navigation (deep link, bookmark, or an evidence-capture bot with no prior same-session visit) hits them with activeTeamId unset — a team-scoped query fired with teamId: undefined gets rejected by GraphQL's $teamId: String! and the view either collapses to a full-page error or hangs, never populating (Sentry OBJECTUVE-CLIENT-7V, OBJ-2348). The fix pattern is two parts, both required: (1) an unscoped bridge query that can resolve a team with no teamId argument, whose result feeds setActiveTeamId() in a watchEffect; (2) an enabled: !!teamId.value guard on the team-scoped query so it never fires with a null/undefined teamId while the bridge is still resolving. Three shipped instances, same shape but a different bridge each time: TeamSettingsView.vue:226-238 bridges off useTeam()'s MY_TEAM_QUERY { myTeam } (resolves by current_user alone); TeamLeaderboardView.vue:88-98 (bridge) and :119-123 (enabled guard) does the same for TEAM_LEADERBOARD_QUERY; TeamHomeView.vue:447-465 (enabled guard) and :497-502 (fallback bridge watchEffect) bridges off MY_TEAMS_QUERY instead of adding a second useTeam() call, since MY_TEAMS_QUERY is already unconditionally in flight in that view for other reasons — a teamIdResolving computed (TeamHomeView.vue:486) additionally guards against a teamUnavailable flash in the window after myTeams loads but before the bridge's setActiveTeamId() call lands. Residual, unverified assumption: the components that consume useTeamScopedView() without their own bridge — 14 files under ionic_frontend/src/components/teams/ (e.g. TeamMembersTab.vue, TeamSwitcher.vue, InviteMemberModal.vue) plus CreateTeamView.vue/JoinTeamView.vue — assume they always render inside a parent that already resolved teamId first. That's true today because every route that mounts them has a resolving parent, but it's an assumption about the render tree, not something useTeamScopedView() itself guarantees — a future route that mounts one of these components as a cold entry point reopens the same bug class. Before adding a new cold-entry-reachable team route, check whether it needs its own bridge, don't assume the composable already covers it.

A single-element dialog steps array collapses isIntro/isOutro — a story can silently render the wrong state while claiming to show another

• A Storybook fixture whose steps array has only one element makes index 0 both the first and the last step, so any component branching on "is this the first step" vs. "is this the last step" collapses those two checks onto the same render — and v-if/v-else-if precedence silently picks one, with no error. FeatureTour.vue:269-270 computes isIntro = currentStepIndex === 0 && layout === 'dialog' and isOutro = isLastStep && layout === 'dialog' — both true at index 0 on a one-element array. The template (FeatureTour.vue:173,177) tests v-if="isIntro" before v-else-if="isOutro", so isIntro always wins the race: the story renders the intro sparkles icon under the outro copy instead of the gold check badge + ring, with no console error, no failing test, no visual diff to flag it. FeatureTour.stories.ts's Outro and CompletionMoment stories both shipped with args: { steps: [OUTRO_STEP] } and both hit this — a story literally named after the state it claims to render was silently showing a different one. Discovered by Codi and confirmed by Roy while independently verifying an unrelated --gradient-gold token repair (OBJ-2871) required a workaround (driving the real 3-step Intro story through onNext() to reach genuine outro) to get a trustworthy screenshot. Fix (shipped): both stories now seed a real preceding step (args: { steps: [INTRO_STEP, OUTRO_STEP] }, index 0 is no longer also the last index) and drive the actual onNext() transition — Outro via a play() function that clicks the intro button, CompletionMoment via the same click-through on mount and on every Replay (its active prop toggling off/on resets currentStepIndex back to 0 via the component's own watch, so it has to re-drive the transition each time, not just once). Each story's play() now asserts the outro title actually renders, so a regression fails visibly in Storybook's interactions panel — but at the time, only when a human opened the built/dev Storybook. @storybook/addon-vitest's storybookTest() plugin wasn't wired into a Vitest run, so these play() assertions weren't a CI gate (tracked as a follow-up, OBJ-2969, backlog). Shipped (OBJ-3390, v4.46 Phase 3): every story's play() now runs as a real Vitest test via CI's storybook-test job, closing that gap for FeatureTour's outro assertion and every other play() in the repo — see A Storybook story can lie about what it renders — play() is now CI-enforced below. General rule: a single-element array/step-list fixture is a foot-gun for any component that branches on both "first" and "last" position — check whether the states you're claiming to cover actually land on different indices, or drive the component through its real transition instead of just setting args. (Root cause: OBJ-2919, PR #2667.)

window.Clerk.signOut() performs its own real document navigation, bypassing Vue Router entirely

• Calling window.Clerk.signOut() with no callback argument lets Clerk perform its own hard document navigation, not an SPA transition — one that bypasses Vue Router's guard chain entirely, including a route's own beforeEnter guard. useAuthStore().signOut() called await window.Clerk?.signOut() unconditionally. On the account-deletion teardown path, Session.logout() had already replaced the route to /account-deleted and set navigate: false specifically so nothing would move the user off that confirmation screen (UI-SPEC OBJ-3059 Ruling 3/4's "no auto-dismiss" contract) — but Clerk's own signOut() still fired its own redirect underneath, since main.ts's clerkPlugin options (main.ts:71-78) never wire the routerPush/routerReplace callbacks that would let Clerk hand that navigation to Vue Router instead of performing it directly. The confirmation screen auto-navigated away immediately after rendering, with no exception and nothing in the app's own routing layer able to see why. Fix (shipped): the navigate flag already threaded from Session.logout() (used to gate the app's own trailing router.push('/sign-in'), session.js:141-142) is now also threaded into useAuthStore().signOut({ navigate }) (useAuthStore.ts:172-203) — when false, it passes Clerk a no-op callback as signOut()'s first argument, which makes Clerk await that callback instead of navigating on its own; when true (ordinary sign-out), behavior is unchanged. General rule: a navigate: false / "stay on this screen" contract threaded through teardown code isn't complete until every SDK call in that path that can perform its own redirect is checked, not just the app's own explicit router.push/router.replace calls — Clerk's signOut() is one instance; any other Clerk (or third-party SDK) call that accepts a callback/redirect option needs the same audit before it can be trusted to respect a "don't navigate" contract. Known latent gap (not fixed here, flagged but not ticketed): ordinary sign-out (navigate: true) still calls window.Clerk?.signOut() with no callback, so it has the same underlying redirect-bypass shape — currently invisible only because Clerk's own default post-sign-out redirect and the app's explicit SignIn push happen to converge on the same outcome. (Root cause/fix: OBJ-3311, PR #2878.)

A prose surface inventory is not a mechanical guarantee

• A UI-SPEC's prose inventory of "every surface that renders X" is not a mechanical guarantee — it can miss real sites with no lint error, no test failure, and no visual regression signal. v4.48 Phase 2's P2-A1 pass (docs/ui-specs/v4.48-streak-motif-accretion-grammar.md §16) converted every "preference-resolving" surface to render the user's chosen streak motif through StreakMetaphorIcon — the app's single motif-resolution point — and explicitly annotated the sites it triaged out (<!-- P2-A1: out of scope — <reason> -->). That inventory counted ten M1 surfaces and thirteen total, verified at commit d6049e3f3. Three live Dashboard.vue surfaces representing the signed-in user's own streak — RepairConfirmationModal.vue:55, WelcomeBackInterstitial.vue:54, WelcomeBackBanner.vue:10 — carried neither the conversion nor the out-of-scope annotation. They were simply missed, and each one rendered a hardcoded generic icon (or, on the banner, a different real motif, flame) to every user regardless of their actual chosen stones/sprout/mountain/waves/flame motif — reported by a user (OBJ-3567) who saw stacked rings on the repair modal instead of the sprout he'd actually chosen. Fix (shipped, OBJ-3567): all three surfaces converted to StreakMetaphorIcon; docs/ui-specs/v4.48-streak-motif-accretion-grammar.md §16/§16.1 corrected to sixteen/thirteen surfaces with the three added to the M1 table; CommunityBadges.vue:44 (a genuinely out-of-scope aggregate community stat) annotated rather than converted. The mechanical fix for the class of bug, not just its instances: ionic_frontend/tests/unit/streakMotifSurfaceGuard.spec.ts now scans ionic_frontend/src/**/*.vue and fails on any AppIcon with name="streak" or name="flame" that lacks an adjacent P2-A1: out of scope annotation — demonstrated failing on a reverted site before being trusted. General rule: when a UI-SPEC or migration pass claims to have converted "every" surface of a class, the annotation-vs-conversion distinction only catches sites the pass actually visited — it says nothing about a site the pass never looked at. This doc's own surface count didn't even agree with itself before the fix (§16.1 said "Ten," §16's own intro said "thirteen" total) — a second, independent signal that a prose count drifts from reality faster than anyone re-reads it. A recurring class of bug needs a repo-scan guard, not another manual re-read of the prose table.

A v-if-gated sibling mounted in front of a presented inline <ion-modal> throws insertBefore NotFoundError

• A component that v-if-mounts as a new sibling immediately before an inline <ion-modal> that is currently presented crashes the whole app into GlobalErrorBoundary with NotFoundError: Failed to execute 'insertBefore' on 'Node', deterministically, and with nothing in the stack naming Ionic. When an inline <ion-modal :is-open="true"> presents, @ionic/core's CoreDelegate.attachViewToDom leaves an <!--ionic teleport--> comment where the element was and physically appendChilds the <ion-modal> into <ion-app>; it is only moved back on dismiss. Vue's vnode still points at the moved element. IonPage renders its slot with h('div', ..., slots.default()), so Vue diffs the page's children with patchKeyedChildren, whose anchor for a newly mounted middle node is the next sibling vnode's .el — the teleported <ion-modal>, which is no longer a child of the ion-page div. parent.insertBefore(newEl, anchor) throws.
• This is how /play (Critical Path) crashed for every first-time player: PlayView.onMounted flipped showTutorial = true (presenting HowToPlayOverlay's inline sheet) before awaiting criticalPathToday; when the query resolved and puzzle was set, <CompletionModal v-if="puzzle"> — listed just before <HowToPlayOverlay> in the template — mounted with the teleported sheet as its anchor. The tutorial-seen key is only written on dismiss, which never happened, so "Try Again" reproduced it every time (Sentry OBJECTUVE-CLIENT-9M, 3 users, all first-time players on Android where the flagged route was live).
• Fix / general rule: components whose root is an inline <ion-modal> (CompletionModal, HowToPlayOverlay, ThemePickerSheet, and any other overlay component) must be mounted unconditionally and gated on :is-open, never v-if'd into existence next to a sibling overlay that might already be presented. If a v-if is genuinely needed, place the conditional element after every inline overlay in the parent template, or wrap it so its anchor is a stable non-overlay element. Vitest can catch the mount-order half (the regression in tests/unit/play/PlayView.spec.ts asserts CompletionModal exists while the tutorial is open and puzzle is still null), but not the DOM teleport itself — jsdom never runs Ionic's present(), so a unit test will stay green on the crashing template unless it asserts the mount order directly. Distinct from the Storybook-only teardown race below (TypeError on a null Reference.parentNode during dismiss); same teleport primitive, opposite direction. (Root cause/fix: ionic_frontend/src/play/PlayView.vue, Sentry OBJECTUVE-CLIENT-9M.)

ActionCable's ConnectionMonitor never reconnects immediately on socket close

• ActionCable's client-side ConnectionMonitor (actioncable@5.2.8-1, vendored — not in node_modules in every checkout, npm pack actioncable@5.2.8-1 to read it) reconnects only via its own polling/backoff schedule, and explicitly refuses to reopen for a minimum of staleThreshold (6 seconds) after a disconnect — independent of whether the browser's real connectivity already came back. Connection.events.close only records the disconnect and notifies subscriptions; it never calls reopen(). The only two call sites for connection.reopen() are ConnectionMonitor.reconnectIfStale() (a recursive setTimeout poll, interval clamp(5·ln(attempts+1), 3, 30) seconds, no jitter in this version) and visibilityDidChange (tab-foreground). Nothing in this app bridged the browser's own online event — already observed by useNetworkStatus.ts for UI-gating — to a forced reconnect, so a genuine socket teardown could take well past what a reconnect-dependent UI would reasonably wait for "back online."
• Correction: this entry originally claimed coach-reconnect-grace.spec.ts's deterministic failure (100%, chromium + mobile-android) traced to this poll/backoff floor — showResumedLine supposedly starved waiting on ConnectionMonitor's own schedule. That causal claim is superseded: Dave's round-2 diagnosis (OBJ-3953) confirmed page.context().setOffline(true) never tore down the spec's already-established /cable WebSocket at all (see context.setOffline(true) doesn't close an already-established WebSocket below), so ActionCable's disconnected event never fired and ConnectionMonitor's poll/backoff was never the operative cause of this spec's failure — it just wasn't exercised. Why showResumedLine didn't render is, as of this writing, still an open question tracked on OBJ-3953; this entry no longer asserts a mechanism for it.
• Fix (shipped, OBJ-3953): apollo-client.ts's new reconnectCable() helper calls cable.ensureActiveConnection() — a no-op if the connection is already open/connecting (checked via isActive()), otherwise connection.open() directly, bypassing ConnectionMonitor's poll/backoff and its 6s floor entirely. This is safer than calling connection.reopen() directly, since reopen() force-closes an already-active connection first and can race a ConnectionMonitor poll that already restored it around the same time. useNetworkStatus.ts calls reconnectCable() from both the web online listener and the native Network.addListener callback on connected: true. A Sentry breadcrumb now fires on every disconnected callback, and a warning (with elapsedMs) if the matching connected callback takes longer than RECONNECT_WARN_THRESHOLD_MS (8s, chosen past the 6s staleThreshold floor) to arrive — the transport link had no Sentry wiring at all before this (apollo-client.ts's errorLink only covers the non-subscription HTTP branch).
• General rule: any reconnect-dependent UI built against onCableConnectionChange (CoachChatPanel.vue, usePartnerStatus.ts, useLiveQuery.ts, AlliesPage.vue) must not assume onConnect fires promptly just because the network recovered — call reconnectCable() from wherever real connectivity is observed, or the reconnect can legitimately take 6s+ with no upper bound. (Root cause: Dave's diagnosis; fix: Codi, ionic_frontend/src/apollo-client.ts, ionic_frontend/src/composables/useNetworkStatus.ts, PR #3324.)

Feature Flags & PostHog

PostHog flags must be registered in both code and PostHog

• PostHog flags must be registered in BOTH places. Adding a key to FEATURE_FLAGS / MULTIVARIATE_FLAGS in ionic_frontend/src/lib/featureFlags.ts is half the job — without a matching flag in PostHog project 368400, posthog.isFeatureEnabled() returns undefined, the isFeatureFlagEnabled wrapper fails closed, and the gated feature stays hidden from everyone silently. Run POSTHOG_PERSONAL_API_KEY=phx_… node scripts/sync-posthog-feature-flags.mjs --apply in the same PR that adds the key. New flags are created disabled + 0% rollout, so this is safe to run on every PR. CI enforces the drift gate via the posthog-flag-drift job in .github/workflows/ci.yml. Full lifecycle (introduce → rollout → GA → retire) is in docs/development/feature-flags.md; use --promote <key> / --promote-all to flip a boolean flag to active + 100% rollout (kill-switch GA), then retire it from code after one release.

Never change PostHog flag state via the API directly

• Never change PostHog flag state by calling the API directly — always go through the sync script. If you hold a feature_flag:write key (Codi does), you can PATCH a flag to active: true / 100% rollout straight through the PostHog API, bypassing the POSTHOG_ALLOW_PROMOTE=1 gate entirely. Don't. Use --apply to register flags and --promote (with POSTHOG_ALLOW_PROMOTE=1) to GA one, and only promote when a human has actually asked for GA. The env gate is a footgun guard — it does its real job, stopping an accidental --promote-all in a scripted or agent run — but it is not an authorization boundary, so the rule has to be the control. Blast radius if you get it wrong is bounded: a wrongly-promoted flag turns a feature on for everyone and is revertible in one click in the PostHog UI, no deploy. See feature-flags.md#credential-gotchas.

Missing VITE_POSTHOG_KEY breaks flag-gated UI evidence (4 surfaces)

• preview.yml missing VITE_POSTHOG_KEY broke UI-evidence capture for every flag-gated route (parity gotcha, 4 surfaces) — Any Vite build step for ionic_frontend/admin_dashboard that sets VITE_API_URL (the signal for "this is a real production build") must also set VITE_POSTHOG_KEY: phc_CcuNbgW2uNpBrK2yMs8mcezGbC3aJ9CDwCinM76idWEd, or posthog.init() (ionic_frontend/src/posthog.ts) runs with an empty key. With no key, PostHog can never fetch a flag payload, so isFeatureFlagEnabled(...) fails closed for every flag-gated route — the router guards (e.g. ionic_frontend/src/router/index.ts:313-319, the /play route's beforeEnter) then silently redirect to /dashboard instead of the requested route, breaking capture-ui-evidence for any PR whose UI-Evidence-Routes points at a flag-gated page. preview.yml's "Build Ionic Frontend" step was missing the key while staging.yml/production.yml/mobile-android*.yml's equivalent steps already had it — writing the CI guard for this (below) surfaced the identical gap independently in three more places, all fixed in the same change: mobile-ios.yml's "Build Web Assets" step (every App Store build shipped with flags failing closed), rollback.yml's "Build and Deploy Frontend" step (every production rollback would too), and staging.yml/production.yml's "Build Admin Dashboard" step (lower severity — admin_dashboard/src/posthog.ts no-ops gracefully on a missing key rather than redirecting, but admin PostHog analytics never initialized in either environment). Guardrail (shipped): ci.yml's lint-workflows job has a "Guard against Vite build steps missing VITE_POSTHOG_KEY" step that scans every .github/workflows/*.yml step for VITE_API_URL: without a paired VITE_POSTHOG_KEY: and fails CI — verified to catch all four gaps on the pre-fix tree and pass clean on the fixed tree. If you add a new Vite build step for either frontend, set VITE_POSTHOG_KEY alongside VITE_API_URL or the guard will fail you. (Root cause: OBJ-1682, PR #1736.)

Flag-gated router guards failed closed on cold boot

• PostHog flag-gated router guards failed closed on cold boot — a third, distinct PostHog-flag failure mode (OBJ-2023) — main.ts resolves the SPA's initial route (await router.isReady(), which runs every beforeEnter guard) before app.mount(), but initPostHog() only ran from a post-mount scheduleIdle idle callback (deliberate "defer non-critical init to after first paint" perf choice, ionic_frontend/src/main.ts:87-102). Any flag-gated guard's first isFeatureFlagEnabled(...) check therefore ran against an uninitialized PostHog client and deterministically failed closed — not a race that sometimes loses, a code path that could not win, since the init that would satisfy the check hadn't even been scheduled yet. Real users mostly dodged it (they land on an unguarded route first, and by the time they navigate client-side the idle callback has usually fired); any hard page.goto(), deep link, bookmark, or shared link straight to a gated route hit it every time. Distinct from the two neighboring gotchas above: PostHog is configured correctly and the flag really is on — this is a boot-order race between main.ts and every gated beforeEnter, not a missing registration or a missing env var. Shipped fix (Direction C — the boot path itself is untouched): the 7 gated beforeEnter call sites (ionic_frontend/src/router/index.ts — evaluateAllyRequestFlowGate at :124-135, shared by 3 routes, plus 4 inline closures at :172-174, :202-204, :285-287, :368-370) now await awaitFeatureFlags() (ionic_frontend/src/lib/featureFlags.ts:154-186) before evaluating their flag. awaitFeatureFlags() calls initPostHog() on demand — initPostHog() is idempotent (module-level initialized guard, ionic_frontend/src/posthog.ts:5,9), so whichever caller reaches it first, this helper or the still-unmoved post-mount idle callback, does the real init and every later call is a no-op — then resolves as soon as posthog.onFeatureFlags() fires or a 1500ms timeout elapses, never rejecting. Falsified assumption — do not re-attempt "just move initPostHog() earlier": an earlier round (Option A) moved posthog.init() onto the pre-mount boot path directly, ahead of router.isReady(). That measured a real ~52ms synchronous cost added to every cold boot, not just the 7 gated routes, and disable_session_recording: true did not reduce it (median stayed ~53ms) — the cost is the SDK's own config/persistence/autocapture wiring, not session recording. Rejected for that reason; the shipped boot path is byte-identical to pre-OBJ-2023 master — pinned by ionic_frontend/tests/unit/mainBootOrder.test.ts, a source-text Vitest asserting initPostHog( stays after app.mount( (never imports main.ts, since it has real top-level side effects; reads it as text instead, mirroring scripts/check-branch-promotion.test.mjs's convention). Guardrail (shipped): ionic_frontend/tests/smoke-playwright/specs/platform/cold-boot-flag-gated-route.spec.ts does a cold page.goto('/team-leaderboard') with teams_leaderboards forced on via registerFlagOverrides and asserts the gated view renders, not the /dashboard redirect — template for the other 4 gated guards. It needs clearPostHogPersistence() (ionic_frontend/tests/smoke-playwright/helpers/flagOverrides.ts) first, on both smoke lanes but for opposite reasons: the staging lane's storageState is captured from a real signed-in browser (auth.staging.setup.ts:223) and already carries a loaded flag payload that would let the spec pass on staging for the wrong reason, while production's hand-built storageState (auth.production.setup.ts) carries none — same spec, different flag semantics, unless both are normalized to the same cold state first. Two previously-vacuous specs (ally-discovery.spec.ts, pending-ally-requests.spec.ts) accepted either the gated view or the redirect and so never actually caught this; both are now single-outcome. If you add a new flag-gated beforeEnter guard, it must await awaitFeatureFlags() before evaluating the flag — see docs/development/feature-flags.md. (Root cause: OBJ-2023, surfaced diagnosing OBJ-1969's UI-evidence-capture failure.)

PostHog sync script needs exact env var POSTHOG_PERSONAL_API_KEY

• The PostHog sync script's env var is POSTHOG_PERSONAL_API_KEY, exactly — scripts/sync-posthog-feature-flags.mjs reads only that name; a key injected under any other name is invisible to it, and the script exits 2 reporting the key is missing even when a valid one is present. If an agent reports no PostHog access, check the env var name before assuming the key is absent. This bit Codi, whose key was injected as POSTHOG_ACCESS_KEY in its Multica per-agent env config until it was renamed on 2026-07-20 — suspected, never proven, to be behind the OBJ-1519/OBJ-1547 "no agent can run the sync" stalls. Codi can now run the sync itself; its key's scopes are recorded in feature-flags.md#credential-gotchas.

Testing & Smoke Suites

Process-global class_attribute toggles leak across spec files

• Rails class_attribute settings (e.g. inheritance_column) are process-global, not example-scoped — toggling one in before(:all)/after(:all) leaks into every other spec file sharing the same RSpec process — PostReaction permanently disables STI at load time (self.inheritance_column = nil in post_reaction.rb, since type stores the reaction kind — 'like', 'love' — not a subclass name). post_reaction_spec.rb used to re-toggle this per-file with before(:all) { PostReaction.inheritance_column = :_type_disabled } / after(:all) { PostReaction.inheritance_column = :type }, but after(:all) restored :type (Rails' STI default) instead of nil (the model's real value). Because inheritance_column is a process-global class_attribute, not reset per-example, this corrupted PostReaction for any other spec file that happened to share the same RSpec shard afterward — e.g. community_query_spec.rb, whose factory writes raw strings into PostReaction#type and raised ActiveRecord::SubclassNotFound once STI was silently re-enabled. bin/partition_specs's round-robin sharding means which files share a process shifts whenever the total spec-file count changes anywhere in the repo, so this class of bug is shard-alignment-dependent and can resurface unpredictably even with no changes to the affected files themselves. Fix (shipped): deleted the redundant spec-level toggle — the model already sets inheritance_column = nil permanently, so no per-file override was needed. Root cause: rails_api/spec/spec_helper.rb still has config.order = :random commented out (RSpec's default), so specs run in file-alphabetical order; randomizing example order is the general regression guard against this whole bug class (any class-level AR setting mutated in before(:all)/after(:all)) but was not enabled as part of this fix — tracked as a follow-up, not yet scheduled. (Root cause: OBJ-1214, surfaced as a required-check failure on PR #1338.)

Specs must use public_id, never user.id

• Mutation/interaction specs: use public_id, never user.id — Specs that pass user.id (integer, or integer converted to string) for a userId input will pass CI because find_by(id: …) resolves against the PK in test, but fail silently in production where the frontend always sends user.public_id (base64 token). Wrong: userId: user.id.to_s or userId: '99999'. Right: userId: user.public_id. Any input named *_id crossing the GraphQL boundary should use public_id in specs. The interaction should use Model.public_find(id) for lookups, not find_by(id: id).

A custom RSpec type: that rspec-rails never auto-infers from file location gets none of use_transactional_fixtures's automatic wrapping

• A hand-assigned type: on a spec whose directory rspec-rails' infer_spec_type_from_file_location! doesn't recognize opts that file out of automatic transactional-fixture wrapping, even though config.use_transactional_fixtures = true is set globally (rails_api/spec/rails_helper.rb:50,75). rspec-rails only mixes in ActiveRecord::TestFixtures's per-example transaction for its own recognized directory patterns (spec/models, spec/requests, spec/jobs, etc.); spec/db/seeds_spec.rb lives outside all of them, so its RSpec.describe 'development seed', type: :seed, slow: true — a type invented for this one file, not one rspec-rails infers — gets no automatic wrapping at all. Every example there load-ed the full db/seeds/development.rb and committed real rows (including a demo@objectuve.com user) straight to the test database with nothing to roll them back once the example or before(:all) block finished. That leftover demo@objectuve.com row then collided with clear_demo_data_job_spec.rb's own fixture of the same email on index_users_on_email, a PG::UniqueViolation that looked like a clear_demo_data_job_spec.rb bug but was actually seeds_spec.rb leaking across files whenever both ran in the same RSpec process (slow: true specs are excluded by default — see the comment on config.filter_run_excluding slow: true in rails_helper.rb:52-57 — so this only surfaces under RUN_SLOW_SPECS=1). Fix (shipped, OBJ-3600): rails_api/spec/support/seed_spec_transactional_wrapper.rb adds a config.around(:each, type: :seed) that opens an unjoinable transaction (connection.begin_transaction(joinable: false)) before each example and rolls it back after — by hand, the same mechanism rspec-rails' auto-wrapping would have provided for a recognized type. A context needing seed data to persist across multiple examples (e.g. seeds_spec.rb's "DEMO-02 diversity contract") layers its own before(:all)/after(:all) transaction on top; nested transactions become real savepoints since the outer one is unjoinable, so the two compose safely. Use this file as the template for the next custom type: that needs the same treatment — don't re-invent the wrapper per spec type. A regression guard (rails_api/spec/config/seeds_spec_before_all_pairing_spec.rb) now fails if seeds_spec.rb ever gains an unpaired before(:all)/after(:all) again, though it counts hook occurrences as plain text (including inside comments), so it can be fooled by an unrelated comment edit — tighten to a real-hook regex before trusting it against a rewritten comment.

Playwright toBeVisible() on skeleton content needs a timeout

• Playwright toBeVisible() on loading-skeleton v-else content requires an explicit timeout — Any toBeVisible() / toBeEnabled() assertion on content rendered inside a v-else of a loading skeleton — or any element that only appears after an API response — must include { timeout: 20_000 }. The default 5 s Playwright timeout fires before the API resolves, causing intermittent or consistent failures on slower environments (PRs #895 and #910 both failed this way). Pattern: await expect(el).toBeVisible({ timeout: 20_000 }). Extended to toHaveURL() (OBJ-2529): a bare expect(page).toHaveURL(pattern) (1 arg, no options) hits the exact same 5s-default trap — 10-existing-user-not-re-onboarded.spec.ts misread as a /welcome-routing regression on mobile-safari when the real cause was the router auth guard needing longer than 5s to settle under CI load; the URL was still sitting at /, never /welcome, when the assertion fired. Use the named budget the failure actually needs (WARM_PAGE_HYDRATION for "page navigation + Apollo hydration", helpers/timeouts.ts), not a guess. eslint.config.js's no-restricted-syntax rule for tests/smoke-playwright/**/*.ts now flags any toHaveURL(...) call with fewer than 2 arguments the same way it already flagged bare toBeVisible()/toBeEnabled()/toBeHidden(). Extended to toHaveCount(N>0) (OBJ-3640): a bare toHaveCount(3) on supporter-overhaul.spec.ts's plan-tier radiogroup hit the same trap one matcher over — the radiogroup renders unconditionally on a v-if free-state gate, but its PlanTierCard options don't exist until PLANS_QUERY (GraphQL) resolves, so the 5s default raced the query and lost on chromium/mobile-safari/mobile-android. The rule is scoped to a non-zero expected count: toHaveCount(0) absence checks aren't waiting on data to appear, so they're exempt.

Playwright isVisible()/isHidden()'s timeout option is deprecated and ignored

• locator.isVisible({ timeout }) (and isHidden/isEnabled/isDisabled) does not wait — the timeout option is deprecated and silently ignored; the call returns the current DOM state immediately (confirmed against the installed playwright-core type declarations: @deprecated This option is ignored. ... does not wait for the element to become visible and returns immediately). This reads like a real, retrying wait (mirroring toBeVisible({ timeout }), which genuinely does poll) but isn't one — connected-apps-connect.spec.ts used .isVisible({ timeout: 5_000 }).catch(() => false) to pick between the empty-state CTA and the "already connected" fallback locator, and on mobile-safari the empty-state CTA had not finished hydrating at the exact instant the (non-waiting) check ran, every time — the check reported false immediately instead of catching the button a moment later, and the test fell through to the fallback locator (OBJ-2687). Use .waitFor({ state: 'visible', timeout }).then(() => true).catch(() => false) instead, which performs a real, retrying wait and only resolves false once the timeout genuinely elapses. eslint.config.js's no-restricted-syntax rule for tests/smoke-playwright/**/*.ts now flags any isVisible/isHidden/isEnabled/isDisabled call whose argument object includes a timeout key.

Playwright WebKit-on-Linux can't combine context.setOffline(true) with a following page.reload()/page.goto()

• page.context().setOffline(true) immediately followed by page.reload() (or page.goto()) crashes the mobile-safari (WebKit) project deterministically, with Error: page.reload: WebKit encountered an internal error — confirmed on two independent real staging runs of offline-cold-start.spec.ts on the merged OBJ-3879 tip (35063798786, 35068806584), all 3 cases, same error at every setOffline(true) → reload() call site. chromium and mobile-android exercise the identical UI-SPEC §6 code path and pass reliably on both runs — this is a WebKit-engine limitation, not a Safari-specific product gap, and this spec has never had a real passing mobile-safari staging run on any commit. Confirmed upstream, not fixable from this repo: microsoft/playwright#34402 tracks the identical setOffline + navigate crash signature and is closed as not planned — WebKit doesn't support toggling offline mid-session the way Chromium does. An independent project (fcw1987/pickleboard#14) hit the same error on Playwright 1.61 WebKit/Linux CI with no available workaround. Fix (shipped, OBJ-3905, PR #3283): test.skip(testInfo.project.name === 'mobile-safari', WEBKIT_OFFLINE_RELOAD_SKIP_REASON) added to all 3 cases in offline-cold-start.spec.ts, with the diagnosis and upstream issue links captured inline in the skip reason so a future reader doesn't mistake it for silently-dropped coverage. General rule: don't chase a fix for this signature — quarantine the affected case(s) on mobile-safari only, per-spec, the same pattern as the request-fixture parse-error retry above (a different WebKit/Linux quirk, same "WebKit-only, not a product bug" shape, 01-pwa-foundation.spec.ts's OBJ-1566 comment) — and confirm chromium/mobile-android still cover the same code path before skipping, so real coverage isn't lost.

context.setOffline(true) doesn't close an already-established WebSocket in Chromium

• context.setOffline(true) / page.context().setOffline(true) does not close or interrupt an already-established WebSocket connection in Chromium — it only blocks new outbound connections. Dave confirmed this directly (OBJ-3953 round 2) against both a local echo server and wss://echo.websocket.org, on this repo's pinned Playwright 1.60/Chromium: with a socket already open, toggling setOffline(true) left readyState at OPEN, fired no close event, no error event, and kept messages flowing bidirectionally for the entire "offline" window. coach-reconnect-grace.spec.ts's original fault injection relied on setOffline(true) to simulate a mid-session /cable drop and never actually produced one — a full Codi→Roy→Dori→Riley→Vicki review-and-ship cycle ran against a symptom this tooling gap produced before Vicki caught it reading per-test CI logs. setOffline is still correct for what it does gate — blocking new HTTP requests (e.g. offline-cold-start.spec.ts) and firing the browser's real online/offline events, which is what useNetworkStatus.ts listens for. It just cannot simulate an existing persistent connection dropping. Fix pattern: use context.routeWebSocket(url, handler) + WebSocketRoute.close({code, reason}) (Playwright 1.60+) to force-close the client-side leg of an already-open socket, producing a genuine close event. And this is not optional: route.close() resolving does not by itself prove the page-side socket actually reached CLOSED — assert the page's own WebSocket state (e.g. poll readyState via page.evaluate, or wrap window.WebSocket in an init script to record lifecycle transitions) before asserting on any app behavior that depends on the drop, so a future fault-injection regression fails with its own distinct message instead of masquerading as an app-level timeout. (Diagnosed: OBJ-3953, Dave. Fix pattern shipped in coach-reconnect-grace.spec.ts: OBJ-3953, Tess.)

@clerk/testing's internal loaded() helper has an unguarded second wait

• Superseded (OBJ-2687): the signOut → signIn chain that caused this race is gone — see "Clerk.signOut() with no sessionId logs out every session sharing the client, not just the caller's" below. clerk-sign-in.spec.ts now runs in an isolated, already-signed-out context and never calls clerk.signOut(), so this exact race can no longer occur in that file. Kept for history/pattern-recognition: if a future spec chains clerk.signOut() → clerk.signIn() in one context again, expect this same class of race.
• clerk.signIn()/clerk.signOut() can throw TypeError: undefined is not an object (evaluating 'window.Clerk.loaded') from inside the vendor library itself, not from any code in this repo — @clerk/testing's internal loaded() helper (node_modules/@clerk/testing/dist/playwright/index.mjs) does two separate page.waitForFunction calls back to back: () => window.Clerk !== undefined, then an unguarded () => window.Clerk.loaded with no ?.. It never re-checks between them. If window.Clerk becomes momentarily undefined again in that gap — e.g. while a preceding clerk.signOut() is still settling — the second wait throws that exact TypeError instead of resolving or timing out normally. clerk-sign-in.spec.ts's former signOut → signIn sequence was the only spec in the smoke suite that chained the two in one context, and had hit three distinct symptoms of this same race, seen only on mobile-safari under CI load: Clerk rejecting the fresh sign-in with "email_code is not enabled" (OBJ-1190), that same rejection recurring after a client-side null-session wait narrowed but didn't close the window (OBJ-1566), and this vendor-internal TypeError (OBJ-2529). Chasing a fourth error-message string is a losing pattern — OBJ-2529 replaced signInWithRetry's message-matching with an unconditional retry (still bounded to 3 attempts) so any future new shape of this same race is covered without another patch. If a future spec touches this pattern again, prefer widening what's retried over adding another message.includes(...) branch — but prefer avoiding the signOut→signIn chain entirely (isolated context) over patching around its races.

Clerk.signOut() with no sessionId logs out every session sharing the client, not just the caller's

• Clerk.signOut() called with no arguments deactivates every session on the current Clerk Client, not just the session the caller cares about — and every Playwright browser context that loads the same storageState snapshot shares that same underlying Clerk Client id server-side (the __client cookie value is copied verbatim into every context that loads the file), even though Playwright treats them as fully isolated browser contexts/processes. clerk-sign-in.spec.ts inherited the suite's default storageState: AUTH_FILE (used by ~150 other specs across all 3 browser projects) and called clerk.signOut({ page }) — no sessionId — to clear it before signing back in. Any of that spec's 3 concurrent project instances (chromium/mobile-safari/mobile-android, all racing to run sometime in the ~10-minute suite) could fire that signOut mid-run and revoke the storageState session server-side for every other spec still using it, in any project — those specs then landed on /sign-in and failed on whatever unrelated locator they were mid-assertion on, reading as a wave of unrelated product bugs or generic environment flakiness rather than the one real cause (the exact failure class fixtures/smoke.ts's OBJ-1729 annotation was built to attribute, but couldn't explain the source of). Confirmed via staging run 32112241677: 16 unrelated specs across communities/goals/monetization/platform all failed within the same ~90s window, every one's error-context snapshot showing the Clerk sign-in page instead of its target route. Fix (shipped, OBJ-2687): clerk-sign-in.spec.ts now runs in an isolated, already-signed-out context (test.use({ storageState: { cookies: [], origins: [] } }), the same pattern clerk-sign-up.spec.ts already used) — nothing shared to sign out, so the dangerous clerk.signOut() call is gone entirely, not just scoped. General rule: any spec that calls clerk.signOut()/window.Clerk.signOut() without an explicit sessionId must run in a context that does NOT inherit the suite's shared storageState — either isolate it (test.use({ storageState: { cookies: [], origins: [] } })) or pass signOutOptions: { sessionId } scoped to a session that spec itself created. Never call unscoped signOut() from a context that reused a storageState snapshot other specs also depend on.

Playwright's request fixture/APIRequestContext mechanism can 401 inside one specific CI job even with zero storageState — swap to plain fetch(), don't chase a second cookie theory

• Correction (OBJ-2773 R2): the storageState-cookie-leak theory below is disproven — kept for the record, see "What actually happened" below. The originally-shipped fix (request.newContext(), a dedicated storageState-free APIRequestContext) still 401'd identically in the next staging run, with zero stored cookies by construction — ruling out session/cookie leakage as the cause entirely.
• Original (R1) theory, disproven: username-less-sign-in.spec.ts's beforeAll used Playwright's request fixture for a config-check GET /v1/environment call. Every browser project (chromium/mobile-safari/mobile-android) shares one use.storageState snapshot — the shared smoke account's session, captured by a real browser sign-in in auth.staging.setup.ts. R1 theorized that storageState's Clerk FAPI cookies were leaking into the otherwise-anonymous call and getting 401'd when the shared session went stale. This was falsified, not merely incomplete: Vicki's post-merge staging run (32367118992) hit the identical CONFIG CHECK FAILED: GET /v1/environment returned HTTP 401 on all 3 browser projects, on a context carrying no cookies at all.
• What actually happened: Tess's R2 investigation confirmed the endpoint itself was healthy and needed no auth — auth.staging.setup.ts's plain fetch() call to the identical URL succeeded in the same CI job both times, and an out-of-CI probe (raw curl plus a standalone Playwright APIRequestContext) both succeeded too. So the failure is specific to Playwright's request mechanism running inside this one CI job — candidates are a burst of near-simultaneous calls (one per browser project's beforeAll) tripping rate-limiting at Clerk's Cloudflare-fronted edge, or a client-fingerprint difference between Playwright's HTTP client and Node's native fetch. Neither was confirmed — don't cite one as settled if this resurfaces.
• Fix (shipped, PR #2522, confirmed live on staging run 32374621264): every FAPI call in the file (the config-check plus all 5 protocol-trace calls) now uses plain fetch() instead of Playwright's request fixture/APIRequestContext — matching the one call pattern already proven to work twice in this exact job (username-less-sign-in.spec.ts:152,209,220,234,273,285). extractToken's header access was adapted from Playwright's headers() method to fetch's Response.headers.get(); form bodies from Playwright's form: option to URLSearchParams.
• General rule: a plausible, mechanism-level theory (session leakage, stale cache, race condition) is not confirmed until the actual failing gate — not just a local repro or a typecheck — goes green. R1 shipped a real, sensible-sounding fix for a genuine Playwright behavior (the shared request fixture does inherit storageState — that fact alone is still true) but never verified it against the staging playwright-smoke run it existed to fix; that run is what caught the false theory before a second, unconfirmed one nearly compounded it. When a fix targets a "gap between two client mechanisms" (Playwright's HTTP client vs. Node's fetch) rather than a named root cause, don't write up the untested candidate causes as if one were confirmed. (Root cause: OBJ-2773; R1 theory Tess/PR #2518, disproven by Vicki's post-merge staging run; R2 fix Tess/PR #2522, confirmed by Vicki's staging run.)

Clerk-hosted <SignUp>/<SignIn> field requirements drift on the dashboard side with zero repo signal

• A Clerk dashboard config change (CAPTCHA mode, legal_consent_enabled, required fields) can silently redraw the hosted <SignUp>/<SignIn> form with no corresponding repo commit — smoke specs interacting with those forms need periodic re-verification against a live trace, not just when CI reds. OBJ-2625 hit this twice inside one issue: the original symptom (clerk-sign-up.spec.ts timing out waiting for the OTP input, zero POST /v1/client/sign_ups in the trace) was first traced to the unpinned @clerk/clerk-js@6 CDN bundle and fixed by pinning the version in clerkPlugin's init (PR #2397, a real, independently-valid fix) — but the spec still failed post-merge with the same symptom, because Clerk's dashboard also has legal_consent_enabled: true and the hosted form now renders an inline consent checkbox the spec never checked (Clerk's own client-side validation silently blocked the click before any request fired — no exception, no network call, just a validation callout in the DOM). docs/architecture/authentication.md:260 already documented legal_consent_enabled: true from OBJ-2599 — landed the same day as the CDN pin — but scoped only to the native OAuth-transfer path; nobody cross-referenced it to the primary web sign-up smoke spec, which is part of why this cost a second merge-and-UAT round-trip (PR #2399 added a count() > 0-guarded checkbox check, mirroring the file's existing username/password conditional). This is the second time this exact form's field set has drifted with zero repo signal — the first was username/password becoming conditionally present/absent (the pre-existing count() > 0 guards in clerk-sign-up.spec.ts), this was the consent checkbox. General rule: when a smoke spec interacts with a Clerk-hosted component, don't assume its field set is fixed just because nothing in this repo changed it — re-verify against a live trace periodically — and when a repo doc records a Clerk-dashboard-side fact (like legal_consent_enabled), cross-reference it from every surface that fact affects, not just the one that prompted the doc entry. (Root cause: OBJ-2625, Dave's diagnosis + Tess's fix, PRs #2397/#2399.)

registerFlagOverrides only reaches the client — a server-side FeatureFlagService.enabled? check is untouched by it

• registerFlagOverrides (ionic_frontend/tests/smoke-playwright/helpers/flagOverrides.ts) intercepts only the browser's own PostHog /flags fetch — any resolver that independently calls FeatureFlagService.enabled? (rails_api/app/services/feature_flag_service.rb) hits real PostHog server-side, keyed on the account's own distinct_id, completely untouched by the client override. A spec that forces such a flag on and then asserts on the branch it's meant to unlock is asserting on a precondition it never actually established — at the flag's real rollout percentage, the backend can silently take the opposite branch. This isn't one call site: FeatureFlagService.enabled? gates well over a dozen backend mutations/resolvers/jobs (async_ai_delivery_enabled, seasonal_events_enabled, teams_leaderboards, privacy_controls_v1, connected_apps_enabled, discovery_ranking_v2, and more) — grep -rn "FeatureFlagService.enabled?('<flag-key>'" rails_api/app before writing a spec that forces one via registerFlagOverrides, and if it's also checked server-side, don't assert on that branch — there is no mechanism today to force FeatureFlagService.enabled?'s answer for a specific account. Surfaced (OBJ-3999): coach-reconnect-grace.spec.ts forced async_ai_delivery_enabled on and asserted on the async-delivery path, but Mutations::Ai::GetAdvice#resolve (get_advice.rb:46) checks the same flag server-side; at 0% real rollout the demo account always took the sync branch, so the code under test was never reached — the observed timeout was the sync branch's real LLM latency outrunning a budget sized for the async path, not a reconnect-grace regression. Fix: quarantined (test.skip(true, ...), commit 41e7dbf0c) rather than asserted on an unverified precondition — same treatment for any other spec that hits this shape. See docs/development/smoke-test-env-gotchas.md (Gotcha 27) for the full incident and the grep-before-you-write rule.

Concurrent smoke suites racing a shared demo account

• Concurrent CI smoke suites racing an unlocked shared demo account produce intermittent, hard-to-attribute UI assertion failures — journey.spec.ts mutates the shared, persistent demo@objectuve.com staging account on every run; with no concurrency: block on playwright-smoke.yml, the Staging Deployment's own post-merge suite and every open PR's Preview suite could race against it, leaving the account in a state neither run expected (getByText("Today's practice") / getByText("My Goals") locator misses). Five fix attempts across three PRs treated this as a single-request cost problem (warmup timing, then a GOALS_QUERY N+1) before the concurrency root cause was found — each shipped a real, worthwhile fix that nonetheless never touched the actual defect. A hand-rolled gh run list-polling lock (re-scope #3) then hit the crew's 3-round fix cap on its own mechanism (timeout → livelock → TOCTOU) before being retired for native concurrency: groups on both lanes plus giving Preview its own dedicated preview-smoke@objectuve.com account (OBJ-1150 re-scope #4, PR #1305) — removing the shared-account collision by construction instead of racing to win it. See docs/operations/smoke-account-concurrency-gotcha.md for the full post-mortem, including the isolation-audit lesson (audit every fallback-auth call site, not just the primary one) and a follow-up correction to the Preview lane's concurrency-cancellation policy (OBJ-1175). Superseded: Milestone v4.4 (OBJ-1322/OBJ-1325/OBJ-1326) further split the single shared preview-smoke@objectuve.com account into a per-PR account (preview-smoke+pr-<N>@objectuve.com) and re-scoped the concurrency group per-PR — see Preview Smoke Lane Isolation — decision record.

Preview smoke account is provisioned and torn down from three sources, not one

• The Preview-lane smoke account is bootstrapped from three separate places that must stay in lockstep, not two — rails_api/lib/tasks/smoke_accounts.rake's smoke:ensure_preview_account task is the long-term reconciliation path, run on every staging.yml deploy. Its smoke:teardown_preview_account counterpart exists (unit-tested in rails_api/spec/lib/tasks/smoke_accounts_spec.rb) but has no automated CI invocation today — no workflow calls it. Instead, .github/workflows/preview.yml's "Ensure PR-scoped preview smoke account exists" job and .github/workflows/preview_teardown.yml's "Teardown PR-scoped preview smoke account" job each run their own inline rails runner script — deliberately, because Preview never builds/deploys the backend, so a Preview run bootstraps the smoke account against whatever image enkidu-api-staging is currently serving, not this PR's own code (root cause OBJ-1150; provisioning mechanics: Playwright smoke gate — per-PR smoke account and concurrency isolation). Fixing only the rake task's provisioning logic (e.g. adding the Team-bootstrap block for OBJ-2516: a bare User with no Team renders TeamHomeView's error branch) does nothing for a Preview-lane PR — Preview never executes that code path at all.
• Triple-sync hazard: until OBJ-2516, nothing kept the three sources' model coverage in lockstep — a model added to the rake task's ensure/teardown logic without the matching update to both inline CI scripts (or vice versa) stayed invisible until a PR actually needed the drifted path, which is exactly what blocked OBJ-2516's own UI-evidence capture. Guard (shipped): scripts/check-smoke-account-parity.mjs + the Smoke Account Parity Guard workflow (.github/workflows/smoke-account-parity-guard.yml) now fail CI whenever one of a fixed model-marker list (User, Team, TeamMembership, AiUsageEvent, CoachConversation) appears in one of the three sources' ensure/teardown logic but not the other two — triggered on any PR touching the rake task or either workflow file. Adding a new model to the provisioning/teardown cascade means updating all three sources and adding the model to MODEL_MARKERS in the parity script — an unlisted model isn't checked. (Root cause: OBJ-2516; guard shipped same issue.)
• OBJ-2516 fixed "zero teams"; OBJ-2378 found "one team" was still not enough — TeamSwitcher.vue only mounts once myTeams.length > 1 (its actual reachability gate), so a smoke account capped at exactly one TeamMembership still left the header switcher structurally uncapturable by capture-ui-evidence, even though TEAM_HOME_QUERY resolved fine. Both provisioning sources (the rake task and preview.yml's inline bootstrap — preview_teardown.yml's teardown script needed no change, since it already destroys .where(user_id:)/.where(billing_owner_id:) generically with no count assumption baked in) now idempotently top up the smoke account to 2 teams, both billing-owned by the account itself. This is a count regression, not a missing-model one, so the marker-based check-smoke-account-parity.mjs guard above can't catch it — a model appearing in all three sources says nothing about how many rows each source creates. Guard (shipped): ionic_frontend/tests/unit/smoke-team-provisioning-parity.spec.ts asserts the rake task and preview.yml's inline script share the same loop-until-2 shape, self-billing-ownership, and owner role assignment — complementary to the marker guard, not a replacement for it. (Root cause: OBJ-2378, PR #2644.)
• OBJ-2516/OBJ-2378 fixed team reachability; OBJ-3111 found the repair-offer surface had never been reachable at all — StreakRepairCard.vue only renders when useStreakRepair.ts's shouldShowRepairCard is true, which requires Gamification::StreakRepairOfferResolver#find_repair_target (rails_api/app/services/gamification/streak_repair_offer_resolver.rb:66-74) to find a habit goal satisfying all of: recurrence_type present, completed: false, streak_repair_eligible_until in the future, pre_break_habit_streak > 0, streak_repaired_count == 0. Those fields are normally set by ProcessHabitStreaksJob when a real streak breaks — nothing in the smoke bootstrap ever ran that job, so capture-ui-evidence's UI-Evidence-Interact: click:button[aria-label*="Bring it back for"] step had no button to click, since the smoke account was created (not a regression). Fix (shipped): both provisioning sources now idempotently re-assert a "Smoke Habit Repair Eligible" goal on identity A (assign_attributes, not find-or-skip, since the 48h streak_repair_eligible_until window would otherwise quietly expire on a long-open PR) and top up the account's XP off the resolver's own COST_BASE/COST_FACTOR/MAX_COST constants via GamificationService.award_xp — not a raw User#update!, which would trip scripts/check-shared-kernel-writes.mjs's shared-kernel guard and leave user.level stale, since only award_xp runs the level-up check. XP is topped up by the shortfall only, so re-running every push doesn't grow it unbounded. The "Smoke Habit " name prefix keeps the goal inside both teardown scripts' existing name ~ '^Smoke (Habit|Goal) ' cleanup regex — no teardown change needed. Not covered by check-smoke-account-parity.mjs's MODEL_MARKERS (both sources provision the goal via the user.goals/u.goals association, never a literal Goal. class reference, and "Goal" is too common a word in the rake task's own prose comments for a marker to stay false-positive-free) — parity is guarded instead by a dedicated spec, ionic_frontend/tests/unit/smoke-streak-repair-provisioning-parity.spec.ts, asserting the rake task and preview.yml's inline script share the goal name, every resolver-required field, and cost-constant sourcing. Watch for silent no-ops when re-verifying this class of fix: a capture-ui-evidence run can go green on a directive-parsing no-op (Parse UI-Evidence-Interact producing zero output, INTERACT_STEPS falling back empty) without ever clicking anything — confirm via the raw job log that the parser actually resolved the PR body's directive, not just that the job exited 0. (Root cause: OBJ-3111, surfaced on OBJ-3079/PR #2736; fix PR #2746.)
• A fourth source also touches the Preview-lane account, and it's a teardown, not a provisioner — demo-cleanup.teardown.ts (ionic_frontend/tests/smoke-playwright/setup/demo-cleanup.teardown.ts), a Playwright setup-project teardown meant to run only against the real, shared demo@objectuve.com production account. Its skip guard trusted process.env.SMOKE_ENV === 'production' as proof it was operating on that real account — it wasn't: playwright-smoke.yml's "Resolve smoke environment" step deliberately maps Firebase preview-channel (*.web.app) URLs to smoke_env=production too (OBJ-644, for an unrelated reason — routing Preview onto the FALLBACK_AUTH_SECRET path since those domains can't complete a Clerk browser flow), even though Preview always runs against its own dedicated preview-smoke+pr-<N>@objectuve.com account, never DEMO_EMAIL. So the teardown ran on every Preview PR too, and since it soft-deletes every goal matching /smoke|^SAFE-\d/i, it caught the Preview account's own "Smoke Habit Repair Eligible" fixture goal mid-run — the exact deletion this doc's OBJ-3111 bullet above traces. Fix (shipped): a new isDemoFallbackAccount() helper (ionic_frontend/tests/smoke-playwright/helpers/env.ts) checks the actually-resolved fallback email (getFallbackEmail(), which honors SMOKE_FALLBACK_EMAIL before defaulting to DEMO_EMAIL) against DEMO_EMAIL, not the SMOKE_ENV label; demo-cleanup.teardown.ts now skips unless both the SMOKE_ENV guard and isDemoFallbackAccount() pass. General rule — reuse isDemoFallbackAccount(), don't re-derive: any future "production-only, and this also touches the fallback-auth account" check needs the same identity-not-label test — SMOKE_ENV/smoke_env records which lane is running, not which account a step's fallback token actually resolves to, and those two have diverged before and will again. (Root cause: OBJ-3111 Task B, Codi; confirmed by Roy's independent re-verification against the raw job log.)

Production smoke: demo goal-seeding guard must not be removed

• smoke_env=production: demo account goal-seeding guard must not be removed — auth.production.setup.ts creates a baseline goal when the demo account has 0 active goals. Without it, hasNoGoals=true and the dashboard renders Branch A (no GoalsSection, no "Create new goal" button), silently breaking any spec that asserts goal-related UI. auth.staging.setup.ts has the same self-healing guard (OBJ-744): it checks goals count inside the if (dismissToken) block, recreates the baseline goal if zero, and throws a hard assertion error if still zero after the mutation. Both files must be kept in sync — consistent with the sync warning in Gotcha 1. See docs/development/smoke-test-env-gotchas.md.

Production smoke: missing coach persona redirects from assessment

• smoke_env=production: missing coach persona redirects away from /enneagram-assessment — Navigating to /enneagram-assessment when the demo account has no coach persona configured triggers an internal redirect to /coach. Any spec that asserts assessment content without first checking page.url() will time out on the wrong page. Pattern: use waitForURL to detect the redirect, then test.skip() if the URL changed. See enneagram-assessment.spec.ts for the reference implementation, and docs/development/smoke-test-env-gotchas.md for full details.

Demo account is a permanent paid Supporter — free tier untestable in PR CI

• demo@objectuve.com is permanently seeded as a paid Supporter; PR-preview-smoke accounts are always free-tier, so the paid-state branch of Supporter-gated UI is untestable from PR CI — rails_api/db/seeds/production.rb:416-428's "Demo Supporter status" block re-asserts supporter_tier: 'yearly', supporter_until: 11.months.from_now on the shared demo@objectuve.com account on every seed run, by design, so the /sign-in?demo experience always showcases the paid state (mirrored for staging/dev in rails_api/db/seeds/development/users.rb:196-208, Raya's yearly tier). Any smoke/E2E assertion touching Supporter-tab copy must branch on isSupporter the way SupporterHero.spec.ts's unit test already does — never assume the free-state copy/heading is reachable via the demo account on staging or production. Meanwhile rails_api/lib/tasks/smoke_accounts.rake's ensure_preview_account task provisions a fresh preview-smoke+pr-<N>@objectuve.com account per PR and never sets supporter_tier — PR-level preview-smoke accounts are always free-tier, so the paid-state branch of any Supporter-tab (or other supporter-gated) code is structurally untestable from PR CI; only staging/production smoke exercises it. Separately, this is not a "no alerting" gap: staging's post-deploy playwright-smoke job has a dedicated alert job, notify-smoke-failure (.github/workflows/staging.yml:1207), that posts a Slack alert on every real failure — confirmed by checking Slack's own ok response body, not just the job's conclusion, across all 8 real failures Dave found in OBJ-3879 (OBJ-3898). The alert also escalates with an @-mention once a real-failure streak reaches 3 consecutive runs (OBJ-3898, merged via PR #3276), instead of repeating the same unescalated line. (Root cause: OBJ-2147 production regression, Settings → Supporter tab — a stale smoke-spec assertion, not a seed/account bug; spun off as OBJ-2149.)

Post-networkidle watchers in Dashboard.vue open blocking overlays

• Smoke: post-networkidle watchers in Dashboard.vue can open overlays that block clicks — watch(actions, …) at Dashboard.vue:1275 fires after the USER_QUERY resolves (after networkidle) and opens BadgeModal.vue via Teleport at z-index: 99999 when the demo account has an unacknowledged streak_comeback action. This blocks all pointer events on the page (~32 s timeout). The production storageState does NOT include dismissed-actions in localStorage, so the guard is always necessary. Apply the waitFor visible → click → waitFor hidden → catch pattern (same as the WelcomeBack guard) to every modal-opening watcher before clicking dashboard UI. See docs/development/smoke-test-env-gotchas.md (Gotcha 5) for the pattern and the full watcher inventory.

A first-ever-visit surface tour auto-fires and blocks pointer events — objectuve_suppress_tours is the deterministic opt-out

• Any of useFeatureTour.ts's five Phase 2 surface tours (Coach, Goals, Communities, Achievements, Activity Feed) can auto-fire a blocking overlay on a smoke spec that's simply first to visit that surface's route against an account that hasn't dismissed the tour yet. Unlike dashboard-first-run, which only fires behind the explicit ?source=wizard trigger (docs/features/feature-tour.md), the five Phase 2 tours fire on a plain first-ever route visit by design — intended onboarding UX, not a bug (docs/features/feature-tour.md). coach-reconnect-grace.spec.ts (OBJ-3909/PR #3305) was the first smoke spec to ever hit /coach fresh and hung for the full 150s timeout: coach-tour's fixed-position overlay intercepted every pointer event on the coach chat send button. Fix (shipped): toursSuppressed() (useFeatureTour.ts:63-71) reads a objectuve_suppress_tours localStorage flag, OR'd into start()'s existing bail-out (:94) — set it via page.addInitScript before page.goto, mirroring AchievementWatcher.vue's existing objectuve_suppress_celebrations opt-out. Not global — opt in per spec, not automatic like objectuve_suppress_celebrations. objectuve_suppress_celebrations is set centrally, in shared setup (auth.staging.setup.ts, auth.production.setup.ts, smoke.warmup.ts), so every smoke spec already gets it. objectuve_suppress_tours is set nowhere centrally — only inside coach-reconnect-grace.spec.ts itself — so this hazard is still live for any future spec that's first to visit /goals, /communities, /achievements, or /feed against a fresh account; each such spec needs its own page.addInitScript call, the same way coach-reconnect-grace.spec.ts:65-71 does. (Root cause: OBJ-3909/OBJ-3932, Tess; PR #3313.)

Smoke: getByText(name) is a strict-mode violation on goal pages

• Smoke specs: getByText(goalOrHabitName) on a goal detail page is a strict-mode violation — RoadmapCapstone.vue re-renders the same name in a <span> — Scope name assertions to the page's real <h1> (GoalHero.vue) via getByRole('heading', { name, exact: false }) instead. Same collision risk applies to .or() combinators over a desktop/mobile variant pair (e.g. MeetCoach.vue's heading + "Setup progress" bar) — CSS hidden lg:block / lg:hidden classes don't remove either element from the DOM, so .or() resolves to 2 matches; poll each locator's own isVisible() independently instead. Also: Clerk's hosted <SignUp> form now requires username + password in addition to email (fill them when present, .count() > 0 guard), and GoalCreate.vue defaults to the Coach-drafted hero phase (OBJ-1082) on a bare /goals/create visit — use the ?source=wizard deep-link to reach the manual form. See docs/development/smoke-test-env-gotchas.md (Gotcha 17) for all four patterns with code. (Source: OBJ-1088, Tess R1-R3, PR #1248.)
• Recurrence — same trap on the Achievements grid, not a goal/habit page. 12-achievements-populated.spec.ts scoped getByText('Unlocked') to [aria-label="Achievements grid"] expecting only the "Unlocked" section <h3> (AchievementsGrid.vue) to match. AchievementCard.vue also renders a per-card community-stat line, "{N}% unlocked" (unlock-pct), on every badge carrying a BADGE_STATS_QUERY percentage — getByText's default case-insensitive substring match resolves 'Unlocked' against every one of those too, so any populated account with more than one such badge (the common case) hits a strict-mode violation deterministically, not flakily. Fix (shipped): getByRole('heading', { name: 'Unlocked', exact: false }) scoped to the same grid region — unlock-pct is a <div>, not a heading, so it's structurally excluded regardless of its text. General rule: this pattern recurs whenever a short, common word used in a section heading ("Unlocked", "Completed", "Locked") is also legitimately reused as a text fragment elsewhere in the same scoped region — grep the target component tree for the assertion word before trusting a bare getByText scope, and prefer getByRole('heading', ...) for section-title assertions the same way the goal-page fix above does. (Root cause: OBJ-3772, Tess R1.)

Unbounded User list fields need a smoke-account purge cron

• An unbounded User/UserType list field will silently accumulate unbounded residue on the shared demo@objectuve.com smoke account unless a dedicated cron purges its backing table — DemoData::ClearDemoDataJob only covers Goal — smoke-goal-cleanup.yml (OBJ-799, extended to staging under OBJ-1333) purges stale Goal residue daily, but nothing purged UserAction/UserNotification/UserFeedItem until OBJ-1609, even though USER_QUERY (ionic_frontend/src/constants/graphql/user.js:143) requests all three as unbounded lists (Types::UserType, rails_api/app/graphql/types/user_type.rb:39-44) and AchievementWatcher.vue polls that query globally every 10s. Years of unpurged smoke-run history on the shared account eventually cost 15–17s server duration / ~4.8–4.9M Ruby object allocations per UserQuery call — enough to starve the concurrently-running staging 02-onboarding Playwright spec (Dave's OBJ-1608 round-2 diagnosis). Fix (shipped): smoke-activity-cleanup.yml, a sibling cron to smoke-goal-cleanup.yml, runs demo:trim_activity (rails_api/lib/tasks/demo.rake) daily against both environments and unconditionally hard-deletes (delete_all; none of the three models have destroy callbacks) all three tables' rows for the shared account. No GraphQL mutation exists to delete these rows (unlike Goal's updateGoal(deleted: true), which smoke-goal-cleanup.yml uses), so this reuses the same rake-task-via-Cloud-Run-Job admin path already used by demo:reset/db:migrate/seed:templates rather than adding a new delete surface. General rule: before adding an unbounded list field to User/UserType, or before assuming one is already bounded, check whether a smoke-*-cleanup.yml cron actually purges its backing table on the shared demo account — the two existing crons (Goal; UserAction/UserNotification/UserFeedItem) cover exactly the tables named in their own workflow files and nothing else. (Root cause: OBJ-1608/OBJ-1609, PR #1670.) Follow-up (shipped) — the actual pagination bound, and the trap it created: OBJ-1617 (PR #1683) gave Types::UserType#actions/#notifications/#feed_items (rails_api/app/graphql/types/user_type.rb:42-62) a limit argument (default 50, hard-clamped to 1–200 via clamp_collection_limit, rails_api/app/graphql/types/user_type.rb:335-345,353-355 — note limit: nil sent explicitly, not merely omitted, must also fall back to the default; an early revision only handled the omitted-argument case and crashed on nil.clamp), and both collections now return created_at DESC. Trap: capping a collection that also backs a derived unlock/membership check silently breaks that check past the limit boundary — it doesn't error. Three frontend consumers — Achievements.vue, SideMenu.vue, Dashboard.vue — used to scan the raw actions list directly to compute which badges were unlocked; once USER_QUERY (ionic_frontend/src/constants/graphql/user.js:198) started sending actions(limit: 50) over the wire, any account with more than 50 lifetime actions would have started reporting incorrect badge/achievement state with no error signal. OBJ-1617 pre-empted this by shipping a companion field, unlockedAchievementKeys: [String!]! (object.actions.distinct.pluck(:action), rails_api/app/graphql/types/user_type.rb:366-368), proven independent of limit by a spec seeding 5,000 actions across 30 keys and asserting all 30 keys return while actions itself is capped at 50. OBJ-1618 (PR #1689) then migrated all three consumers off the raw actions scan onto unlockedAchievementKeys (ionic_frontend/src/views/Achievements.vue:331,342, ionic_frontend/src/components/SideMenu.vue:377-378, ionic_frontend/src/views/Dashboard.vue:783,1587) in the same change that added limit: 50 to USER_QUERY's actions/notifications/feedItems (ionic_frontend/src/constants/graphql/user.js:198,206,214) — the limit and its escape hatch shipped together, per Task 2's explicit "do not ship the limit without this field" instruction. One consumer, AchievementWatcher.vue (ionic_frontend/src/components/achievements/AchievementWatcher.vue:70-106), still watches the raw, bounded, newest-first actions list — that's safe there only because it's diffing for new unlocks arriving this session against a local seen-set, not performing a complete unlock-history check; don't treat it as a precedent for reading full badge state off actions. General rule: never ship a limit on a collection that also backs a derived boolean/set computation (unlock state, membership, dedup) without adding an independent, limit-proof field for that computation in the same change, and test the nil-vs-omitted-argument distinction explicitly when a GraphQL argument has a default_value — GraphQL's argument coercion treats an explicit null as "provided," so the field's own default does not kick in and a naive .clamp/method call on the bare argument will raise. (Root cause: OBJ-1608 Tasks 2–3, PRs #1683/#1689.)

Side-menu-gated assertions must be chromium-only

• Any smoke assertion on an element inside SideMenu.vue's v-if="isExpanded" block will time out 100% of the time on mobile-safari/mobile-android, misreading as an auth or environment flake — useMenuState's watch(isMobile, ...) forces isExpanded.value = false whenever the viewport matches matchMedia('(max-width: 767px)'), and both mobile smoke projects (devices['iPhone 15'], devices['Pixel 7'], ionic_frontend/playwright.config.ts:80-90) are narrower than that breakpoint by design — the rail collapses to an icon-only strip and every isExpanded-gated node (.version-info, .nav-label, .footer-label, the profile section, etc.) simply never renders there. There is currently no UI path that re-expands the rail on mobile (toggleMenu() in useMenuState.ts is exported but never called by any component — the ion-menu-button in AppHeader.vue has no matching <ion-menu> to control), so this isn't a timing race a longer timeout fixes — the element is structurally absent on those two projects, every run. 00-version-tag.spec.ts hit this for 3 consecutive staging deploys (OBJ-2513) and was initially misdiagnosed as a Clerk auth.staging.setup.ts fixture failure, because the failure surfaced on the same red CI run as unrelated concurrent-deploy flakes (see the gotcha above) — always check which test actually failed in the Playwright report before attributing a smoke-suite failure to the auth setup step. Fix (shipped): test.skip(testInfo.project.name !== 'chromium', '...'), the same pattern settings-profile-photo-upload.spec.ts/settings-profile-photo-purge-latency.spec.ts already use for a different reason — chromium's desktop viewport keeps the rail expanded by default, and a deploy-tag check only needs to run once. General rule: before writing or reviewing any smoke assertion that targets .version-info, .nav-label, .footer-label, or any other isExpanded-gated SideMenu.vue node, either scope the test chromium-only or explicitly drive the mobile UI to the state that renders it first — don't assume page.goto() alone puts a mobile viewport into the same DOM shape as desktop. (Root cause: OBJ-2513, Tess R1.)

username.required flip stranded a pre-existing Clerk smoke user in a pending session

• A Clerk instance flipping attributes.username.required to true doesn't just gate new sign-ups — an existing user missing that field gets a pending, not active, session on every future sign-in, and @clerk/testing's clerk.signIn() never resolves the follow-up requirement — auth.staging.setup.ts's ensureStagingUser(email) only provisioned the +clerk_test smoke user (username set) when getUserList found zero matches; once the user existed, every later run skipped provisioning unconditionally, so an account created before OBJ-2722's username.required: true flip (scripts/clerk-auth-config.expected.json) stayed permanently username-less. clerk.signIn() completed the first factor without throwing, Clerk.loaded resolved fine, but the app never observed a fully active session — the SPA sat on its static index.html "Loading Objectuve" splash forever, and page.waitForURL(/dashboard|welcome/, { timeout: 20_000 }) timed out on every attempt, 2 of 2 runs after the flip. Only a real Clerk sign-in (auth.staging.setup.ts) can hit this — auth.production.setup.ts uses the fallback-token path and has no Clerk session to leave incomplete. Fix (shipped): ensureStagingUser now backfills username on an existing user too (clerkClient.users.updateUser(user.id, { username })) when user.username is blank, not just at creation — idempotent, so it's a no-op once the account has a username. The post-auth page.waitForURL timeout was also widened 20s → 30s; that budget had no CI-variance headroom of its own the way the Clerk.loaded wait above it does. General rule: a "create if missing" account-provisioning guard for a third-party identity provider needs a matching "still meets current requirements" check on the existing path too, or a provider-side field requirement can strand an old test fixture indefinitely with a symptom (post-auth navigation hang) that looks nothing like a missing-field error. (Root cause: OBJ-2722, Tess R1.)

• Follow-up — once an attribute is required, no Backend API path can manufacture a subject that lacks it, not even to test the gap above. OBJ-2729's COHORT-1 reproduction tried to build a fresh username-less Clerk staging user to trace the sign-in/lockout behavior directly, and hit the same username.required flip from the creation side instead of the existing-account side this section already covers: clerkClient.users.createUser with no username → 422 form_data_missing (["username"] data doesn't match user requirements set for this instance); the fallback of creating with a placeholder username and then stripping it via clerkClient.users.updateUser(id, { username: null }) → 422 form_param_missing ("username must be included."). Confirmed identically for both a web-surface and a native-surface subject in the same staging run. General rule: Clerk's Backend API enforces a required-attribute flag on every mutation path that touches that attribute — createUser and updateUser alike — not just the hosted <SignUp>/<SignIn> forms. The practical consequence outlives this incident: once an instance requires an attribute, you cannot manufacture a subject that lacks it through any API path, so a config flip can create a user cohort that is, from that moment on, impossible to reproduce in a test — the only surviving instances are accounts that predate the flip and haven't since been touched by a self-heal guard like the one directly above.

• A second, independent Backend API requirement surfaced in the same reproduction: legal_accepted_at must be supplied up front when the instance has sign_up.legal_consent_enabled: true, unlike the hosted <SignUp>/<SignIn> form, which collects consent post-hoc via its own inline checkbox. clerkClient.users.createUser without legalAcceptedAt 422s with form_param_missing: "legal_accepted_at must be included" on both staging and production (both have legal_consent_enabled: true). This is also why ensureStagingUser's fresh-create branch (above) had never actually been exercised since the consent flip — every real CI run hits the already-exists+backfill branch, never fresh creation — until it was fixed to pass legalAcceptedAt explicitly (PR #2479). General rule: don't assume a Backend API create call mirrors the hosted form's field/timing requirements — a hosted form can legitimately defer a requirement the Backend API enforces synchronously at creation, and the only way to know is to read the full error body, not just the HTTP status. (Root cause: OBJ-2722 flip, reproduced via OBJ-2729, Tess.)

A provisioning fixture for a third-party identity provider must satisfy its current required-field set, not a snapshot from when the fixture was written

• Generalizes the username.required flip gotcha above. A Clerk-hosted <SignUp> form's required-attribute set drifted a second time — first_name/last_name joined username as required — and both clerk-sign-up.spec.ts (fills the real Clerk UI) and auth.staging.setup.ts's ensureStagingUser (provisions/backfills the smoke account via the Backend API) still only knew about the OLD required set, because both encoded a snapshot of Clerk's requirements taken when each was last written, not a live check. clerk-sign-up.spec.ts never filled firstName/lastName, so clicking Continue tripped the browser's native constraint validation on those two fields and the OTP step never rendered — otpSegments.first().waitFor() burned its full 40s budget and the run reported as "Clerk OTP verification timing out," a symptom that names the wrong layer entirely; nothing was wrong with OTP or email delivery. ensureStagingUser's existing-user backfill was narrower still: if (!user.username) only, so a user provisioned before the flip and missing firstName/lastName would have been silently accepted and left to strand in a pending Clerk session on a later sign-in — the exact OBJ-2722 failure shape, latent rather than firing only because the shared staging smoke user's firstName/lastName happened to already be set. Fix (shipped): clerk-sign-up.spec.ts now fills firstName/lastName with the same count() > 0 conditional pattern already used for username/password/consent, so it tolerates the requirement disappearing again. ensureStagingUser no longer hardcodes a field list at all — it calls a new fetchRequiredClerkAttributes() that reads user_settings.attributes straight off the live /v1/environment FAPI endpoint (the same endpoint scripts/check-clerk-auth-config.mjs's drift guard polls) and backfills/creates whichever of username/first_name/last_name that response currently marks enabled && required — on both the create path and the existing-user backfill path. General rule: a provisioning fixture (or a UI-driving spec) for a third-party identity provider must satisfy the provider's current required-field set, not a snapshot of it taken when the fixture was written — encode "read what's required live," not "hardcode what was required on the day this was authored." The tell: the failure signature is always a timeout at whatever step comes after submission (OTP render, post-auth navigation, a pending-session hang) — never a missing-field error — because the provider's own client-side validation silently swallows the blocked attempt before any request fires. (Root cause: OBJ-2748, Tess R1; generalizes OBJ-2722.)

A live Clerk config window posted only on its own issue reads as an unexplained regression on a concurrent sibling gate

• An authorized, time-boxed live-config mutation window opened on one issue can present as an unexplained regression to a concurrent ship-gate check on a sibling issue in the same milestone, if the window is only posted on its own issue thread — Vicki HOLD'd Phase 1 (OBJ-2607) after a live re-probe found staging Clerk attributes.username.required reverted to false, reading as a regression of the same flip documented in "username.required flip stranded a pre-existing Clerk smoke user in a pending session" above. It turned out to be an authorized, time-boxed config window opened by a sibling issue in the same milestone (OBJ-2729/COHORT-1) to provision a test fixture: Vicki's probe landed ~30 seconds before Josh's own "window open" comment on OBJ-2729. Nothing on OBJ-2607's thread indicated a concurrent mutation was in flight elsewhere, because the window was only posted on the issue that owns the mutation (OBJ-2729), not on OBJ-2607, which reads/gates on the same attribute. General rule: when authorizing a live mutation of a shared external resource (e.g. Clerk instance config) that another concurrently active issue also reads or gates on, post the window (attribute, open time, expected close, owning issue) on both threads, or a shared coordination point — not just the issue that owns the mutation. (Root cause: OBJ-2607 HOLD / OBJ-2729 T2 window, 2026-08-19, Dave's diagnosis.)

admin_dashboard staging smoke: no setup-phase browser warm-up means the first spec eats the full CDN cold-start cost

• admin_dashboard's fallback-token setup project never opens a real page (request-fixture only), so unlike ionic_frontend's Clerk-based staging setup — which incidentally pays the Firebase Hosting CDN cold-start cost during its own generously-timed setup project run — admin's very first REAL browser navigation happens inside the timed chromium project's first spec, with only a 60s project-level test timeout and a 20s toBeVisible sub-budget — On a run right after Deploy Frontend to Firebase (Staging), curling the origin (Wait for Admin Dashboard step) returns instantly because Firebase Hosting's edge already has index.html cached, but that says nothing about the freshly hashed JS chunk files a real browser has to fetch — those are cold at the CDN edge until the first real client requests them. ops-board-capture.spec.ts (alphabetically first) ate that cold fetch and hit page.goto's 60000ms timeout outright (run 32200019689); its retry, and every subsequent spec in the same run, then landed right at the edge of the 20s toBeVisible({ name: 'Operations' }) / theme-toggle-button budget (18.4–21.1s observed) — close enough to flip pass/fail on run-to-run CDN/network jitter, reading as "flaky," not "cold." The Operations heading and the theme-toggle button are both rendered unconditionally in their component's template (not gated behind a data fetch), so the delay is bundle/route-chunk fetch time, not a slow GraphQL query. Fix (shipped): auth.staging.setup.ts now opens a real browser context (via the already-built storageState) and navigates to / and /ai-workforce with waitUntil: 'domcontentloaded' before the timed specs run — best-effort (a warm-up failure doesn't fail setup); the setup project's own timeout was bumped 60s → 120s to give those two navigations room; the three specs' toBeVisible timeouts were widened 20s → 30s as a second line of defense. General rule: an API-only Playwright setup project that builds storageState without ever loading a page provides zero CDN/bundle warm-up for the timed specs that follow it — if the app is deployed via a CDN-backed static host (Firebase Hosting, similar), either warm the routes the specs will hit during setup, or budget the specs' own timeouts assuming a cold edge. (Root cause: OBJ-2722, Tess R1.)

A fresh sign-up spec that navigates straight to any protected route (not just /dashboard//welcome) hits the mandatory onboarding gate

• A newly signed-up Clerk account has no onboardingCompletedAt, so evaluateOnboardingGate (router/index.ts) redirects it away from EVERY non-exempt route — including one a spec page.goto()s straight to right after sign-up, not just the first post-auth landing. settings-account-delete.spec.ts's OBJ-3059 "confirming deletion" test drove a real Clerk sign-up, correctly waited for /dashboard or /welcome (mirroring clerk-sign-up.spec.ts), then immediately page.goto('/settings') and asserted the Settings heading — which timed out 100% of the time on every staging run that actually executed it from the day OBJ-3059 merged (confirmed via gh run log archaeology across a dozen staging deploys, all identical getByRole('heading', {name:'Settings'}) timeout at the same line), because a fresh account always resolves to /welcome and the onboarding wizard is mandatory at 100% rollout (useOnboardingGate.ts) — /settings was never reachable from that state. clerk-sign-up.spec.ts already documents this exact fact inline (its own comment: "a fresh account has no completed onboarding, so evaluateOnboardingGate redirects to /welcome/start") but the fact lived only in that one file's comment, not centrally, so a second spec doing the same real-sign-up flow missed it. Fix (shipped): after the /dashboard-or-/welcome wait, check pathname.startsWith('/welcome') and if so click the wizard's own getByRole('button', { name: 'Skip', exact: true }) (WelcomeLayout.vue's handleSkip — sets onboardingCompletedAt and commits STORE_ONBOARDING_STATE_MUTATION with completedVia: 'skip', the same completion path DoneSlide uses) and wait for /dashboard before proceeding — a full 14-step wizard walkthrough is not warranted for a spec whose actual subject is account deletion, not onboarding. General rule: any spec driving a real fresh Clerk sign-up and then navigating to a route other than /dashboard//welcome themselves must account for the onboarding gate first — landing on /dashboard or /welcome is not the same as being onboarded, and only /welcome/* and /profile/complete are gate-exempt. (Root cause: OBJ-3311, Tess R1; generalizes the onboarding-gate fact clerk-sign-up.spec.ts already knew but didn't centralize.)

A page.waitForURL matching multiple landing routes can resolve on a transient one an instant before an async client-side redirect supersedes it

• The Skip-button fix directly above (OBJ-3311 R1) had its own race: reading page.url() synchronously right after page.waitForURL((url) => url.pathname.startsWith('/dashboard') || url.pathname.startsWith('/welcome')) resolves can observe the transient /dashboard pathname on a fresh Clerk sign-up, an instant before evaluateOnboardingGate's own redirect to /welcome lands. Clerk's post-signup force-redirect to /dashboard is a real document navigation that resolves first; the onboarding gate's client-side redirect to /welcome runs asynchronously afterward, inside the router guard, after syncClerkUser() resolves — so the two-route wait can win the race on the wrong side. R1's synchronous page.url() check landed on that pre-redirect /dashboard state and skipped the Skip-button click entirely (confirmed via staging run 33734256861's trace: no click between the wait and the /settings goto()), so the next hard navigation hit the onboarding gate fresh and bounced back to /welcome step 1 — misread at first as a backend persistence race (ruled out: StoreOnboardingState's DB write is synchronous, no async job in the write path) rather than a client-side navigation-timing race in the spec itself. Fix (shipped): after the initial /dashboard-or-/welcome wait, explicitly await page.waitForURL((url) => url.pathname.startsWith('/welcome'), { timeout: WARM_PAGE_HYDRATION }) inside a try/catch (tolerant of a timeout — onboarding is mandatory at 100% rollout, so /welcome is expected, but nothing to skip if the app never routes there) before trusting page.url()'s pathname to decide whether to click Skip. General rule: when a page.waitForURL matches multiple possible landing routes and one of them is itself a real third-party redirect that a same-request async client-side redirect can still supersede, don't branch on whichever route the wait resolved on — wait for the specific expected terminal route (bounded, tolerant of a timeout) first. (Root cause: OBJ-3311, Roy's post-merge trace via staging run 33734256861; source: ionic_frontend/tests/smoke-playwright/specs/platform/settings-account-delete.spec.ts:199-220 on master.)

A Node test spawning a scripts/*.mjs from os.tmpdir() must realpathSync the temp dir first, on macOS

• A Node test that spawns one of this repo's scripts/*.mjs entrypoint scripts from a directory under os.tmpdir() must realpathSync that directory before building the script's path, or the spawned script silently never runs its main() — locally, on macOS only. Every scripts/*.mjs entrypoint guards its main() call with if (import.meta.url === `file://${process.argv[1]}`). On macOS, os.tmpdir() resolves to /var/folders/..., itself a symlink to /private/var/folders/.... Node resolves that symlink when computing import.meta.url for the spawned module but passes process.argv[1] through literally (unresolved), so the two sides of the comparison never match. Symptom: the spawned process exits 0 with no stdout/stderr — main() silently never executes. A test asserting a nonzero exit fails with a confusing "expected 1, got 0" and nothing printed to explain why; a test written to assert exit 0 would pass vacuously, for the wrong reason. Fix: resolve the temp dir immediately after creating it — const dir = realpathSync(mkdtempSync(join(tmpdir(), 'prefix-'))) — before building the script path from it. realpathSync is a harmless no-op on Linux CI, where /tmp isn't a symlink, so this reproduces only locally on macOS, never in CI. (Found writing the exit-path test for scripts/check-ruleset-required-contexts.mjs, PR #3067.)

A resubscribed GraphQL subscription channel's own re-executed push can be a deliberate no-op

• Proving an ActionCable/GraphQL subscription channel resubscribed after an outage proves nothing about whether data lost during that outage was recovered — the subscription's own subscribe resolver can be a deliberate no-op by design, with the actual recovery living entirely in a separate HTTP fallback. Subscriptions::AiRequestUpdate#subscribe (rails_api/app/graphql/subscriptions/ai_request_update.rb:20-25) returns nil at subscribe time on purpose (payload_type is nullable for exactly this reason) — resubscribing the channel delivers no payload by itself. The actual recovery path is useAiRequest.ts's catchUp() (ionic_frontend/src/composables/useAiRequest.ts:70-78), a separate network-only aiRequest(id:) query fired from the channel's own reconnect callback (subscribeToRequest's useLiveQuery reconnect handler, useAiRequest.ts:80-93). A test or trace that only asserts the channel reopened is asserting the wrong layer — it can pass while the recovery it's meant to prove never fired. General rule: when a broadcast can be lost during a connectivity gap and the recovery design is "reconnect, then re-fetch," assert that the re-fetch actually fired (a request in the trace, a spy call count, a network log entry) — not just that the channel resubscribed. The channel reopening and the data recovering are two independent facts; a passing assertion on one says nothing about the other. Worked example: Dave's OBJ-3953 round-2 diagnosis found zero aiRequest(id:) catch-up queries in either of two failing traces, even though both traces showed the subscription cleanly resubscribing — the coach reply never arrived not because the channel failed to reconnect, but because nothing was re-fetching the terminal state once it did. (Root cause: OBJ-3953, fix shipped PR #3324, b33234ff90c6d1ff86c49639f762c8fb6f4a3f97.)

Mobile (iOS/Android)

New iOS bundle ID needs Developer Portal + match setup first

• Adding a new iOS bundle ID (widget extension, App Clip, etc.) needs Apple Developer Portal + fastlane match setup BEFORE opening the PR — mobile-ios.yml's ios-signing-check job ("Verify iOS signing assets") runs automatically on any PR touching ionic_frontend/ios/** or ionic_frontend/fastlane/**, calling the Fastfile's signing_check lane, which runs match(type: "appstore", app_identifier: [...], readonly: true). readonly: true is intentional — it fetches existing certs/profiles but never creates new ones. This gate ships with the WidgetBridge milestone (gsd/v4.1-engagement-retention, PR #1306) and is not yet on master — until that branch merges, a new-bundle-ID PR against master will not hit this check, so treat the steps below as required practice regardless (the gate will catch you once it lands, and the underlying Apple Developer Portal / match-repo requirement is real either way). Before opening a PR that introduces a new bundle identifier, on a machine with Apple Developer Portal + fastlane match credentials:

  1. Register the new App ID on the Apple Developer Portal (plus any App Group / capability the extension needs — register the App Group itself first if it doesn't already exist).
  2. Run non-readonly match to generate the profile(s) and push them to the match repo: cd ionic_frontend && bundle exec fastlane match appstore --app_identifier "<all,bundle,ids,comma,separated>" (list every existing ID alongside the new one).

  Skip either step and, once the gate is live, "Verify iOS signing assets" fails deterministically — it is not a flake, don't re-run the job, it will fail identically until the profile exists in the match repo. Worked example (App Group group.com.objectuve.ionic, App ID com.objectuve.ionic.HomeWidget, added for the WidgetKit extension): see OBJ-1196.

Manual mobile build can race a tag's release-guard correction

• A manually dispatched or tag-triggered mobile build can race a tag's own release-guard correction, shipping a version label paired with a stale bundled CHANGELOG.md — production.yml has always verified the tagged commit's ionic_frontend/CHANGELOG.md contains a ## [<version>] entry before deploying; mobile-android.yml/mobile-ios.yml didn't, until v4.2.0 was cut by hand (skipping scripts/release.sh), production.yml correctly rejected it, and — in the ~30-minute window before the tag was corrected — both mobile workflows were manually dispatched against the bad ref and shipped VITE_APP_VERSION=4.2.0 bundled with a CHANGELOG.md still frozen at 4.1.4. Fix (shipped, PR #1738): both workflows now carry the same changelog-presence guard production.yml does, right after version resolution. General rule: any workflow resolving a version label from a tag/ref name, independent of a content check on what that ref's commit actually contains, needs the same guard — not just for CHANGELOG.md, for any future release-consistency assertion the primary deploy pipeline gains. Known gap: mobile-android-release.yml has the identical exposure (confirmed via its own run history to have hit this same incident) and is not yet guarded. See docs/operations/mobile-builds.md (Troubleshooting) for the full write-up. (Root cause: OBJ-1684, PR #1738.)

Android adaptive launcher icon regressions (broken twice)

• Regenerating the Android adaptive launcher icon has broken production twice, for two different reasons, with zero CI signal either time — v4.4.5's icon rebrand (75b0bf57d) shipped every mipmap-*/ic_launcher_foreground.png at the legacy 48dp size table instead of the adaptive 108dp table, so Android stretched the mark 2.25× and the adaptive mask cropped it — broken for every Android 8+ user, themed icons on or off. The icon has also never had a <monochrome> layer, so Android 13+'s Material You theming has always had to auto-synthesize one from the foreground alone. No .github/workflows/mobile-android*.yml workflow greps for icon/mipmap/launcher/adaptive, so neither class of bug has any CI gate. Fix (shipped, PR #2194, OBJ-2296): replaced the five mis-scaled PNGs with a single scale-safe vector foreground and added the repo's first <monochrome> layer. If you regenerate this icon again: verify the foreground asset against the 108dp table (not 48dp), confirm both mipmap-anydpi-v26/*.xml files carry a <monochrome> layer, and run Android Studio's themed-icon preview before merging. See docs/operations/mobile-builds.md (Troubleshooting → "Android adaptive icon regressions") for the full write-up. (Root cause: OBJ-2296. First occurrence: bfe1dae0a.)

Supporter-adjacent strings need an explicit isNative branch

• Any new supporter-adjacent user-facing string needs an explicit isNative branch decision at write time — two App Store rejections (4.2.0, 4.4.1) trace to strings added without one — Apple Guideline 3.1.1 forbids donation/funding/contribution framing on a native build, and 2.3.7 forbids quoting a price outside StoreKit; a copy string that reads fine on web can silently violate both the moment it ships unbranched to a native build, with no CI signal — Playwright smoke only runs against a browser, where Capacitor.isNativePlatform() is always false, so it structurally cannot catch a native-only copy leak. Pattern: assign Capacitor.isNativePlatform() once to a module-scope const isNative (or reuse a file's existing isNativePlatform), then branch a computed (ctaCopy in OurStory.vue) or an inline ternary at the string's call site — never call isNativePlatform() inline in the template. Native framing is perks + purchase — custom color themes, Supporter profile badge, early access, and (as of v4.67, Streak Insurance) a functional recovery convenience — not "help fund the mission." The prohibition is on donation/funding framing and non-StoreKit pricing, not on a perk being functional — a v4.67 doc pass (OBJ-4026) widened this entry's enumeration after Streak Insurance shipped the first non-cosmetic Supporter perk; nothing shipped was at risk (the merged native footer already made no funding claim), but the enumeration itself had gone stale. Canonical branch examples (OBJ-2182, PR #2116): SettingsHub.vue:463 (Supporter tile hint), OurStory.vue:60-65,134 (timeline entry + hero CTA routed through ctaCopy.button), CheckoutSuccess.vue:16,102-104 (post-purchase confirmation body), PaidConfirmationCard.vue:29 (an optional headline prop — the component itself stays presentational, no Capacitor import; the branch decision is made by its caller, Settings.vue:1010-1011). Scope note on 4.4.1's 2.3.7 citation: on submission 1772712c-42d6-436a-bfbf-80e43765d6e8 (reviewed 2026-08-03, version 4.4.1 build 116) Apple cited 2.3.7 against the App Store screenshots ("The app screenshots include references to the price of the app or the service it provides"), not against any in-app string — that half was remediated in scripts/capture-store-screenshots.mjs + docs/marketing/store-assets/README.md; the in-app native-copy obligation on that same submission is its separate 3.1.1 citation, which is what the isNative branch rule above enforces. Full contract and audited surface inventory: docs/features/supporter.md#native-copy-contract-app-store-311--237.

iOS native deps are SPM-only, not CocoaPods

• pod: command not found in a fresh agent workdir does not mean CocoaPods provisioning is a blocking iOS build gap in this repo — it isn't, and never has been. ionic_frontend/ios/App integrates Capacitor and its native plugins via Swift Package Manager (ios/App/CapApp-SPM/Package.swift, "DO NOT MODIFY — managed by Capacitor CLI commands"); there has never been a Podfile here, and npx cap sync ios never shells out to CocoaPods. OBJ-2507 was opened after an agent hit the missing-pod-on-PATH symptom and reasonably but incorrectly concluded CocoaPods + a committed Podfile.lock were required to unblock a native build — a clean npm ci && npm run build && npx cap sync ios && xcodebuild build succeeds end-to-end with zero CocoaPods involvement, and mobile-ios-boot-smoke.yml proves the same thing on every PR touching ios/**. Fix (shipped, PR #2346): scripts/check-ios-no-cocoapods.mjs, run in .github/workflows/ios-cocoapods-drift-guard.yml on any PR/push touching ionic_frontend/ios/** or the guard script itself, fails CI if a Podfile/Podfile.lock is ever committed under ionic_frontend/ios, or if CapApp-SPM/Package.swift disappears — so this misdiagnosis can't quietly resurface. The actual gap OBJ-2507 surfaced was backend/creds, not tooling — see docs/operations/mobile-builds.md (Release-Build Safety Gates → "iOS device/QA lane") for the staging-dispatch fix that closes it. (Root cause: OBJ-2507.)

npx cap sync ios silently drops the hand-added GoogleSignIn SPM dependency

• ios/App/CapApp-SPM/Package.swift is regenerated wholesale by the Capacitor CLI on every npx cap sync ios, so the GoogleSignIn-iOS dependency added by hand in OBJ-2528 disappears without a word — and the resulting build error names a module cache problem, not the real cause. The file's own header says // DO NOT MODIFY THIS FILE - managed by Capacitor CLI commands; Capacitor rebuilds it from the installed Capacitor plugin list, and GoogleSignIn-iOS is not a Capacitor plugin — it's a raw SPM package added directly to that manifest by c2c71d110 (native Google Sign-In, OBJ-2528). Sync drops both the .package(url: "https://github.com/google/GoogleSignIn-iOS", …) line and the .product(name: "GoogleSignIn", package: "GoogleSignIn-iOS") entry, and the next xcodebuild fails with AppDelegate.swift:3:8: error: unable to resolve module dependency: 'GoogleSignIn'. That message reads like a stale DerivedData / SPM-cache problem, and it is not — wiping DerivedData, resolving packages, or rebuilding changes nothing, because the dependency genuinely is no longer declared. This misdiagnosis has already cost one full verification cycle during OBJ-2602's device testing, where the failure was written up as environmental before the manifest diff was checked. Fix: after any npx cap sync ios, restore both files from git before building — git checkout -- ionic_frontend/ios/App/CapApp-SPM/Package.swift ionic_frontend/ios/App/App.xcodeproj/project.xcworkspace/xcshareddata/swiftpm/Package.resolved — and confirm with grep -c GoogleSignIn ionic_frontend/ios/App/CapApp-SPM/Package.swift (expect 2). General rule: when a build error names a module that a generated manifest is supposed to declare, diff the manifest against git before touching caches — a regenerated file that dropped a hand-added entry produces the same symptom as a cache miss, and only one of the two is real. (Surfaced during OBJ-2602 device testing; dependency introduced by OBJ-2528, c2c71d110.)

An empty xcconfig *_DEFAULT with an unwired *_OVERRIDE ships silently broken to every real build

• An xcconfig build setting shaped FOO = $(FOO_OVERRIDE:default=$(FOO_DEFAULT)) degrades to whatever FOO_DEFAULT is the moment nobody sets FOO_OVERRIDE — and if that default is itself empty, every real build ships with FOO silently empty, with no error anywhere in the pipeline. ionic_frontend/ios/google-signin.xcconfig declares GOOGLE_IOS_CLIENT_ID_DEFAULT = (blank) and resolves GOOGLE_IOS_CLIENT_ID from GOOGLE_IOS_CLIENT_ID_OVERRIDE, same for GOOGLE_IOS_REVERSED_CLIENT_ID. Nothing in .github/workflows/mobile-ios.yml, ionic_frontend/fastlane/Fastfile, or any script ever set either override, so Info.plist's GIDClientID resolved to an empty string in every build the pipeline produced, including every TestFlight/App Store upload — native Google Sign-In was 100% broken in production from 2026-08-16 (Sentry OBJECTUVE-CLIENT-C2) until this was caught. Why it stayed invisible so long: no crash, no CI failure, no log line. GoogleSignInPlugin.swift's load() checks !clientId.isEmpty and no-ops rather than configuring GIDSignIn (guard let clientId = ... as? String, !clientId.isEmpty else { return }) — deliberately, so the app doesn't crash at launch on a missing credential — and signIn() then rejects the call cleanly with "Google Sign-In has no client ID configured." Both are correct code behaving exactly as designed for a missing credential; the bug was entirely upstream, in the pipeline never setting the override. The contrast that makes the general pattern visible: the sibling ionic_frontend/ios/graphql.xcconfig uses the identical override mechanism (GRAPHQL_BASE_URL = $(GRAPHQL_BASE_URL_OVERRIDE:default=$(GRAPHQL_BASE_URL_PRODUCTION_DEFAULT))) but gives its default a real value (https://api.objectuve.com/graphql) — so an unwired override there just degrades to the production API, not to nothing. An *_OVERRIDE mechanism with an empty *_DEFAULT has zero safety margin: the override is the only value, and if the pipeline that's supposed to set it doesn't, the build silently ships a config-shaped hole instead of failing loud. Fix (shipped, OBJ-2772, PR branch fix/obj-2772-ios-google-client-id-wiring): mobile-ios.yml now wires GOOGLE_IOS_CLIENT_ID/GOOGLE_IOS_REVERSED_CLIENT_ID secrets into both overrides for the build-ios job, Fastfile's google_signin_xcargs passes them through build_app's xcargs explicitly, and scripts/check-ios-google-client-id.mjs adds two gates: a PR-time structural check (.github/workflows/ios-google-client-id-guard.yml) that the wiring exists at all — checkable with no secret — and a dispatch-time check that reads the resolved xcodebuild -showBuildSettings value (not just the env var) and fails before archive/upload on a testflight/appstore dispatch if it's empty, naming both secrets in the error. google-signin.xcconfig itself is untouched — the override mechanism was already correct; only the wiring was missing. General rule: when you add an *_OVERRIDE-over-*_DEFAULT xcconfig var (or the equivalent pattern in any build-config language), either give the default a real, safe fallback value the way graphql.xcconfig does, or add a CI gate asserting the resolved value is non-empty before any real (TestFlight/App Store/production) build — an unwired override with an empty default is a silent, load-bearing config gap that no amount of "it built successfully" catches. Same failure shape as validate-secrets.sh never runs on PRs — only at deploy time above, one layer further out: that gotcha is a missing-secret check that only runs at deploy time; this one is a missing-secret check that, before this fix, didn't exist at any point in the pipeline at all. (Root cause: credential wiring never landed as Task 0 of OBJ-2528. Fix: OBJ-2772.)

A snapshot mutated natively must clear or recompute every field derived from what it changed

• A widget's optimistic native-side mutation of one snapshot field can silently invalidate every other field that's derived from it, unless the mutation clears or recomputes them together — the home widget's interactive check-off writes an optimistic local update straight from the widget process (Android's WidgetSnapshot.withOptimisticCheckin, iOS's WidgetSnapshot.checkingIn), flipping a habit's checkedInToday ahead of the next real JS sync so the tap redraws immediately. coachLine is computed client-side (resolveCoachLineState, useWidgetSnapshot.ts) as a pure function of the due-habit list's checked-in count — so a native transform that flips checkedInToday but carries coachLine through verbatim leaves a derived field asserting a fact its own source of truth just changed. OBJ-2428's reported screenshot is the failure mode made visible: the not-yet-new-day Coach line ("Add today's stone…") rendered next to a 5-of-5 ALL_DONE card and five checked rows — truncated and wrong, because the optimistic transform updated every habit's checkedInToday but left coachLine frozen at whatever JS had last synced. This class of bug is easy to miss in review because the mutation itself is correct in isolation — checkedInToday really did flip, on time, with the right value — the defect is entirely in what the transform didn't touch. Fix (OBJ-2436, both platforms): clear the derived field instead of recomputing it natively. withOptimisticCheckin now calls obj.remove("coachLine") in the same JSON edit that flips checkedInToday; checkingIn now constructs the updated snapshot with coachLine: nil. Recomputing coachLine natively (porting resolveCoachLineState's four-branch template selection into Kotlin/Swift) was considered and rejected — it would duplicate a JS-authored string table into two native codebases, and the phase-3 UI-SPEC already documents that coachLine's templates can change in a web release without a matching native release, so a native copy would drift silently. Clearing respects the existing "JS owns this string" boundary: Android's showsCoachLine gates the whole Coach row on coachLine != null, so clearing it hides the row until the next real sync; iOS's kindlingCoachLine/largeCoachLine fall back to reactive.captionText, a value computed at render time from extendedToday/state, never the stale string. The general rule: when a native transform mutates a snapshot field optimistically, audit every other field in that snapshot for a client-side derivation keyed on the field you just changed — carrying a derived field through unmodified is the default, and it's silently wrong the moment the source value it depends on moves out from under it. Full write-up: docs/features/home-widgets.md#coach-line-ownership--widget-native-mutation-contract-obj-2428obj-2436.

CODE_SIGNING_ALLOWED=NO produces errSecMissingEntitlement (-34018) on any Keychain-touching flow

• A completely unsigned iOS test binary has no entitlements for Keychain Services to check at all — any call through a keychain-access-group entitlement throws errSecMissingEntitlement (-34018), and a generic catch-all UI error can make that indistinguishable from an unrelated auth failure. sign-in-ui-test (OBJ-2508's XCUITest suite) initially mirrored boot-smoke's own CODE_SIGNING_ALLOWED=NO signing setting. That's fine for boot-smoke, which never touches Keychain — but sign-in-ui-test drives a real password sign-in through useClerkNative.ts's secureStorage.setClientToken() (@aparajita/capacitor-secure-storage, backed by the keychain-access-group App.entitlements declares). A fully unsigned binary has no entitlements at all, so that call threw -34018 on every run. SignIn.vue's generic passwordLogin() catch-all collapsed that native exception into the same "Can't reach the sign-in service" text a genuine Clerk token failure would produce — so the symptom alone couldn't distinguish "Clerk is broken" from "this binary can't touch Keychain," and a source-only diagnosis (no live device access in that runtime) initially misread it as a Clerk native client-token-extraction bug before a live re-run traced it to the real mechanism. Fix (OBJ-2508, PR #2404): switched sign-in-ui-test's signing from CODE_SIGNING_ALLOWED=NO to ad-hoc "Sign to Run Locally" (CODE_SIGN_IDENTITY=-, CODE_SIGN_STYLE=Manual, empty DEVELOPMENT_TEAM, GENERATE_INFOPLIST_FILE=YES — codesign refuses to run without an Info.plist) — no Apple Developer account or certificate needed, verified against a runner with zero codesigning identities present. boot-smoke's own CODE_SIGNING_ALLOWED=NO is untouched; it never touches Keychain, so it was never broken. General rule: CODE_SIGNING_ALLOWED=NO is safe only for a build that never exercises a Keychain-backed entitlement — the moment a UI test flow touches secure storage, sign it (even ad-hoc) or expect -34018. And when a UI test surfaces a generic error string, don't trust that the message names the real failure — a catch-all can flatten two unrelated causes into identical user-facing text; check the underlying native exception before diagnosing from the symptom alone. Full mechanism write-up: ionic_frontend/ios/App/AppUITests/SignInUITests.swift's file header comment; signing config: .github/workflows/mobile-ios-boot-smoke.yml's sign-in-ui-test job. (Root cause + fix: OBJ-2508, PR #2404.)

WKWebView visualViewport.scale drifts on iOS resume — fixed-position chrome goes blank, and the repair is a scroll nudge, not a repaint

• After a background → foreground cycle on iOS, window.visualViewport.scale can come back as 1.6875 (= 810/480 on the test iPad) instead of 1, and position: fixed chrome (the collapsed .side-nav rail) is laid out against the stale scale and paints nothing — while every DOM, computed-style, layout-rect, hit-test and app-state reading stays byte-identical to a healthy resume. OBJ-2774 burned three speculative fixes (a paint-property toggle in PR #2523, a display:none forced reflow in PR #2549, and a v-if remount + backdrop-filter removal) because each assumed a compositor backing-store drop and each was falsified on device. A geometry/hit-test overlay isolated the only differing reading — visualViewport.scale — and three candidate repairs were tried live: re-applying the viewport meta did nothing, a window.scrollTo(0, 1); window.scrollTo(0, 0) nudge repainted instantly (the same thing rotation does: it re-runs WKWebView's viewport configuration). Fix: ionic_frontend/src/composables/useResumeRepaint.ts runs that nudge on every native Capacitor resume (two rAF after the document is visible, again at 500 ms), no-op on web; verified across repeated ≥10 s backgrounds on the iPad with the rail collapsed. General rule: when fixed chrome vanishes on iOS resume and the DOM/styles/rects all look right, read visualViewport.scale before theorizing about compositor layers — and expect the repair to be a viewport nudge (scroll or rotation), not a remount or paint-property toggle. (Root cause + fix: OBJ-2774.)

Branches, Releases & Planning

Tags must point to commits on master

• Tags must point to commits on master — production.yml validates that the tagged commit is an ancestor of origin/master before deploying. Tags on pre-squash feature-branch tips (which have no Docker image) fail validation within seconds with a clear error message. Always use scripts/release.sh to cut releases — it guarantees the tag lands on a master commit. Manual git tag <sha> on a feature-branch tip will be rejected. See Production deploy safeguards for the full error message and recovery path.

A clean merge isn't proof a branch is scoped correctly

• PR-scope gotcha: a "clean merge" is not proof a branch is scoped to its intended change — conflict-scoped diffing can't see non-conflicting additions — PR #1790 (phase-theme-creator-custom-colors) squash-merged to master (7910d9a95) carrying the entire, still-unmerged gsd/v4.14-activity-nexus-connected-apps tree (206 files changed vs. its parent, 144 present only on that unmerged branch) alongside its intended ~29-file Theme Creator diff, hard-failing staging.yml's deploy-lint gate on unwired STRAVA_* env vars and forcing an emergency revert (b4ead33e7, PR #1795) that broke staging for the whole team. Root-cause investigation found the contamination was already present at the branch's very first commit (801a22f57) — not because that commit's own git add swept in stray files (git diff --stat 87e326edb0 801a22f57 shows only the 2 intended planning docs), but because its parent (87e326edb0) was itself several commits into the unmerged Activity Nexus integration line, not master — i.e., the branch was cut from the wrong ref entirely, roughly 20 hours before the multi-round "master-drift" conflict-resolution saga the crew spent hours diagnosing even began; that saga was real work against an already-wrong base, not the source of the contamination. Every downstream step (three conflict rounds, a manual rebase, a UI-spec branch cut from the poisoned commit, two rounds of code review) inherited it undetected, because git merge-tree/textual conflict resolution only surfaces textual overlaps — a file that exists cleanly on one side with no corresponding change on the other produces no conflict at all and merges silently. Guardrail (shipped): .github/workflows/pr-scope-guard.yml + scripts/check-pr-scope.mjs run on every PR open/synchronize with two independent checks — a required, non-bypassable cross-branch-overlap hard-fail (the PR's changed files must not intersect with files added on any unmerged long-lived gsd/* integration branch, dynamically discovered) and a soft-fail file-count outlier check (>75 changed files, bypassable via the oversized-pr-ok label); both checks are required in branch-protection ruleset 17927598, so a red guard actually blocks merge. scripts/check-pr-scope.test.mjs replays the real incident straight from this repo's own git history (poisoned commit 7910d9a95 fails; the clean re-ship f86e371db/#1796 passes). General rule: before cutting a new phase branch, verify a clean git status --porcelain and branch off a freshly-fetched origin/master — never reuse a workdir that may still hold another branch's staged/on-disk files. See CONTRIBUTING.md § Branch Creation Hygiene for the checklist and Branch Hygiene: Poisoned First-Commit — OBJ-1766 Post-mortem for the full write-up. (Root cause: OBJ-1766. Guard: OBJ-1767, PR #1797.) Addendum (OBJ-2649) — the guard's own self-test depended on the same branch staying alive, and broke twice. scripts/check-pr-scope.test.mjs's POISON_BRANCH fixture required gsd/v4.14-activity-nexus-connected-apps — the very branch this gotcha's incident concerns — to stay a live ref on origin forever: three assertions resolved the branch by name, backed by git branch -r --merged, which only resolves refs/remotes/. That constraint was documented only in a source comment at check-pr-scope.test.mjs:38-47 — nothing in this doc, CONTRIBUTING.md's Branch Creation Hygiene, or the milestone-close ritual flagged it, so every signal told a human doing routine gsd/* cleanup it was safe to delete. It was deleted twice (2026-08-02, caught within 20 minutes by luck; 2026-08-17, not caught until an unrelated PR exercised the full required-check suite), blocking every PR on master both times. Fixed: gatherUnmergedIntegrationBranches gained an injectable ref namespace (defaulting to today's refs/remotes/ behavior for production callers), and its merged-branch lookup switched from git branch -r --merged (namespace-locked) to git for-each-ref --merged=<base> (namespace-agnostic); check-pr-scope.test.mjs now creates its own SHA-pinned fixture ref under a private refs/pr-scope-fixtures/ namespace via git update-ref, torn down in before/after hooks plus a self-healing pre-sweep. gsd/v4.14-activity-nexus-connected-apps is no longer load-bearing for CI and needs no branch-protection rule — a future gsd/* cleanup pass can delete it (once its milestone is otherwise resolved) with no guard-side consequence. General rule: a CI fixture must never depend on mutable repo state — a live branch, tag, or remote ref — surviving. Pin a SHA, or create the ref inside the test's own setup and tear it down after. In a Multica agent workdir specifically, a fixture ref must never be written under refs/remotes/: that namespace lives in the shared-mirror common git dir and is visible to every other agent checkout on the host, so a leaked ref (skipped teardown, crashed process) would sit in the exact namespace gatherUnmergedIntegrationBranches's production callers glob, producing false-positive overlap violations on unrelated PRs. (Fix: OBJ-2649, PR #2409.)

A release must actually come from the standard release path

• Release-integrity gotcha: a v* tag or a master commit must actually come from the standard release path, or nothing enforces it — v4.0.3 (tag c8d9186716857095fb892c114b8281648077f043) was committed, tagged, and pushed directly to master under a leftover git identity, bypassing scripts/release.sh and scheduled-release.yml entirely; at the time, production.yml's changelog check was advisory-only (a bare ::warning:: that never blocked a deploy) and no attribution/diff-shape gate existed at all. Guardrail (shipped): production.yml's changelog-presence step now exit 1s instead of warning, and .github/workflows/release-integrity-guard.yml + scripts/check-release-integrity.mjs gate three independent failure modes — every commit landing on master must be either the release-bot@objectuve.com identity or have at least one merged PR associated with it (commit-attribution), a v* tag's pointed-to commit must match scripts/release.sh's exact deterministic output: only ionic_frontend/CHANGELOG.md + ionic_frontend/package.json touched, commit message chore(release): update CHANGELOG for vX.Y.Z, tag message Release vX.Y.Z (tag-release-shape), and a merged commit's Co-authored-by: trailer for a named crew agent must match that agent's canonical identity in scripts/crew-identity-map.json (evaluateCoAuthorAttribution, added OBJ-2930 — see addendum below). scripts/check-release-integrity.test.mjs replays the real v4.0.3 tag/commit straight from this repo's own git history and asserts the tag/commit-attribution checks still flag it red, and a clean release (v4.1.0) passes both green. (Root cause: OBJ-1473, comment b0421b69. Guard: OBJ-1475, PR #1558.) Addendum (OBJ-2930) — a third failure mode, one layer up from the tag/commit shape above: a crew agent's own commit identity can drift between merges, and nothing verified it. Ambient git-identity inheritance on the shared Multica runtime host let Orion's and Dori's bare, un-pinned git commit calls (no inline -c user.name=/-c user.email= override) pick up whatever identity a prior, unrelated session last left configured, producing three different email domains for the literal same agent across three consecutive merges — orion@objectuve.dev (42eddd11b, #2634), orion@objectuve.com (4dba41345, #2636), orion@objectuve.ai (26c2508b2, #2638). GitHub's require_extra_approval_for_unattributed_changes ruleset couldn't resolve the drifting identity to a known reviewer, and Roy's own approval lives in the Multica comment thread rather than GitHub's native review UI, so the rule's extra-approval requirement was never satisfied either way — the two gaps stacked into a routine --admin merge bypass on all three PRs (#2600, #2622, #2638). Fixed at the source: Orion's, Dori's, and Riley's git-mutating instruction steps now carry inline -c identity overrides matching scripts/crew-identity-map.json, closing the ambient-inheritance path itself. [Correction, added with the OBJ-3610 addendum below: this claim didn't hold against docs/guides/multica-agent-crew.md's actual text. A direct grep of that doc during OBJ-3610 found zero -c user.name=/-c user.email= overrides on any of Orion's, Dori's, or Riley's git-mutating steps — not a partial miss, none. Cross-checking Dori's own live Multica platform instructions (a separately-maintained artifact from this reference doc) found the override present on exactly one of her two commit-producing steps (the milestone-close final action) and absent from the other (the routine "commit onto Codi's branch" step) — so even where the fix landed, it landed incompletely, and it never reached this doc at all for any of the three agents. Whether that's regression or the fix simply never got mirrored into this doc when it shipped is not established either way; what's confirmed is the doc's current state as of 2026-09-10. OBJ-3610 added the missing overrides to this doc for Orion, Dori (both steps), and Riley, using each agent's scripts/crew-identity-map.json identity — see that addendum for specifics.]** Fixed as a check: evaluateCoAuthorAttribution (scripts/check-release-integrity.mjs), wired into the same runCommitCheck job as commit-attribution above, validates every merged commit's Co-authored-by: trailer for a named crew agent against the identity map; a name absent from the map (Tess, Roy, Vicki as of this addendum — mixed or insufficient sample evidence) is not evaluated, not passed. scripts/check-release-integrity.test.mjs replays the same three real historical commits (.dev fails, .com passes, .ai fails, non-crew co-authors like Claude/multica-agent correctly ignored). Not yet fixed: the shared joshlock3 GitHub push login every crew agent uses (docs/operations/multica-github-auth.md, "Fix #1") is the identical defect class one layer up the stack — untouched by this addendum — and the policy call on linking crew identities to real GitHub accounts vs. rescoping the ruleset to trust Multica-thread approvals remains open. (Diagnosis: OBJ-2930, comment eaeedbef. Guard: OBJ-2930, PR #2661.) Addendum (OBJ-3299) — a second, independently-discovered instance of the same unpinned-instruction root cause, this time landing as a full identity impersonation rather than a domain drift. Tess's branch agent/tess/2dc700944aaf (commit 5a0b08154, PR #2842, a .github/workflows/ fix that was entirely her own work) carried git author/committer identity Codi <codi@objectuve.com> end to end — not a drifted Tess variant at all, but a fully inherited Codi identity — and the squash-merge (36c533daea55f6c12faaee064f903cbbe3f2abf9) preserved a Co-authored-by: Codi <codi@objectuve.com> trailer that evaluateCoAuthorAttribution correctly flagged red on the next push-to-master scan. Root cause matched the original OBJ-2930 mechanism exactly, just never closed for Tess: her live agent instructions had zero literal git commands anywhere — not even Riley's/Dori's post-OBJ-2930 pattern of an inline -c user.name=/-c user.email= override embedded at the point of commit — so every commit she made ran under whatever git config user.* the shared runtime host last happened to have configured. git log --all for any "Tess" identity showed the predicted drift wasn't a one-off: four distinct email domains in active use over two months — tess@objectuve.ai (73 author / 80 committer commits, 2026-07-09 → 2026-09-02, the dominant/intended identity), tess@multica.objectuve.com (26 commits, clustered 2026-08-31), tess@objectuve.com (4 commits, scattered 2026-08-18 → 2026-09-02, recurring as recently as this incident's own day), and tess@objectuve-agents.local (1 commit, 2026-08-19) — with this incident's full Codi impersonation as a fifth, more severe variant of the same gap. Fixed at the source: Tess's "Core behavior" bullet in both her live agent instructions and the source doc docs/guides/multica-agent-crew.md now embeds git -c user.name="Tess" -c user.email="tess@objectuve.ai" commit -m "...", placed in the recurring behavior bullet rather than a single final-action block — unlike Riley/Dori, she produces commits throughout a run, not at one fixed step. Deliberately not fixed as a check yet: scripts/crew-identity-map.json still excludes Tess — the map's stated bar (inline override present, or 50+ real commits with zero drift) wasn't met at fix time on either count, and adding her before the instruction fix proves out over a clean run or two would just start flagging her own historical drift noise instead of catching future regressions. Also closed as part of this incident: scripts/signal-surfaces.mjs gained a release_integrity entry (gate category E) covering release-integrity-guard.yml, check-release-integrity.mjs, check-release-integrity.test.mjs, and crew-identity-map.json — until this fix, the monthly signal-instrumentation sweep never exercised this guard's own health, so a future silent regression in the guard itself (e.g. a reintroduced duplicate env: key) would have gone uncaught except by another reactive incident review. (Diagnosis: OBJ-3299, Dave. Fix: OBJ-3299, PR #2866.) Addendum (OBJ-3610) — the one committing crew role the OBJ-2930 fix set never reached, plus a stale claim in this very addendum. Codi's two "commit your work" instruction steps in docs/guides/multica-agent-crew.md (the branch-Orion-named step and the Branch/PR-boundary step) had never received an inline -c user.name=/-c user.email= override — not a regression, the original OBJ-2930 fix set covered Orion, Dori, and Riley by name and simply never included Codi, the crew's only other committing role. Ambient identity on the shared Multica runtime host picked up a leftover Riley <289265804+objectuve-multica-bot@users.noreply.github.com> value (plausibly left by a prior Riley invocation on the same host — Riley has no committing role of her own, so this identity had nowhere legitimate to be consumed) and landed on all 3 of Codi's commits on agent/codi/9f2a102f86d2; the squash-merge for PR #3057 correctly carried it forward as a Co-authored-by: Riley trailer, which evaluateCoAuthorAttribution correctly flagged red against Riley's real mapped identity (riley@objectuve.ai) on the next push-to-master scan — the guard worked, the identity feeding it was wrong at the source. Separately, verifying the fix while diagnosing this found the OBJ-2930 addendum's "Fixed at the source" claim about Orion/Dori/Riley didn't hold against this doc's actual text either (see the correction inline in that addendum, above) — a grep found zero inline overrides for any of the three. Fixed at the source: Codi's two steps now carry git -c user.name="Codi" -c user.email="codi@objectuve.ai" commit -m "…", matching her existing scripts/crew-identity-map.json entry; Orion's milestone-kickoff git commit step now carries -c user.name="Orion" -c user.email="orion@objectuve.com"; Dori's two commit-producing steps (the routine "commit onto Codi's branch" step and the milestone-close final action) now carry -c user.name="Dori" -c user.email="dori@objectuve.com"; and Riley's mechanical-conflict git merge origin/master step now carries -c user.name="Riley" -c user.email="riley@objectuve.ai". Not fixed as a check: the regression-guard Dave proposed — a conformance test asserting every name in scripts/crew-identity-map.json appears in docs/guides/multica-agent-crew.md paired with an inline override near that agent's own commit/merge step, which would have caught the Codi gap before it ever reached a real commit — is a code-adjacent test file, not a docs change, and stays with Codi. Likely one shared mechanism across this whole family, confirmed later (OBJ-3657): the domain/identity drift here and in the OBJ-2930/OBJ-3299 addenda above reads as "whatever a prior session last left configured" — per-session ambient state — but is very likely each a different stray write landing in the same shared file: see the shared-mirror pollution gotcha's OBJ-3657 addendum, which confirmed the mechanism for a stray [user] block (not this incident's specific leftover-Riley-identity value, which predates that confirmation and was never independently re-diagnosed). (Diagnosis: OBJ-3610, Dave. Docs fix: OBJ-3610, Dori.)

A milestone can finish every phase and never reach master

• Stranded-milestone-branch gotcha: a milestone can complete every phase, GA its feature flag, close its tickets, and still never reach master — nothing in the pipeline checks — A GSD milestone ships by squash-merging each phase PR into its gsd/vX.Y-<slug> integration branch. Promoting that branch to master is a separate, hand-created "integration merge to master" issue (v4.11 = OBJ-1622 → PR #1687) with no automation and no owner. When milestone throughput outran that manual step, branches finished every phase and simply sat. gsd/v4.15-dashboard-hierarchy completed all 5 phases on 2026-07-26 and was never promoted — while its dashboard_hierarchy PostHog flag had already GA'd to 100% and had then been retired from code by a later phase PR on the same branch. So PostHog read "on, 100%", the milestone read shipped, every ticket was closed, master had zero consuming code, and no check anywhere reported a problem: it was found only when a human asked why a shipped feature wasn't visible in the app. Six other branches were stranded identically (v4.14, v4.13, v4.19, v4.3, plus v2.1-frontend-view-decomposition at 61 orphaned files and v1.18-pre-launch-readiness at 52). The failure then compounded: once pr-scope-guard landed (2026-07-25, gotcha above), every stranded branch became a cross-branch-overlap blocker for every other branch trying to promote. Guardrail (shipped): .github/workflows/branch-promotion-guard.yml + scripts/check-branch-promotion.mjs run daily (14:00 UTC), file/update a single deduplicated tracking issue, and exit non-zero so staleness stays red in the Actions tab. It is deliberately a scheduled nag, not a required PR check — making it required would wedge CI on day one, since several branches are legitimately parked pending external work (v4.14 needs Strava API secrets). The detection signal is content presence, and getting there required rejecting three plausible alternatives, each measured against this repo's real history: ancestry (git branch --merged) is useless — every phase PR is squash-merged, so a fully-promoted branch is never an ancestor of master; "has no PR to master" over-reports — gsd/v4.5-runtime-least-privilege has no PR from its head yet all 302 of its created files are in master (it promoted via individual phase PRs); and a naive git diff --diff-filter=A master branch reports every file master deleted after the fork as a branch "addition" — 20 phantom additions for v4.5 alone. What works is files the branch itself created relative to its own merge-base with master that are still absent from master's tree — v4.5 → 0, v4.14 → 126. Only two prefixes are excluded, and both narrowly: .planning/phases/ (per-phase PLAN/UI-SPEC/mockup scratch) and .claude/worktrees/. Do not widen those to .planning/ or .claude/ wholesale — master tracks 250+ files under .claude/ (skills, agents, hooks) and 146 under .planning/quick/, so a blanket exclusion silently suppresses a milestone that ships a new skill, and the promoted-branch control proves milestone docs do travel with a promotion (v4.5 created .planning/milestones/v4.5-runtime-least-privilege-ROADMAP.md and still exonerates at 0). scripts/check-branch-promotion.test.mjs replays all of that from this repo's own git history at pinned SHAs, including the 20-vs-0 gap, so nobody "simplifies" the merge-base call back out. Second guardrail (shipped), catching the same incident from the PostHog side: scripts/sync-posthog-feature-flags.mjs --reverse-drift inverts the existing posthog-flag-drift PR gate — instead of "registered in code, missing from PostHog" it reports any flag live in PostHog (active, rollout > 0) whose key is absent from the feature-flag registry as it exists on origin/master, which is precisely the state dashboard_hierarchy sat in for four days. .github/workflows/posthog-flag-reverse-drift.yml runs it weekly (Mondays 14:30 UTC) as another nag-not-gate — on the day it was written it reported five flags, three of them legitimate retirements awaiting manual archival, so as a required check it would have wedged master's CI immediately. Registry membership, not a text search, is the presence test: git grep dashboard_hierarchy origin/master matches three things that are not consumption — the historical retirement note in featureFlags.ts's own header comment, the unrelated longer key dashboard_hierarchy_migrated_v1 (a localStorage migration marker in useFocusMode.ts), and the CHANGELOG/docs prose describing the retirement — so a grep-based check flips to "present" the moment the milestone lands and falsely exonerates the flag; the registry is the typed single source of truth and has none of those failure modes. scripts/sync-posthog-feature-flags.test.mjs pins both master SHAs (b611654e7 during the strand, acb9ca09d after promotion) and asserts the grep-vs-registry divergence directly. General rule: a milestone is not shipped when its phases are merged — it is shipped when master contains its code. Before marking a milestone ✅ shipped in .planning/ROADMAP.md, verify master containment; see docs/operations/planning-canon.md for the close ritual, which now requires it. (Root cause: found 2026-07-30 investigating why the GA'd dashboard_hierarchy flag had no visible feature; v4.15 promotion = PR #1891, review tracked as OBJ-1909.) Addendum (OBJ-2039): the separate milestone-close guard that runs per-PR (scripts/check-milestone-close-branch.mjs, .github/workflows/milestone-close-branch-guard.yml — not the daily-nag check-branch-promotion.mjs above) had its own declared-branch parser fixed. extractDeclaredBranch() is now anchored to the line that actually declares a milestone's branch (**Branch:** / Branch: / Delivery model: / an "integration branch" phrase, accepting bare or backticked tokens), instead of grabbing the first backtick-quoted gsd/... token anywhere in the document — so a branch merely mentioned in a decision log (e.g. one milestone's rescope note citing a different, already-closed milestone's branch) no longer resolves as this ROADMAP's own. The guard also now enforces the branch-landed check only when the milestone is being closed by the current PR (done at head, not at base) or its declared branch changed between base and head — a PR that merely touches an already-closed milestone's ROADMAP, including one whose branch was deleted from origin after a healthy promotion (the common case), is no longer re-adjudicated and no longer needs a milestone-close-superseded-ok override to pass. Addendum (OBJ-2161): "verify master containment" only closes half of what "shipped" requires — the other half is that .planning/ROADMAP.md's ledger actually says so. OBJ-2150's audit (.planning/audits/milestone-ledger-audit-2026-08-04.md) found 14 milestones that had shipped without ever getting a ledger row, and one (v1.19-critical-path-full-product) whose row asserted the opposite of what master contains — the same "reads shipped everywhere, isn't" shape as the v4.15 strand above, just on the ledger side instead of the branch side. scripts/check-planning-canon.mjs's rule 3 (OBJ-2161) now gates this: every .planning/milestones/v<X.Y>-*-ROADMAP.md file on the tree must have a matching [milestones/<filename>](milestones/<filename>) link inside .planning/ROADMAP.md's ## Milestones section, checked on every PR and push to master — not just at close, since the file's mere existence on the tree is the obligation to index it. See docs/operations/planning-canon.md for the close-ritual requirement. Addendum (OBJ-2160) — a flagged branch is not automatically dead work; read it before ruling. The daily nag's own detection method is content-blind to intent — it can tell you a branch has orphaned files, not whether that content is a finished milestone worth promoting or genuinely abandoned scratch. The four branches this gotcha's root-cause paragraph lists as "stranded identically" (v4.14, v4.13, v4.19, v4.3, v2.1-frontend-view-decomposition, v1.18-pre-launch-readiness) turned out to need three different resolutions once someone actually read them (OBJ-2150 → OBJ-2160, 2026-08-04): PROMOTE gsd/v4.3-keystone-arch-sweep (docs-only, 9 phases complete, zero app-code changes) via PR #2110, merge commit ed4bdd1bb; DELETE its zero-unique-content sibling gsd/v4.3-keystone-arch-sweep-phase-1-goal-tracking, archived as archive/gsd-v4.3-keystone-arch-sweep-phase-1-goal-tracking-superseded-by-2110; ABANDON+tag gsd/v1.18-pre-launch-readiness (incomplete milestone, superseded by master's AnalyticsView) as shelved/v1.18-pre-launch-readiness @ eb36ce168; and confirm-tag gsd/v2.1-frontend-view-decomposition as shelved/v2.1-frontend-view-decomposition @ bf81b8a8e — that one had already been Josh-shelved 2026-05-31 (OBJ-882), so the guard flagging it was a false positive on a decision that predated the guard itself, not a new loss. Full record + precedent write-up: .planning/audits/milestone-ledger-audit-2026-08-04.md's "Stranded branches" section. Addendum (OBJ-2167) — the guard now honors a recorded adjudication instead of only deletion. OBJ-2160's ABANDON/PROMOTE/DELETE resolutions above are all one-time, hand-driven interventions on a branch the guard had already flagged; nothing recorded the decision anywhere the guard itself could read, so a branch legitimately parked on external work (e.g. v4.14's Strava secrets) or preserved as a permanent reference had no way to stop re-flagging short of deletion — which throws away exactly the branches meant to be kept. scripts/check-branch-promotion.mjs now reads a parked/<slug> (temporary, expires after PARKED_MAX_DAYS — default 90 — days) or shelved/<slug> (permanent) git tag at a branch's exact tip (not a reachable ancestor — a new commit moves the tip off the tag and the branch re-enters the report automatically) via gatherSuppressedTips, honored by check/report/body. contains — the milestone-close gate above — is deliberately untouched and stays byte-identical regardless of any tag, so a park/shelve decision can never let a milestone close claim master containment it doesn't have. See Parking a flagged branch for the two-command recipe.

A forbidden command fixed in two docs recurred in a third

• Fixing a forbidden command in two docs doesn't fix it in a third — milestone-close direct-pushed straight to master for two milestones running — OBJ-1421 (2026-07-13) corrected docs/operations/planning-canon.md and .claude/skills/generating-project-roadmaps/SKILL.md to stop recommending /gsd-complete-milestone in this repo (it archives .planning/ROADMAP.md as if it were a live phase list, which it never is here), but missed a third doc: docs/guides/multica-agent-crew.md's Dori "Final action — milestone closed" step still literally instructed /gsd-complete-milestone — the exact command the other two docs forbade. That command's upstream implementation commits directly against whatever branch is checked out with no PR step, and at milestone-close time in this repo that branch is master (the integration branch is already squash-merged by then). This silently landed two direct-push commits on master with no PR — 7eba9fdfb (Dori, v4.25 milestone-close) and a7ec26b87 (Orion, v4.24 milestone-close) — both undetected until release-integrity-guard.yml's commit-attribution check finally collided with a release attempt (OBJ-1911) weeks later; the guard is push-triggered + daily-cron, so it can only report after a bad commit has already landed, and there was no PR for either commit to gate on before merge. Fix (shipped): docs/guides/multica-agent-crew.md now matches the canon everywhere the forbidden command was quoted, and both that doc and the generating-project-roadmaps skill add an explicit branch step for this specific action — milestone-close has no pre-existing open branch to attach to (unlike every other Dori action, which piggybacks on Codi's branch), so it now cuts a short-lived branch off master and hands off to Riley for a normal PR. General rule: when a forbidden command gets corrected under one issue, grep the whole doc tree for the literal string being forbidden, not just the doc(s) that prompted the fix — a crew-brief runbook that disagrees with the "canonical" doc wins in practice, silently, until something downstream surfaces the collision. See Milestone-Close Direct-Push — OBJ-1911 Post-mortem for the full write-up. (Root cause: OBJ-1421 (incomplete fix), surfaced by OBJ-1911. Fix: OBJ-1912.)

A promotion PR must open from the branch's own ref, not a rebase

• A promotion PR must open from the stranded branch's own ref, never a rebased copy — pr-scope-guard's overlap check excludes a PR's source branch by tip sha, not by name — When promoting a stranded gsd/* integration branch to master (see the stranded-milestone-branch gotcha above), the instinct is to cut a fresh, rebased branch (e.g. promote/...) to get a cleaner diff. Don't: pr-scope-guard's required, non-bypassable overlap check discovers unmerged gsd/* branches and flags any PR whose changed files intersect with them, and it recognizes "this PR is one of those branches" only by matching the PR head's tip sha against the branch's tip sha — a same-content-different-name copy doesn't match, so the original gsd/* ref is still counted as a separate unmerged branch and every file it created flags as a violation against the PR trying to land it. Measured on the gsd/v4.19-teams-v1-wiring promotion (OBJ-2003): opening from a rebased copy would have flagged all 15 non-.planning/ files the branch created as cross-branch-overlap violations — a guard that exists to catch exactly this failure mode instead blocking the fix for it. Correct pattern (used for v4.19, OBJ-2012): merge master into the branch in place, push back to the same ref (gsd/v4.19-teams-v1-wiring), and open the PR from that ref — the tip sha then matches and pr-scope-guard self-excludes it. Merge with a merge commit, not squash, so the branch becomes a true ancestor of master and both the overlap check and branch-promotion-guard clear permanently, not just for this PR. General rule: for any stranded-branch promotion, the PR head must be the branch itself, updated in place — never a copy under a new name, however tempting a clean rebase looks. (Root cause/evidence: OBJ-2003 measurement, promotion landed via PR #1956, merge commit 7b021a6c0.)

pr-scope-guard's overlap check false-flags when a PR's base is itself a stale long-lived branch

• pr-scope-guard's overlap check false-flags a PR whose base is itself a stale long-lived branch — the two-dot diff can't tell "authored by the branch" from "inherited from an old master ancestor" — gatherBranchOnlyAdditions() computes a two-dot diff (git diff --diff-filter=A base branch) directly against the PR's base ref, by design, so it still works when an integration branch shares no merge-base with base at all. That design has a blind spot: it assumes base is master or something equally current. When base is instead a stale long-lived sibling branch that hasn't taken master's recent commits, any file that landed on master after base diverged — and that the branch under check inherited from its own, more-recent master ancestor — is absent from base's tree and present in the branch's tree, so the two-dot diff misattributes it as "added by" the branch. PR #2446 (base feat/obj-2602-native-legal-consent, stale since ~2026-08-15) hit this: 17 files already on origin/master (e.g. docs/ui-specs/obj-2599-native-legal-consent.md, scripts/check-clerk-auth-config.mjs, SignInUITests.swift) were flagged as leaked from an unrelated gsd/* branch, blocking an otherwise fully-green PR on the guard's sole no-label-bypass check. Fix (shipped): gatherBranchOnlyAdditions() now accepts an optional masterFiles set (from the new gatherMasterTreeFiles(), an origin/master tree listing) and drops any file already present there before attributing it to the branch — it can't be an unmerged-content leak if master already has it. runOverlapCheck fetches masterFiles alongside its existing changedFiles/branches calls via the same Promise.all, so it costs one extra git call total, not one per branch. The real OBJ-1766 leak (144 files, genuinely absent from master at the time) is still fully caught — scripts/check-pr-scope.test.mjs's non-bypass regression test snapshots masterFiles to master's state at that time and confirms the fix narrows the false-positive path without blunting real detection. General rule: this two-dot design trades a merge-base requirement for a currency assumption on base — any future PR whose base is a stale long-lived branch pulling in master content will hit the same shape unless masterFiles filtering stays wired into the call. (Root cause/fix: OBJ-2702, PR #2454.)

Re-landing a reverted merge requires reverting the revert

• After a merge commit is reverted with git revert -m 1, the source branch stays a permanent ancestor of master with its content absent — re-merging the same branch is a silent no-op; the re-land must revert the revert — when PR #2003 (v4.14 Activity Nexus promotion) was rolled back, git revert -m 1 <merge-sha> removed its content from master but left the merge commit itself, and therefore gsd/v4.14-activity-nexus-connected-apps, as an ancestor of master's history. A naive attempt to re-land by merging that same branch again would produce an empty diff — git already considers every one of its commits reachable from master, revert notwithstanding. Fix: every re-land in the OBJ-1982 saga (attempts 2 and 3) instead ran git revert -m 1 <revert-sha> — reverting the revert — from a new branch cut off fresh origin/master, which correctly reintroduces the original content as new commits. Do not delete the original stranded branch or its revert commits when doing this; they remain the reference points the revert-of-a-revert depends on. (Root cause: OBJ-1982 attempt 1 rollback, PR #2017; mechanics confirmed again at attempt 2 → 3, PR #2024 → PR #2034.)

Acceptance criteria that assume a fix's shape can be unsatisfiable

• An acceptance criterion that specifies a fix's shape, not just its outcome, can be impossible to satisfy — and the implementer who fails it may be right, not the criterion — v4.36 Phase 4's (OBJ-2433) task package required "selected-state affordance in DashboardModeSwitcher/HabitCalendar remains distinguishable at ≥ 3:1 against its unselected sibling," written assuming the fix would keep the existing fill-based affordance and just adjust its alpha. Desi measured that no candidate fill pair reaches 3:1 at all — the shipped selected/unselected fill pair sits at 1.00–1.11:1, a palette property, not a call-site choice — so the criterion as written was unsatisfiable by construction, not merely hard. The actual fix relocated the affordance entirely (an opaque border at 5.07:1 plus an accessible Check glyph at 4.32:1), which the criterion's literal wording doesn't describe but whose intent — a genuinely distinguishable selected state — it fully satisfies. General rule: when a phase's acceptance criteria bake in a specific mechanism ("move the alpha," "add a border," "swap the token") rather than the measurable property that mechanism was meant to produce, expect the implementer to report the property they actually achieved, not just the number the literal mechanism failed to reach — and write the next criterion as a property assertion the implementation is free to satisfy by any means, the same way this milestone's own contrast matrix asserts a ratio rather than a technique. (Source: .planning/milestones/v4.36-tint-ink-contrast-floor-ROADMAP.md Shipped Ledger, Phase 4 deviation row; OBJ-2433, PR #2282.)

An acceptance criterion pinned to a named spec assertion can be invalidated by an unrelated merge

• An acceptance criterion that names a specific spec assertion by file and line can stop being satisfiable through no fault of the fix it's gating — a later, unrelated merge can quarantine that exact spec first, and nobody chasing the criterion may notice the target moved. OBJ-3998's original criterion 1 named coach-reconnect-grace.spec.ts:508's resumedLine assertion as the thing a correct fix must make pass. Before that criterion was satisfied, PR #3349 (OBJ-3999) quarantined the entire spec — test.skip(true, SERVER_SIDE_FLAG_GAP_SKIP_REASON) (ionic_frontend/tests/smoke-playwright/specs/coach-enneagram/coach-reconnect-grace.spec.ts:389) — for the client-side-vs-server-side flag-override gap this doc already covers above ("registerFlagOverrides only reaches the client…"), tracked separately at OBJ-4001/OBJ-4005 and not restated here; that gap was unrelated to OBJ-3998's own fix. Two fix rounds were spent chasing the wrong layer — reasoning about the client-side fix's correctness — before anyone noticed the spec itself could no longer pass regardless, because it never ran at all. The line number had also already drifted by the time this was traced: the resumedLine assertion the criterion named at :508 sits at coach-reconnect-grace.spec.ts:546-548 on current master, two rounds of unrelated edits later. General rule: write acceptance criteria as property assertions against code you control ("the reconnect fires exactly one catch-up query within N ms of the socket reopening"), plus a named issue that owns the end-to-end proof when a live spec is the only thing that can observe the full path — never as "assertion X at line N of file Y passes." A named line number is a claim about the codebase's state at authoring time, not a stable target; anything from a routine edit to an entire unrelated quarantine can invalidate it before the criterion is ever re-checked. (Root cause: OBJ-3998 criterion 1 / OBJ-3999 PR #3349; re-enable tracked as OBJ-4006.)

A shared-branch revert can be silently resurrected by a stale agent branch

• Revert-resurrection gotcha: when two agent runs target one shared feature branch and one of them reverts a commit, a later push from the other run's stale branch can silently resurrect the reverted commit — the symptom is indistinguishable from a rogue re-trigger — OBJ-1821/PR #2234: Codi landed a stray commit (213ccd84e, fix(teams): implement team-switcher motion timing (120ms dismiss delay, 150ms header crossfade)) on the shared feat/team-switcher branch, then reverted it (bd13fb2fc) because Roy had already approved the PR without it. Seventeen minutes later the identical diff reappeared on the same branch (53211f5e3) — not new work, but a git replay: all three commits share patch-id 7da6d24591620f00863363a48d34af8b0f22c18b, confirmed directly (git show 213ccd84e | git patch-id --stable and git show 53211f5e3 | git patch-id --stable both resolve to it). The cheap tell: 53211f5e3 preserves 213ccd84e's original author date (2026-08-09T12:16:39-05:00) but carries a fresh committer date (2026-08-09T12:33:19-05:00) — that author/committer split is the signature of a cherry-pick/rebase replay; a genuinely new run or a hand-written re-implementation would have a fresh author date too. The mechanism: Codi works on throwaway agent/* branches pushed onto the shared feature branch, and the run whose agent/* branch was cut before the revert still carried 213ccd84e — a later merge from that stale branch replayed it back in (visible in the merge commit 0887e098b's own message, "Merge remote-tracking branch 'origin/master' into agent/codi/bfc2069e"). A third revert would have had the same odds of being undone as the second — the failure mode doesn't decay with retries — nothing about it is specific to a repeat; it needs only two runs sharing one branch and one revert, and the second occurrence is exactly as likely as a hypothetical third would have been. The recovery loop looks identical to a rogue re-trigger, and hunting for a phantom dispatch is the wrong first move — Riley correctly declined a third silent revert and asked for root cause, and Maggie escalated to a human rather than guess again; the actual cause (stale-branch replay, not re-dispatch) only surfaced from diffing patch-ids and commit dates against the real git history, not from looking for a rogue autopilot or duplicate webhook. Guardrail (shipped): scripts/check-revert-resurrection.mjs + .github/workflows/revert-resurrection-guard.yml hard-fail a PR whose commit range re-applies a patch that was reverted earlier in that same range, using content-based git patch-id matching — never keyed off the Revert "…" subject line, so hand-rolled reverts are caught too — and bypassable only via the revert-reland-ok PR label for an intentional re-land. scripts/check-revert-resurrection.test.mjs replays the real incident straight from this repo's own git history (pinned at archive ref archive/obj-2413-revert-resurrection-incident @ 53211f5e3, since feat/team-switcher was squash-merged and became eligible for the stale-branch sweep) and asserts it flags all three SHAs red; a clean range passes green. General rule (crew-wide): one run, one branch — a run pushes only to the branch of the issue it is working, never to a branch owned by another issue's hop; see docs/guides/multica-agent-crew.md, rule 8. Riley additionally re-verifies the branch tip against the last report at every hop that touches it (PR-open, post-Roy-approval, pre-Vicki), since this incident's resurrection landed in exactly the post-approval window. (Root cause: OBJ-2413, incident OBJ-1821/PR #2234. Guard: OBJ-2420.)

A shared-mirror git remote add inside one task's checkout permanently pollutes every other task's checkout

• A git remote add (or a git fetch <path>) run inside a Multica-managed checkout is not scoped to that task — every worktree shares the mirror's ref store and config, so the remote is written into shared, durable host state and inherited by every later checkout — root-caused from the 2026-08-11 checkout brownouts (OBJ-2449): the shared enkidu mirror on joshuas-mbp-2 had accumulated 44,520 refs vs 682 on origin. Twelve remotes had been added over time from inside agent task worktrees — codi-fix, codi-fix-116, codi-phase116, codi-source, codi-workspace3, codi-worktree, codi_fix, codi_workspace, dori-eced1e24, dori-local, dori-workspace, orion-workspace — each pointing at another task's ephemeral workdir (agents cross-fetching each other's checkouts). Each git remote add + git fetch from inside a task dumped thousands of remote-tracking refs (and ultimately ~40k reflog files) into the shared cache, which made the daemon's per-cycle reflog expire --all / gc time out every cycle (OBJ-2476) and browned out checkouts for 20–40 min every ~2h. General rule: never git remote add inside a Multica-managed checkout, and never git fetch <path> pointing at another task's working directory — every branch you need is on origin, fetch it there (git fetch origin <branch>). Codified in the multica-handoff-protocol workspace skill's "Git remotes — origin only" section and crew-wide rule 9 in docs/guides/multica-agent-crew.md. The rule is no longer unenforced. A 2026-08-19 re-audit (OBJ-2739) found the OBJ-2477 cleanup this entry used to describe as "done" had never actually run — all 12 remotes and 92.7% of the mirror's refs were still junk. Fixed for real this time via scripts/multica-mirror-hygiene.mjs (--audit/--prune/--sweep, allowlist-based, refuses to touch anything outside ~/multica_workspaces/.repos/) plus an hourly --sweep --yes LaunchAgent (scripts/install-mirror-hygiene-agent.sh) that clears any non-origin remote and stale ref locks before they can re-accumulate for more than an hour — not yet installed on the host as of this writing, a pending operational decision, not a code gap. Full write-up including the measured before/after numbers and the corrected stale-ref-lock causality model (it's downstream residue of the daemon's own 30s-timeout SIGKILL, not an independent contributor — see OBJ-2759): shared-mirror-remote-pollution-gotcha.md. (Root cause: OBJ-2449. Original prevention: OBJ-2477. Companion gc-timeout escalation: OBJ-2476. Re-audit + enforced guard: OBJ-2739. Daemon-side SIGKILL bug: OBJ-2759.)

• Addendum (OBJ-3657) — the identical shared-mirror-config-write mechanism poisons [user] as easily as [remote], and reframes three "domain drift" incidents as one shared file, not per-session ambient state. A stray git config user.name/user.email — no --global, no -c override — run inside any one task's worktree writes into the mirror's own local config (not ~/.gitconfig), which outranks every agent's correct global identity for every worktree checked out from that mirror, by every agent, going forward. This is the exact mechanism behind the release-integrity gotcha's f58e1ba4 incident: four of Codi's commits on phase-6-docs-assistant landed as Codi <josh@objectuve.com> (the workspace owner's own personal email) instead of codi@objectuve.ai, which then squash-merged into PR #3097's Co-authored-by: trailer and tripped release-integrity-guard.yml's commit-attribution job red on master. Evidence: a stray [user] block (name = Codi, email = josh@objectuve.com) sat at line 2064 of the mirror's 4,612-line config, among 1,529 [branch] entries — cleared by hand during OBJ-3657 planning (recorded verbatim in that issue's task-package comment, since a mirror config isn't itself version-controlled). The mirror's extensions.worktreeConfig=true makes a per-worktree config.worktree file an equally invisible variant of the same trap; none of the ~1,500 worktree-scoped config files on this mirror carried one at audit time. Why this reframes the OBJ-2930 / OBJ-3299 / OBJ-3610 addenda above: each reads its recurring domain drift as "whatever a prior session last left configured" — a per-session, ambient-state framing. It's one shared file. Each incident is a different stray write landing in that same place, which is why fixing it per-agent (adding an inline -c override to one more agent's instructions each time) kept not generalizing to the next one. The residual, still true after the manual clear: the runtime host's own ~/.gitconfig carries Codi <codi@objectuve.ai> — correct, but still an agent identity sitting in a human's global config, one scope up from the mirror. An un-pinned commit anywhere in the crew now lands silently as Codi <codi@objectuve.ai>, which passes evaluateCoAuthorAttribution — the inline -c user.name=/-c user.email= override at the point of commit stays mandatory for every agent regardless of whether the mirror or the host config is clean. The scan-window trap, worth reading twice: release-integrity-guard.yml's commit-attribution job only evaluates github.event.before..after on a push, and the last 24h of history on its daily 17 6 * * * cron (.github/workflows/release-integrity-guard.yml) — a co-author violation that lands on master scrolls back out of every future scan window within a day, whether or not anyone actually fixed it. Green stops meaning remediated the moment the bad commit is older than the window. Fixed: scripts/multica-mirror-hygiene.mjs now audits and sweeps user.* keys (FORBIDDEN_CONFIG_PREFIXES) in both --local config and any config.worktree file, using the same allowlist-based, scope-fenced design as its [remote] sweep above; its installer now copies the script to a durable ~/.multica/bin/ path so the hourly LaunchAgent survives the installing task workdir being torn down — the job had never actually been running on the host before this fix (its log's last entry was 2026-08-19, the one-time manual cleanup, three-plus weeks before this fix). evaluateCoAuthorAttribution (scripts/check-release-integrity.mjs) gained a remediated-incident allowlist mirroring its sibling evaluateCommitAttribution check, with the already-merged f58e1ba4332dfc6cd9738d402853590a788cff98 entered in scripts/release-integrity-allowlist.json citing this issue. (Diagnosis: OBJ-3657, Dave. Guard: OBJ-3657, Codi. Docs: OBJ-3657, Dori.)

A blocking follow-up flagged in an execution report isn't a tracked issue until someone creates one

• A blocking prerequisite can be flagged in three independent places at once — a code comment, a doc caveat, and an execution report's "Follow-ups" section — and none of those is itself a tracking mechanism, so the gap can still sit invisibly until it manifests as a live incident — PR #2567 (47782f871, merged 2026-08-22, OBJ-2822) added a staging-only Mailtrap sandbox SMTP override sourced from two new env vars, and flagged that they weren't provisioned yet in all three surfaces: the commit itself left an inline comment a few lines above the reference (rails_api/config/environments/staging.rb:17-18, "...same SMTP delivery path, but captured rather than relayed. Credentials are staging-only secrets, not yet provisioned — see docs/architecture/email.md."); docs/architecture/email.md's env-var reference repeated the caveat ("Not yet provisioned as of this writing — see the Follow-ups note on the OBJ-2822 issue"); and Codi's execution report on OBJ-2822 explicitly called it out as blocking: "MAILTRAP_SANDBOX_SMTP_USERNAME and MAILTRAP_SANDBOX_SMTP_PASSWORD... not yet provisioned as GCP Secret Manager secrets, and not yet wired into .github/workflows/staging.yml's --set-secrets lists... Until then, staging SMTP auth will fail closed." All three are correct, specific, and visible to anyone who reads that PR or that issue — and none of them created a tracked issue. multica issue search "MAILTRAP_SANDBOX" returned zero results as of 2026-08-24, so nothing routed the gap to whoever holds Mailtrap dashboard + GCP Secret Manager access. It sat for two days until Email::DeliveryCanarySendJob (an unrelated OBJ-2820/2825 feature that happened to ship the same window) ran on Crono on the first tick after the merge (2026-08-23T07:00Z) and became the first thing to actually exercise ActionMailer::Base.mail.deliver_now on staging post-merge: user_name resolved nil, authentication: :plain was still in effect (inherited from production.rb, never disabled for the staging override), and Net::SMTP raised ArgumentError: SMTP-AUTH requested but missing user name on every attempt since — surfaced as a Sentry issue and /health's checks.email_canary reading stale (full diagnosis on OBJ-2893). General rule: when a PR flags something that will break correct runtime behavior once its code path is actually exercised — not a nice-to-have, not a style cleanup — open the tracked issue at the moment you flag it, assigned to whoever holds the missing capability (credentials, infra access, a dashboard only a human can reach), rather than trusting a code comment, a doc caveat, or an issue thread to carry it forward on its own. Writing the caveat in three places felt like redundant coverage; none of the three is a tracking mechanism, so it was really zero. (Root cause: OBJ-2822/PR #2567's follow-up, never ticketed. Surfaced as incident: OBJ-2893. Companion guard-blind-spot gap in the same incident: Three separate guards missed the same unwired secret, each for a structurally different reason, OBJ-2894.)
• A run-only diagnostic autopilot with no existing-issue dedup step re-diagnoses the same still-open incident on every scheduled cycle it stays unresolved, producing a fresh, independent issue chain each time — the incident above surfaced a second, distinct gap in the same window: "Dave — Staging health check" (execution_mode: run_only, scheduled every 4h Mon–Fri 8 AM–8 PM CT) ran at ~13:00 CT and again at ~17:00 CT on 2026-08-24 against the exact same unresolved Sentry issue behind that incident (OBJECTUVE-API-2S, permalink https://objectuve.sentry.io/issues/7687459109/) — identical short ID, first-seen, last-seen, and event count on both runs, confirming it was the same live incident, not a recurrence. The autopilot's own Decision section has no dedup step: "Any FAIL (and no BLOCKED): confirm by re-running the failing check 30 seconds later. If still failing, create degradation issue. Assign to Maggie" — nothing checks whether a matching issue is already open first. The 13:00 run correctly produced OBJ-2893 (root cause) → OBJ-2894 (validate-secrets.sh fix) → OBJ-2895 (this gotcha entry's own original write-up); the 17:00 run, unaware of any of that, independently re-diagnosed the identical Sentry issue from scratch and opened a second full chain, OBJ-2907 → OBJ-2908 → OBJ-2909 → OBJ-2910, duplicating OBJ-2893's root-cause diagnosis, OBJ-2894's fix, and this very doc entry's ask. Maggie reconciled it on discovery: OBJ-2908 cancelled as duplicate, OBJ-2909 rescoped to its one non-duplicate fix and routed to Codi, OBJ-2907 cancelled as duplicate diagnostic output. Why this one's easy to miss: two sibling Dave autopilots already carry the fix — "Dave — Clerk auth tripwire"'s Decision section: "Dedupe first. Look up OBJ-2532 ... Do NOT open a duplicate"; "Dave — Signal instrumentation audit monthly"'s: "dedupe first: search open issues for title containing 'Signal instrumentation audit' ... If one is already open ... comment the new findings onto it instead of creating a duplicate" — but "Dave — Staging health check" was never updated to match, so the dedup pattern exists twice in this same autopilot family and is absent from a third. General rule: any run-only or create-issue autopilot whose FAIL path creates a new issue needs an explicit dedup step — search open issues for the identifying fact of the failure (a Sentry short ID/permalink, a check name, an error signature) before creating anything — matching the pattern already proven out on the Clerk auth tripwire and Signal audit autopilots. Not yet fixed: "Dave — Staging health check"'s Decision section still has no dedup step as of this writing (its updated_at is 2026-08-24T17:00:12Z, from the very run that hit this gap) — a follow-up should add the same "look up before opening" language its two siblings already use. (Discovered: OBJ-2910, Dori, while investigating the entry above. Duplicate chain: OBJ-2907/2908/2909, reconciled by Maggie against OBJ-2893/2894/2895.)
• Addendum (OBJ-2962) — the dedup step shipped since the "not yet fixed" note above, but it only searches open issues, so a fix that closed its own tracking issue minutes before the next scheduled probe still produces a duplicate. "Dave — Staging health check" ran again ~21:00 UTC on 2026-08-28 against Sentry OBJECTUVE-API-2W (Email::DeliveryCanaryVerifyJob), still unresolved/regressed, and — per this run's own dedupe rule, "comment on an open issue tracking the same signal, else file new" — filed a new issue (OBJ-2962) rather than commenting on an existing one, confirming the dedup step named as missing above has since shipped. It just didn't find a match: the correct dedupe target, OBJ-2956, had already fixed this exact signal (find_by_subject_scan wasn't passing the search query param Mailtrap requires, so a canary message could fall off page 1 of a shared, unpurged inbox — PR #2655, merged 2026-08-28T16:39:51Z, live on staging by ~19:05 UTC) and was status: done by the time OBJ-2962's probe ran, so it fell outside an open-issues-only search. Sentry itself couldn't disambiguate either: it won't flip OBJECTUVE-API-2W to resolved until a fresh cron cycle succeeds, the canary runs once daily (~07:00 UTC), and the 19:05 UTC fix hadn't had one yet — the 21:00 UTC probe was reading the same pre-fix unresolved state Sentry had shown all day. Dave re-diagnosed on OBJ-2962 and confirmed no event newer than the 07:31 UTC occurrence OBJ-2956 already captured, confirmed the fix live via gcloud run revisions describe, and closed it as a duplicate — no code fix needed. Refined rule: an open-issues-only dedup lookup is not enough for a signal identified by something more durable than issue status — a Sentry shortId/permalink, a check name, an error signature. The lookup window should also cover issues closed in roughly the last 24–48h sharing that same identifying fact, especially when the closing fix's deploy timestamp postdates the signal's last confirmed-good cron cycle — that gap (deploy live, but not yet cron-verified) is exactly the window a closed-but-unproven fix sits in. This is a Multica autopilot-config change, not application code — the fix lives in "Dave — Staging health check"'s own Decision section (Multica platform config), out of this repo's remit; flagged for Maggie/human to action directly. For anyone reading this incident chain cold: OBJ-2951 (field-name bug: Mailtrap::EmailLogs read the sandbox field id instead of the sending-logs API's message_id, PR #2650) → OBJ-2956 (page-1/shared-inbox bug above, PR #2655) → OBJ-2962 (this addendum, duplicate) are three issues for one underlying Mailtrap email-canary defect chain, not three separate incidents — follow the chain rather than treating any one of them as a fresh unresolved thread. (Root cause: OBJ-2951/OBJ-2956. Diagnosed duplicate: OBJ-2962, Dave. Addendum: OBJ-2963, Dori.)
• A second, distinct instance of this exact failure mode: a tracking issue number named in this doc's own prose, that was never actually created. The OBJ-2958 addendum above (Mailtrap sending-logs pagination gap) named "OBJ-2986, v4.7.1 release-prep" as the follow-up's tracking pointer when it was written — but no Multica issue OBJ-2986 was ever created. The diagnosis sat as accurate, well-evidenced prose in this file while the underlying bug kept re-firing (Sentry OBJECTUVE-API-2W, recurring since 2026-08-25, email_canary:last_success:production's 48h TTL repeatedly expiring) with nothing tracking it, until Dave re-diagnosed the same root cause from scratch on OBJ-3051 (2026-08-31) — the pointer has since been corrected in place above, but the underlying gap (a dead issue-number reference silently surviving in a doc nobody re-validates against multica issue get) is now confirmed to recur, not a one-off. Standing rule, going forward: a tracking-issue number is never written into this doc speculatively. Only cite an issue number here once multica issue create (or equivalent) has actually returned that number — if a fix is flagged as needing a follow-up before the issue exists, write "not yet ticketed" instead of inventing or pre-assigning a number, and update the doc with the real number once the issue is created. A doc audit (objectuve-atlas) that checks every OBJ-\d+ reference against a live Multica lookup would catch a dead pointer like this mechanically instead of relying on someone re-diagnosing the underlying bug first. (Dead pointer: OBJ-2958 addendum, this file. Never-created issue: "OBJ-2986." Re-diagnosed and re-ticketed as: OBJ-3051, Dave. Doc fix: OBJ-3076, Dori.)
• A third, distinct instance: a PR's own test-plan checklist named the exact prerequisite gap that later broke a deploy, and it still wasn't ticketed at the time. PR #2949 (2615247e5, OBJ-3416, "repoint Cloud Run staging at dedicated Cloud SQL instance," merged 2026-09-06) repointed staging's DATABASE_PASSWORD at a new staging-db-password:2 secret — but its own test-plan explicitly flagged, as an unchecked item, that disabling the old secret version required "Secret Manager IAM this session's identity doesn't have — falls to whoever verifies the deploy." What the checklist didn't separately flag, and what actually broke, was the mirror-image gap: nobody had confirmed the new version's IAM grants either. enkidu-run-jobs-staging@enkidu-488723.iam.gserviceaccount.com never got roles/secretmanager.secretAccessor on staging-db-password:2, so the first run to reference it — the PR's own merge commit — failed at the migrate step with PERMISSION_DENIED. As with the OBJ-2822/PR #2567 case above, the flagged prerequisite was correct, specific, and visible in the merged PR — and, same as that case, naming it in a checklist item is not a tracking mechanism. This surfaced as OBJ-3425 only once Dave traced a live staging-blocker back to it, not from the PR's own follow-up being routed anywhere at merge time. The general rule from the OBJ-2822 case holds without modification: when a PR's own test-plan flags a prerequisite as gated on access the current identity doesn't have, open the tracked issue for it at merge time, not just an unchecked checkbox — a checked-off "requires IAM I don't have" line and an unticketed one look identical in a merged PR's diff. (Root cause: OBJ-3416/PR #2949. Surfaced as incident: OBJ-3425.)

Nothing mechanically coupled a milestone's ROADMAP ✅ flip to its own phase tick

• A milestone-close PR and its own last phase-tick PR touch disjoint files by design — which is exactly why pr-scope-guard's cross-branch-overlap check can't see one racing ahead of the other — PR #2631 (the v4.45 milestone close) merged 2026-08-24, roughly 29 minutes before PR #2628 (the PR that ticked v4.45's own Phase 4 complete in .planning/milestones/v4.45-*-ROADMAP.md), inverting the intended tick-then-close order; PR #2628 ended up documenting the inversion in its own body rather than anything preventing it. The mechanism initially suspected — no reviewer requested — wasn't the real gap. A reviewer was requested on #2631 within 2 seconds of open, and it changed nothing: the master CI gate branch-protection ruleset has required_approving_review_count: 0, so a review request is advisory only and cannot block a merge. pr-scope-guard's overlap check couldn't catch the race either, for a structural reason, not a bug — it flags file-set intersections between unmerged branches, and the tick PR and the close PR share zero files: #2628 touches only .planning/milestones/v4.45-*-ROADMAP.md, #2631 touches .planning/ROADMAP.md plus the audit and narrative docs. A clean, non-overlapping diff is precisely the shape that let the sequencing bug through undetected. Not a one-off: the same shape had already shipped once before unnoticed — PR #2386 (v4.37 close, 2026-08-16) closed with its own Phase 6 never ticked. Guardrail (shipped): scripts/check-milestone-close-ordering.mjs + .github/workflows/milestone-close-ordering-guard.yml run on every PR (unfiltered, so the check can later be promoted to required) and fail any PR that flips a .planning/ROADMAP.md entry to ✅ shipped while that milestone's own .planning/milestones/v<X.Y>-*-ROADMAP.md still has an unticked - [ ] **Phase N line at the PR's head. Diff-scoped, not a static invariant — a static sweep over the whole tree hits 5 already-shipped milestones with unticked phases (v1.21, v2.0, v2.3, v4.14, v4.37), so only a PR that itself flips the ledger is in scope. Bypass via the milestone-close-unticked-ok label plus an Unticked-Justification: <reason> PR-body line — both required, mirroring the existing milestone-close-superseded-ok/Superseded-Justification: pattern — for the legitimate case of closing with a phase deliberately dropped (the v4.37 shape). scripts/check-milestone-close-ordering.test.mjs replays all three PRs straight from this repo's own git history (#2631 → fail, #2601 → pass, #2386 → fail). Not yet promoted to a hard gate: the guard isn't in master CI gate's required status checks (ruleset 17927598) — it shows red, it doesn't block — promoting it is a repo-admin action, deliberately out of agent scope. General rule: on this repo's default branch-protection settings, a requested reviewer is not a merge gate (required_approving_review_count: 0) — don't design a sequencing safeguard around one; and a cross-branch-overlap guard only catches races between PRs that share files, not races between PRs whose correctness depends on one another while touching none of the same paths. See the milestone-close ritual's ordering section for the enforced rule and Dori's runbook pre-flight step. (Root cause: OBJ-2947, incident PRs #2628/#2631, retroactive second instance #2386. Guard: OBJ-2947, Codi.)

Squash-merge defeats branch-name PR lookup

• A branch-name-based "has a merged PR" check and an ahead-of-master ancestry count are both structurally wrong for this repo's workflow, and will misreport every worker branch that ever ships cleanly as abandoned — two facts compound: (1) worker branches commit to agent/<name>/<hash>, but Riley opens the PR from Riley's own branch, so a worker branch's name never matches a PR's headRefName, no matter how cleanly its work shipped; (2) git rev-list origin/master..<tip> never reaches zero after a squash-merge, since squashing rewrites the branch's commits into one new SHA on master, leaving the originals permanently non-ancestral. The weekly Riley — Stale branch pruning autopilot used exactly these two signals and flagged 16 branches as "no PR, no merge, real unmerged work" (OBJ-3088) — full triage found all 16 were already fully landed on master. Fix (shipped): scripts/multica-stale-branch-triage.mjs (OBJ-3099) resolves per unique commit instead — GET /repos/{owner}/{repo}/commits/{sha}/pulls, which survives a squash because GitHub keeps the commit→PR association independent of ancestry — falling back to byte-identical content-on-master comparison only for a commit with no PR match at all. Neither signal met ⇒ unlanded (unproven, never "abandoned"); any API error, malformed response, or page-cap hit ⇒ unknown, never deletable. Two traps hit building the fix itself: the field is merged_at (REST, snake_case), never mergedAt (GraphQL — gh pr view --json mergedAt is a different field, and using it reads as "never merged" for a demonstrably-merged PR); and never gate on .state, which reports "closed" for a merged PR too. Full mechanism, the four independent destructive-path defects fixed in the same script, and the honest 1-landed/0-content-landed/15-unlanded replay of the original 16 branches: squash-merge-defeats-branch-name-pr-lookup-gotcha.md. (Root cause + false positives: OBJ-3088. Fix: OBJ-3099. Pending autopilot rewire: OBJ-3110.)

An autopilot fire can silently skip, not fail, with no issue created when a trigger's owner is missing

• multica autopilot runs <id> records a per-fire status that isn't limited to completed/failed — a status: skipped entry means the platform rejected dispatch before the assignee agent was ever invoked, and no issue or task exists anywhere to point at it later. The Riley — Stale PR sweep autopilot's 2026-09-09T19:00:27Z weekday fire returned status: skipped with failure_reason: "this trigger's owner lacks access to the private assignee agent, or the trigger records no owner" — a reason that reads as a config/ownership break (a schedule trigger's owner field going unset/null), not a transient dispatch blip like the known "agent runtime is offline at dispatch time" skipped cases (10 occurrences across 5 autopilots, clustered on 2026-06-01/06-16). Dave's investigation (multica autopilot get/runs, multica agent get, multica workspace member list) ruled out an agent-level or systemic platform cause: this workspace has exactly one member owning every agent and autopilot identically, no other trigger of similar age skipped the same day, and every other skipped occurrence workspace-wide matches the known offline-runtime pattern — isolating this to one trigger row. There is no CLI-exposed way to inspect or set a trigger's owner directly (trigger-add/trigger-update take no --owner flag), so it can only be observed via this symptom, not confirmed or repaired client-side. Check run status, not just failed entries, when an autopilot appears to have gone quiet for one cycle — a clean-looking history of completed/failed runs can still hide a skipped fire in the gap between them, and a skipped fire creates nothing else to alert on. (Diagnosed: OBJ-3598, Dave. Not yet recurred as of this writing; if the trigger skips again, the fix needs Josh directly — trigger ownership can't be reassigned by a crew agent.)

A webhook autopilot's POST body never reaches the run — trigger_payload is always null

• A Multica webhook autopilot's run never sees the payload POSTed to trigger it — anything the run needs must be rediscoverable from workspace state, never read off the webhook body. Every fire of the Riley — CI wake autopilot (webhook trigger 8a8876d5-65fc-4d12-89ca-27b0cdc4761a, sent by .github/workflows/multica-ci-wake.yml) records trigger_payload: null in multica autopilot runs <id>, even though the workflow's step 4 curl POSTs a real JSON body ({pr, head_branch, workflow, conclusion}) on every send. Orion's diagnosis on OBJ-3590 found trigger_payload: null across all 94 consecutive webhook runs sampled; a spot-check of the latest page while writing this entry found the same — 0 of 18 webhook runs returned had a non-null payload. This isn't a malformed send: the workflow's curl -sf fails fast and logs a gh api failed-style notice (then exits 0, fail-open by design) if the POST itself doesn't leave CI, so admission is succeeding — Multica's webhook ingress just doesn't surface the body to the run. Design around it: a webhook-triggered run has to re-derive everything it needs (which issue, which PR, what changed) from multica issue list/gh pr checks/etc. at run time — never assume a payload field is available, on this or any other generic-provider webhook autopilot. Whether Multica's admission layer honors an Idempotency-Key header for a generic-provider webhook is a separate, still-untested question (see OBJ-3589/OBJ-3590) — the payload gap here doesn't answer it either way. (Diagnosed: OBJ-3586/OBJ-3590, Orion.)

BLOCKED + empty reviewDecision + all-green checks: the trigger is required_status_checks, not the unattributed-changes gate

• A PR showing mergeStateStatus: BLOCKED, reviewDecision as an empty string (not REVIEW_REQUIRED), and every required check green is not waiting on a reviewer assignment, and is not the unattributed-changes gate either — it is required_status_checks, where a required context goes "expected" (never posted) rather than failing. The master CI gate ruleset (17927598) does carry require_extra_approval_for_unattributed_changes: true on its pull_request rule alongside required_approving_review_count: 0, and every Multica crew commit is unattributed (gh api .../pulls/<n>/commits returns author: null, because crew committer emails like dori@objectuve.com map to no GitHub account) — but those two true facts do not combine into a block, and assuming they do is the trap this entry exists to close. The decisive evidence (2026-09-05): of the 25 rule suites in the retained window on master, 23 are result: pass and only 2 are bypass — and 20 of those 23 passing suites are PRs whose commits are entirely author: null. Unattributed PRs merge organically, without an admin bypass, twenty times over. A gate that fired on unattributed commits could not produce that record. On the one PR that did block: #2924 sat BLOCKED ~5h with all 14 required contexts success on its head SHA 8647bc59 and cleared only via gh pr merge 2924 --admin --squash. Its bypass is recorded in the org audit log with overridden_codes: ["required_status_checks"] — that code, alone. GitHub names what it overrode, and it did not name the review policy. review_policy_not_satisfied is a code this log emits (it appears on suites 3954616644 and 3924823553), but both of those are direct pushes to master, not PR merges — their pull_request rule failed with "Changes must be made through a pull request." The unattributed-changes gate has still not been observed to fire on any PR merge in the recorded audit-log window — the log's retention is short (25 suites, ~24h as sampled on 2026-09-05), so this is a repeatedly-checked negative over the observable record, not an all-time proof. Two prior corrections were both wrong, in opposite directions: the original entry claimed a claude/multica-agent co-author trailer fires the gate (falsified 2026-09-04 — trailer-carrying PRs merged clean); the 2026-09-05 first pass then claimed commit attribution was the real discriminator (falsified here by the 23 pass suites). On the handful of PRs in front of me at the time, attribution lined up with the outcome closely enough to look causal, which is exactly why it survived one round of scrutiny — widen the sample to all 25 suites and it collapses. Diagnose in this order, and stop at the first answer: (1) gh api "/orgs/objectuve-softworks/audit-log?phrase=action:protected_branch.policy_override" — for an already-bypassed merge, overridden_codes names the rule outright and rule_suite_id links the per-rule detail; (2) for a PR still sitting BLOCKED (no suite exists yet, since GitHub records a suite only at push time), compare the ruleset's 14 required contexts against what has actually posted on the head SHA: comm -23 <(gh api repos/objectuve-softworks/enkidu/rulesets/17927598 --jq '.rules[]|select(.type=="required_status_checks").parameters.required_status_checks[].context' | sort -u) <(gh api repos/objectuve-softworks/enkidu/commits/<head-sha>/check-runs --jq '.check_runs[].name' | sort -u) (both sides must be sorted or comm returns wrong output silently — hence sort -u on each). A context in the required list that never posted is the block; re-running CI does not help if the workflow never triggers for that context. Note the head SHA can read all-14-green after the fact — as #2924's does — so a green rollup today is not evidence the contexts posted in time. Do not infer the cause from mergeStateStatus/reviewDecision alone, and do not infer it from the ruleset's parameter list: a parameter being true says only that it is enabled, never that it evaluated against you. Recovery: gh pr merge <n> --admin --squash by a human with OrganizationAdmin. Self-approval is structurally impossible regardless — crew runners' gh CLI authenticates as joshlock3, so that account is always the PR author, and GitHub forbids self-approval; objectuve-multica-bot is the only other write collaborator. Which two of the 14 contexts go "expected," and why, remains unidentified — that is the real open question here. (Diagnosed: OBJ-3264/PR #2843, Josh. Documented: OBJ-3319/OBJ-3320. Corrected 2026-09-04 against a co-author-trailer probe; corrected again 2026-09-05 against a full 25-suite audit plus the #2924 audit-log override codes.)

A post-merge push to an already-merged PR's branch is silently orphaned

• GitHub squashes whatever the PR's recorded headRefOid was at merge time — commits pushed to that branch afterward never enter master, and nothing signals it: the PR still reads MERGED, the branch still shows the commits, git log on the branch looks correct, and no warning fires anywhere. PR #3067 (OBJ-3613) landed its squash commit 33ec20a88 at 2026-09-11T09:48:03-05:00 against a recorded headRefOid of 5b225081b (3 commits). Two follow-up commits — 555eccef7 (09:51:08, +3 min, restoring the fail-closed floor) and 3ec05298e (09:57:41, +9 min, a test exercising it) — were pushed to the same branch after the merge had already happened, so neither was ever part of what GitHub squashed; git merge-base --is-ancestor 3ec05298e master exits 1. The consequence: the required-context guard's fail-closed floor silently never reached master, while a receipt in the same session claimed it had shipped. Diagnose with this triple, in order: (1) gh pr view <n> --json headRefOid,commits — compare the recorded headRefOid against the branch's current tip; a tip that's moved past it is the tell; (2) git merge-base --is-ancestor <sha> master; echo $? — exit 1 means that commit is orphaned, full stop; (3) compare each commit's committer date against the squash commit's date — anything after is guaranteed orphaned. The rule worth internalizing: gh pr view reporting MERGED is not proof your commits landed — only an ancestry check is. Recovery doesn't need force-pushes or branch surgery — cherry-pick the orphaned commits onto a fresh branch off master and open a new PR (done here as PR #3075). Scope: this is a squash-merge trap specifically — the orphaning follows from GitHub squashing the headRefOid recorded at merge time, so don't generalize the mechanism to other merge strategies; the ancestry check in step (2) is the right falsification under any of them.

Marketing Site

A shared JS contract change must migrate every consuming page

• A shared JS behavior-contract change is only safe if it migrates every page that uses it — orphaning the attribute on unmigrated pages renders silently wrong, not empty, and the existing referenced-undefined guard cannot see it — js/main.js is shared, unconditionally, across every marketing_landing/*.html page regardless of which CSS bundle that page loads. Commit b34d45bfa ("shared header, nav & motion contract," PR #2178, 2026-08-07) rewrote main.js's scroll-reveal IntersectionObserver to target only .reveal/[data-enter] and migrated the markup on index.html/meet-coach.html to match, stating in its own commit message that it "drops the dead [data-animate]/.scrolled paths." That classification was wrong: about.html (9 occurrences) and contact.html (2 occurrences) still used [data-animate] — unmigrated because they run on the legacy css/styles.css page-chrome contract, not landing-2026.css, so they fell outside the PR's own "all 10 landing-2026.css pages verified unregressed" check. styles.css:1778-1787 still defines [data-animate] { opacity: 0 } / [data-animate].is-visible { opacity: 1 }, but nothing in the new main.js ever selects [data-animate] again, so nothing ever adds .is-visible — every wrapped content node stays permanently at opacity: 0 for any visitor with prefers-reduced-motion: no-preference (the default). The page returns HTTP 200, nav/footer/section backgrounds render fine, there's no console error and no full-page crash — only the content is invisible. This sat live for ~2 days before the first report (OBJ-2377). Why the existing guard didn't catch it: scripts/check-marketing-referenced-undefined.mjs catches a class/id referenced in markup with no matching CSS/JS definition anywhere — [data-animate] is still defined in CSS, so it isn't "undefined," it's orphaned: defined in both HTML and CSS, with only the JS half of the contract silently removed. The guard's documented scope (scripts/check-marketing-referenced-undefined.mjs:20-35) is explicitly one-directional (referenced-but-undefined, not the reverse of "defined but unimplemented") and does not cover this shape in either direction. When migrating a shared JS behavior contract (main.js, or any script shared across pages that don't all load the same CSS bundle): grep every marketing_landing/*.html file for the attribute/class being retired, not just the pages the PR's stated intent touches — a page on a different CSS contract (legacy styles.css vs. landing-2026.css) is not a signal that it's also off the shared JS contract. See Dave's diagnosis on OBJ-2377 for the full reproduction and hypothesis log. (Root cause: OBJ-2377, commit b34d45bfa, PR #2178.)

Tooling & Evidence

jq -r '.path' on an absent key returns the string "null"

• jq -r '.path' on an absent JSON key returns the literal string "null", not an empty string and not a non-zero exit — A missing key prints null to stdout with exit code 0, so [[ -z "$X" ]] guards pass through a string that is not empty, and downstream consumers receive the four-character word null. This silently corrupted Firebase preview-URL extraction (a missing .result["enkidu-app"].url became "null", then a literal null URL was polled for 300 s). Fix: use the // empty alternative so an absent path yields a bash-empty string and a non-zero pipeline: jq -r '.result."enkidu-app".url // empty' deploy_output.json. Always pair jq -r extraction of an optional field with // empty (or // "default"), then test the result. (Source: OBJ-651, Dave's diagnosis.)

A staleness gate can fire against its own prior correction

• A staleness gate can fire against its own prior correction, mistaking "the fix just landed" for "the fix needs redoing" — scripts/capture-doc-evidence.mjs's lastCommitEpoch() used a raw git log -1 for a doc's "last commit." A footer-only hygiene fix (correctly backdating Last updated to the real last content change) became that new "last commit" itself — and since the footer wasn't bumped to today, currency()'s footerBehindCommit check re-tripped on every subsequent sweep, forcing an endless re-"fix" of a footer that was never wrong (docs/development/fm-fade-up-error-remount-gotcha.md hit this twice, PR #1705 then PR #2066). Same family as the continue-on-error/PostHog-reverse-drift gotchas above: a gate whose own correction becomes new input it then flags as a violation. Fix (shipped): lastCommitEpoch() now walks history newest-first and skips commits whose diff for the path is footer-only, landing on the real content commit instead. General rule: when a gate detects "X is behind Y," check whether fixing X can itself become the new Y — if so, the gate needs to look past its own correction commits, or it will never converge. (Root cause: OBJ-2137, PR #2080.)

A single-line footer that chains narrative guarantees concurrent PR conflicts

• A Last updated: footer that accumulates narrative from multiple tickets onto one physical line makes every concurrent PR touching that doc conflict with certainty, and on a hot doc the conflict reopens faster than one CI cycle can clear it — PR #2341 (docs(auth): fix authentication.md drift, OBJ-2537) hit three fresh merge conflicts in a row on the exact same line of docs/architecture/authentication.md: its **Last updated:** footer, which had grown to 2,432 characters across 6 stacked entries (Previous entry: / Same day: / Prior entry: / Earlier entry: chaining). Git merges line-by-line, so any two PRs editing that doc collide on that one line regardless of what else they touch — and because the line carries prose from multiple unrelated tickets, resolving it requires synthesizing narrative, not a mechanical pick. authentication.md took 6 commits in 30 days (5 of them in the final 2 days: #2344, #2351, #2349, #2352, #2341 itself), so master re-conflicted the footer faster than a single resolve-and-CI hop could clear the previous conflict — round 1 was a Dori synthesis, round 2 reopened before CI even finished, and round 3 reopened before Riley could start the round-2 resolution. Riley correctly declined a fourth hand-merge and escalated as a loop rather than resolving again. Root cause: the chained footer was drift from this repo's own documented convention (docs/development/docs-authoring.md#page-footers specifies a single line) — nothing asked for chaining, it accreted one "helpful" preserved-history edit at a time. At the time of the incident, 328 of 341 docs/ footers were still compliant one-liners; authentication.md and accessibility.md (3,106 chars) were the worst offenders. Fix (shipped): PR #2341 landed by resetting its footer edit to master's current line (a footer-change-free diff can't re-conflict on content it doesn't touch) with GitHub auto-merge armed so no agent hop sat between green CI and merge; a follow-up PR then trimmed authentication.md's footer to one canonical entry and made the one-entry rule explicit in docs-authoring.md. General rule: never chain footer entries — replace the whole line on every edit. If a doc's footer is already chained and over ~400 characters, trim it to one entry the next time you touch that file; don't add a 7th entry on top. (Root cause: OBJ-2562, PR #2341; three-round loop documented on OBJ-2562 comment thread, Riley's escalation.)

A shallow clone can poison path-limited git log results

• A git log -- <pathspec> walk on a shallow clone can misattribute the shallow-boundary commit as touching a path it never touched, because git can't diff that commit against a parent it doesn't have locally and can't prove it's unrelated — scripts/capture-doc-evidence.mjs's lastCommitEpoch()/lastCommitEpochBatch() (and capture-doc-evidence.test.mjs's own naiveEpoch sanity check) shell out to git log -1 --format=%at -- <path>-style calls. A Multica agent's default checkout is a shallow clone; the commit sitting at the shallow boundary has no locally-available parent, so git's pathspec filter includes it as a match by default rather than proving it unrelated. This surfaced as three of capture-doc-evidence.test.mjs's fm-fade-up-error-remount-gotcha.md-fixture tests failing with a plausible-looking "the doc's real history drifted, the hardcoded SHA needs refreshing" symptom (OBJ-2576) — the committed SHAs were correct all along; git fetch --unshallow alone made all 21 tests in the file pass unchanged, no fixture edit needed. Fix (shipped): the affected tests now skip, with an explicit reason, when git rev-parse --is-shallow-repository is true, instead of asserting against corrupted data; currency() now warns once to stderr on a shallow checkout, since the same corruption can silently affect lastCommit/footerBehindCommit for any real doc, not just this fixture. General rule: before trusting git log/git diff output path-limited against a Multica checkout, confirm git rev-parse --is-shallow-repository is false, or git fetch --unshallow first — a shallow clone's answer to "did commit X touch path Y" is not reliable at the boundary. (Root cause: OBJ-2576.)

Evidence images are WebP — ffmpeg cannot produce that encode

• Committed screenshot/mockup evidence is WebP, not PNG/GIF — and ffmpeg cannot do that encode, even though it's already the repo's GIF/WebM tool — ionic_frontend/scripts/desi-render.mjs and scripts/capture-mockups.mjs route captures through scripts/lib/webp-encode.mjs (cwebp/gif2webp, libwebp) automatically; a PR adding a .png/.gif under .planning/phases/** or docs/ui-specs/** over 400 KB fails scripts/check-image-size.mjs's CI gate unless carved out or allowlisted. The trap: this repo already uses ffmpeg for GIF/WebM motion evidence (scripts/lib/motion-gif.mjs), so it's a reasonable assumption that the same binary can also produce WebP — it can't. Homebrew's ffmpeg build has no libwebp encoder compiled in (Automatic encoder selection failed ... codec webp is probably disabled), confirmed on this repo's runtime. Install libwebp instead — brew install webp (macOS) / sudo apt-get install -y webp (CI) — which is what actually provides cwebp/gif2webp. If libwebp isn't installed, capture still succeeds (the PNG/GIF is kept and the reason is reported — fail-soft, not fail-closed), so a missing encoder shows up as an oversized-file CI gate failure, not a capture-time error. Since OBJ-3375, when the encoder is present, a --max-bytes byte budget (default: the gate's own threshold) absorbs most of this failure class — a still that lands over budget after the pinned default encode escalates to cwebp -size target mode automatically, so an over-sized-but-encoded capture rarely reaches the CI gate anymore. The missing-encoder case above is untouched by the budget — with no cwebp there's no WebP encode to escalate, so that path still surfaces as a gate failure exactly as described. Full reference: docs/development/evidence-image-encoding.md. (Root cause: OBJ-2385, PR #2249.)

capture-ui-evidence (CI) can't reach Capacitor-only UI branches without opting in

• preview.yml's capture-ui-evidence job runs headless Chromium against a web preview — Capacitor.isNativePlatform() is always false there, so any view's native branch is structurally unreachable unless the PR opts in — a PR that only changes a Capacitor-gated branch (native sign-up/sign-in fallback, native OAuth, a native-only Settings row) got a "successful" capture that silently showed the web sibling the whole run, with nothing in the run or the PR comment saying the evidence didn't cover the changed code. Concretely: OBJ-2580 / PR #2362's captured stills/GIF never showed the native fallback card the PR actually changed — Roy had to verify the fix by reading source instead of looking at the "evidence." The trap compounds because scripts/capture-ui-evidence.mjs has supported --native (forces Capacitor.isNativePlatform() true via the Capacitor-documented window.CapacitorCustomPlatform hook) since OBJ-1818 — the capability existed, but preview.yml's own CI job never passed the flag, so it silently went unused on every native-gated PR. Fix (shipped): a UI-Evidence-Platform: native PR-body directive (same own-line contract as UI-Evidence-Motion/-Interact) wires --native into the job; the PR comment now always states which platform was captured, even when the directive is absent, so a reviewer gets an explicit "these screenshots show the web fallback, not the native branch" note instead of inferring coverage from a bare "Captured". General rule: a PR touching a Capacitor.isNativePlatform()-gated branch needs the UI-Evidence-Platform: native directive to get real evidence for that branch — the default web capture is not a proxy for it. See Native-gated (Capacitor) surfaces in the UI-evidence skill mirror. (Root cause: OBJ-2582, discovered on OBJ-2580 / PR #2362.)

capture-ui-evidence.mjs can't capture /sign-in without --public (fail-closed by design)

• capture-ui-evidence.mjs structurally cannot target /sign-in — this is deliberate, not a gap to work around. Every capture always ran login() first and then assertAuthenticated() after, and assertAuthenticated() throws on any URL matching /sign-in by design — that check is the negative proof auth didn't silently fail, the same fail-closed contract that makes every other route's evidence trustworthy. So no amount of local setup (demo credentials, a live backend) gets a real login and a /sign-in capture in the same run — the two are mutually exclusive by construction. Discovered on OBJ-2497 (sign-in password affordances) after two genuine attempts: a local dev-server run failed at the login step (no demo credentials/backend in that workdir), and standing up the full stack wouldn't have helped either, since the fail-close fires on the destination URL, not on a login failure.
• capture-public-pages.mjs is not a substitute — it's a bespoke docs-asset script hardcoded to /u/:username and /public_goal/:id, has no CLI flags, writes to docs/assets/, and isn't wired into preview.yml's PR-evidence gate. Extending it produces screenshots that never reach the check the sign-in work actually needed to satisfy.
• Fix (OBJ-2505): capture-ui-evidence.mjs --public inverts the fail-closed check instead of relaxing it. Under --public, login()/assertAuthenticated() are skipped entirely (and the run refuses to proceed if STORAGE_STATE is also set — the two are mutually exclusive), and every capture is instead gated by assertPublicRoute(): a pre-flight allowlist rejection — scripts/lib/public-routes.mjs's PUBLIC_ROUTE_PATTERNS, the 7 routes carrying meta: { public: true } in ionic_frontend/src/router/index.ts (/sign-in, /sign-up, /sso-callback, /public_goal/:goalId, /invite/:token, /join-team/:code, /u/:username) — before the browser ever launches, plus a post-navigation final-path-equality check and a per-route DOM landmark check (/sign-in's landmark is .cl-rootBox, deliberately not h1, since Clerk's loading/timeout card shares the ready state's <h1> — see the selector's comment in public-routes.mjs). --public --routes /dashboard still fails closed exactly as before: the allowlist rejects it before anything is captured. In CI, a UI-Evidence-Public: /sign-in PR-body line (same own-line contract as UI-Evidence-Routes/-Motion/-Interact/-Platform) wires a second, unauthenticated pass into preview.yml's existing Capture UI evidence step, alongside the authed pass; a route named in both directives is captured once via the public pass and dropped from the authed one, with a ::warning::.
• Don't "simplify" this by relaxing assertAuthenticated instead — that was already proposed and rejected once (the script's header documents the rejected --fail-on-unauth history). --public is an inversion of the fail-close, not a bypass of it, and the allowlist is intentionally hand-maintained against the router rather than derived from the URL alone: public-routes.test.mjs includes a drift test that parses ionic_frontend/src/router/index.ts directly and fails if PUBLIC_ROUTE_PATTERNS and the router's meta: { public: true } routes disagree — this is deliberately not /design-system or /welcome, both in the separate NAVLESS_ROUTE_PATTERNS list but still authenticated/onboarding routes; conflating navless with public is exactly how an authed route could slip into an unauthenticated capture. See Native-gated (Capacitor) surfaces in the UI-evidence skill mirror for the sibling UI-Evidence-Platform: native directive --public composes with. (OBJ-2497 discovery, OBJ-2505 fix, PR #2368.)

<ion-modal> DOM presence ≠ open state

• Ionic keeps <ion-modal>-wrapped elements in the DOM at all times, regardless of is-open — a plain document.querySelector against an Ionic modal wrapper class cannot distinguish "closed" from "shown." Ionic renders <ion-modal> overlays into the DOM on mount and toggles visibility internally rather than removing/re-adding the element, so any diagnostic, test, or assertion that infers "is this surface shown right now" purely from DOM presence gets a false positive on an Ionic modal that exists closed, same as it would if the modal had actually won arbitration and opened. Found live while diagnosing OBJ-3111's capture-ui-evidence interstitial-arbitration failure (see OBJ-3111 comment 44a534f0-24d2-404e-8615-dcb39a8f50dd, §4/§7): a DOM-presence sweep over the app's interstitial-class surfaces reported pushPermissionPrompt, teamPulseSurvey, and badgeModal as "mounted" on a completely ordinary authenticated dashboard load — all three are <ion-modal>-wrapped and were simply present-but-closed, not evidence any of them won the arbitration frame. Contrast case: surfaces gated with v-if (streakRepairCard, featureTour, installPwaPrompt, allyRequestToast, screenTimeNudge) are genuinely absent from the DOM when hidden, so a presence check is trustworthy for those — the trap is specific to the <ion-modal> wrapper shape. General rule: to check whether an <ion-modal>-wrapped surface is actually visible, check its open/presentation state (e.g. the modal's own is-open prop value or the Ionic overlay API), never bare document.querySelector presence. (Discovered live: OBJ-3111, comment 44a534f0-24d2-404e-8615-dcb39a8f50dd.)

The Preview UI-evidence gate proves the frontend believes a query resolves, never that it actually does

• preview.yml never deploys a PR's own backend — every Preview GraphQL call hits whatever's currently live on shared enkidu-api-staging, not the PR's schema. The "Build Ionic Frontend" step's VITE_API_URL points at PREVIEW_API_HOST: staging.api.objectuve.com (preview.yml:96), and the auto-posted PR comment states this explicitly: **API:** https://staging.api.objectuve.com (shared staging) (preview.yml:283). The capture-ui-evidence job's own header comment names why this exists and what it's for: Multica agents have no local backend to capture live screenshots against, so CI captures the evidence itself "against the live Firebase preview (which already runs against the real staging API — see the 'Build Ionic Frontend' step's VITE_API_URL above) ... This is evidence for a human/agent review loop, not a correctness gate" (preview.yml:507-517). What this means in practice: a PR that adds or changes a GraphQL field/type gets a green "Capture UI Evidence (Preview)" check the instant the frontend code believes the query is well-formed and renders — that check cannot and does not prove the new field actually resolves on a real backend, because the backend it ran against was never rebuilt from this PR's code. A schema-changing PR needs its own verification (a local docker-compose run against the PR's own rails_api, or waiting for the staging deploy after merge) — the Preview evidence gate is a UI-rendering aid, not a schema-compatibility gate. Riley/Roy/Vicki should calibrate what a green Preview evidence check promises accordingly before treating it as proof a schema change works end-to-end. See Playwright Smoke Gate for the same "advisory, not correctness" caveat applied to the smoke suite running against the same shared-backend Preview channel. (OBJ-2348)

Storybook evidence capture: cold Chromium launch can return a uniform blank first screenshot

• The first page.screenshot() after a cold Chromium launch can come back a uniform blank PNG despite the DOM and computed styles being correct — scripts/capture-storybook-evidence.mjs hit this on the very first capture of an independent run twice: on PR #2747 (v4.49 Phase 1, default--light--mobile, ink125=0.00000, a hard check-png-legibility.mjs FAIL) and again on Roy's first independent Phase 2 run. Every CI run launches a fresh Chromium, so this is a flake CI hits on every run, not an occasional one. Fix (shipped, v4.49 Phase 2, CI-6): captureWithColdStartRetry() re-screenshots and compares, scoped to the first capture of a run only (not a blanket warm-up navigation, which would pay the cost on every run whether or not the flake fires) — "blank" is decided narrowly (isUniformPixelData: every pixel byte-identical), distinct from check-png-legibility.mjs's broader near-invisible-contrast judgment, which stays wired in as an independent backstop and is not replaced by this mitigation. The retry fires at most once: a first capture that comes back uniform gets exactly one re-screenshot, judged on its own — if that one is still uniform, the capture fails loudly rather than being silently accepted, so a story that is genuinely blank is never retried into a false pass. See isUniformPixelData/isBlankScreenshot/captureWithColdStartRetry in scripts/capture-storybook-evidence.mjs. General rule: any new Playwright-based capture script that launches a fresh browser per run should budget for this same flake — a bare "did the screenshot succeed" check is not enough, since a screenshot can succeed and still be uniformly blank. (Root cause: v4.49 Phase 1/OBJ-3079, mitigation: Phase 2/OBJ-3115.)
• A related but distinct defect — captured light and dark screenshots coming back byte-identical (not blank) — is tracked separately as OBJ-3127 and is not part of this cold-start flake; do not conflate the two. assertModesDiffer() is the independent guard that catches the byte-identical case: it refuses to write one theme's render under the other's filename and fails the job instead of reporting a false pass. Root cause (fixed, PR #2761): storybook-evidence.yml's "Serve built Storybook" step ran npx serve storybook-static with no config; serve's default cleanUrls: true 301-redirected /iframe.html?id=<story>&globals=theme:<mode> to a bare /iframe, dropping the query string entirely — every capture rendered Storybook's "no story selected" placeholder at the theme's default value for both light and dark, which is why they came back byte-identical. Not a theme-forcing timing race, despite the symptom looking like one. Fixed by serving with cleanUrls: false (ionic_frontend/.storybook/serve.json, wired via serve -c "$PWD/.storybook/serve.json").

<ion-refresher> MD-mode async bootstrap races Storybook's next-story teardown

• <ion-refresher>'s MD-mode setupMDNativeRefresher() is a fire-and-forget async chain — await import(...) then createGesture({ el: this.scrollEl }) → enable() — and Storybook's portable-stories vitest runtime doesn't unmount story N until the start of story N+1's own test run, not in any afterEach. .storybook/preview.ts's app.use(IonicVue) sets no mode, so headless desktop Chromium resolves to md and this native-refresher path runs on every mount. When the dynamic import was still pending at the moment the next story's implicit unmount disconnected scrollEl, enable() fired against an already-disconnected element and threw TypeError: Cannot read properties of undefined (reading '__zone_symbol__addEventListener') inside @ionic/core. This is not a concurrency problem — Tess/Codi reproduced it in isolated single-file runs (NotificationHistory.stories.ts alone failed 4 of 5 standalone repeats), so worker count and file-level sharding were never viable mitigations; the race is between two stories in the same file. The failing test name is a misattribution, not a lead: HARNESS-FIX-6's afterEach rejection-attributor (ionic_frontend/.storybook/vitest.setup.ts) blames whichever story is executing when the rejection actually fires, which is story N+1, not the story that triggered it — this is why the named story rotated run to run (Error, Loading, Empty No History, Populated (2 allies), …) with no correlation to its own fixture data. Exposed surface: the 5 story files whose .vue renders <ion-refresher> — AlliesPage.stories.ts, NotificationHistory.stories.ts, PendingAllyRequestsPage.stories.ts, ActivityFeed.stories.ts, Dashboard.stories.ts — the last two hadn't failed yet when this was diagnosed, which was luck, not immunity. No production evidence: Sentry (objectuve/objectuve-client, 90d) had zero issues matching this stack while the app was reporting healthily overall — production route changes don't unmount inside the dynamic-import window the way story-to-story teardown does, so this is a test-harness race, not a latent production crash. Fix (shipped): flushPendingRefresherBootstrap() (ionic_frontend/.storybook/vitest.setup.ts:74-85), called from the shared afterEach (:144-146) before HARNESS-FIX-6's rejection-attribution check runs. No-op unless the test's DOM has an <ion-refresher> element; when it does, polls (40 × 40ms, bailing early once a rejection lands) so the pending bootstrap chain gets time to resolve against a still-connected element before the next story's implicit unmount — an empirical spike confirmed this file's afterEach runs before Storybook's own next-story unmount, so draining here is reachable. HARNESS-FIX-6's attributor logic itself is untouched; the fix drains before that check runs rather than weakening it. Rejected alternatives: a scoped mode change was unnecessary once the drain held, and a blanket app.use(IonicVue, { mode: 'ios' }) was out of scope regardless (re-renders all 434 story files in iOS mode, silently changing every existing visual assertion); a patch-package guard on @ionic/core was ranked last and also unneeded, since the zero-Sentry-evidence finding meant there was no production bug to justify a runtime patch. Runtime tax fixed (OBJ-3259, PR #2871): flushPendingRefresherBootstrap() now polls the refresher's own terminal bootstrap property (gesture on the MD/fallback path, scrollListenerCallback on the iOS path) and exits as soon as every rendered <ion-refresher> reports bootstrapped, instead of always sleeping the full 40×40ms budget on every green run. A green run now typically exits in 1-5 iterations (40-200ms) — verified against a live run (12/12 AlliesPage stories, all exited early, refresher still connected every time); the 40-iteration ceiling stays as the fallback for contended full-suite runs and for any refresher that never bootstraps at all. Caveat fixed (OBJ-3308, PR #2898, commit 0e65db348): the bootstrapped check was originally 'gesture' in el || 'scrollListenerCallback' in el — an in presence check, not a truthiness check. @ionic/core's Refresher#disconnectedCallback assigns this.gesture = undefined on teardown rather than deleting the key, so a refresher that disconnects and reconnects within the same test before its new bootstrap chain resolves would read as already-bootstrapped from the stale key, letting the poll exit before the current mount's real bootstrap finishes — reopening this race silently for that specific case. Not hit by the story set at the time (single mount per test assumed by both Tess's 11/11 verification runs and the code comment's own empirical check). Swapped to a truthiness check (!!(el as any).gesture || !!(el as any).scrollListenerCallback, ionic_frontend/.storybook/vitest.setup.ts:84) as a pre-req for Phase 3's storybook-test job, which is not yet wired into ci.yml; refresher-rejection count confirmed at 0 across the same 3 consecutive clean full-suite runs used to verify OBJ-3308's own fix. General rule: any Ionic component with an async native-bootstrap chain reading an element ref after an await is a candidate for this same race in a harness that defers unmount to the next test's setup — check whether the component's connect/disconnect lifecycle assumes synchronous bootstrap before trusting a story-to-story teardown sequence. Two other flakes in the same async-teardown-race family were observed at the time and tracked separately — both since fixed, also by OBJ-3259/PR #2871: the Tooltip.stories.ts timing race (decoupled the 350ms regression bound from the CSS fade by asserting aria-expanded, set synchronously by UiTooltip's isVisible watcher, instead of racing toBeVisible() against the fade) and the CollectiveGoalOptInModal.stories.ts > Dark Mode ion-modal dismiss-teardown race (removeViewFromDom/insertBefore on null — see HARNESS-FIX-8 below for the full root cause and fix). (Root cause/fix: OBJ-3183, PR #2836.)

<ion-modal isOpen> with no play() toggle can crash on Storybook's next-story teardown

• Any <ion-modal> rendered with isOpen="true" and no play() step to toggle it (84 stories across this repo) defers its real DOM-teleporting present() to the next animation frame — not story-specific, Stencil's own componentDidLoad: if (this.isOpen === true) raf(() => this.present()). If a story's test finishes before that frame fires, or before present() itself resolves, the modal is still in a teardown-vulnerable state when the next test's setup removes this test's DOM root. <ion-modal> tries to self-heal from an unexpectedly-removed parent via its own parentRemovalObserver (a MutationObserver watching document.body), calling this.dismiss(undefined, 'parent-removed') — but by the time that fires, the DOM node its own CoreDelegate needs to restore position (Reference, the placeholder comment present() left behind at the modal's original spot) has already been detached along with the rest of the removed subtree, so Reference.parentNode.insertBefore(...) throws TypeError: Cannot read properties of null (reading 'insertBefore') (@ionic/core's utils/framework-delegate.js, CoreDelegate#removeViewFromDom). Reproduced live against CollectiveGoalOptInModal.stories.ts's Dark Mode — same async-teardown-race family as the <ion-refresher> bootstrap race above, a different component (ion-modal's overlay controller, not ion-refresher), and independently ruled out against fa5ca1574 (#2834, a pure Tailwind class change to the same file the same day, no DOM-lifecycle code touched).
• Fix (shipped, OBJ-3259, PR #2871): settleOpenModals() (ionic_frontend/.storybook/vitest.setup.ts), called from the shared afterEach before flushPendingRefresherBootstrap(). Skips any <ion-modal> whose isOpen is falsy — it was never scheduled to present, so presented would never flip and the loop would otherwise burn its full budget for nothing. For the rest, polls presented (40 × 40ms ceiling — same remedy shape as the refresher fix above) and, once true, calls modal.dismiss(undefined, 'handler'). presented only flips true once attachComponent/attachViewToDom has already resolved (verified by reading modal.js's own present() sequencing), so polling it is a safe "attach completed" signal; dismissing while the modal's original DOM position is still intact both avoids the crash and disconnects parentRemovalObserver, so the later container removal never triggers a second, racing dismiss at all. The 'handler' role is deliberate: dismiss() special-cases it to skip the component's own :can-dismiss check entirely (if (role !== 'handler' && !(await this.checkCanDismiss(...)))) — a blocking canDismiss (e.g. CollectiveGoalOptInModal's own !props.loading || !props.isOpen, which the Submitting story trips with loading: true) is a UX policy for real user dismiss attempts, irrelevant to this harness-safety teardown, which tears the component down regardless of what canDismiss would say.
• General rule: same as the <ion-refresher> race above — any Ionic overlay component with an async native-presentation chain deferred past the current DOM lifecycle is a candidate for this race in a harness that defers unmount to the next test's setup. (Root cause/fix: OBJ-3259, PR #2871.)

A Storybook story can lie about what it renders — play() is now CI-enforced

• A Storybook story can assert nothing about what it actually renders, and nothing outside a human opening the built/dev Storybook ever catches it. FeatureTour.stories.ts's Outro/CompletionMoment stories both shipped named after the state they claimed to render (see the single-element steps array gotcha above) while silently showing the wrong one — the play() functions added to fix that only ran in Storybook's own interactions panel, never in CI, because @storybook/addon-vitest's storybookTest() plugin wasn't wired into any Vitest config (OBJ-2919, tracked as a follow-up under OBJ-2969).
• Shipped (v4.46 Phase 3, OBJ-3390): a dedicated ionic_frontend/vitest.storybook.config.ts wires storybookTest() into a standalone Vitest browser-mode (headless Chromium) run, and CI's storybook-test job (.github/workflows/ci.yml) makes npm run test:storybook a required check with no continue-on-error. Every story's play() — and every story's mere render, whether or not it defines one — now executes on every PR. See docs/development/testing.md § Storybook for the coverage policy (render-only stories are legitimate smoke coverage; a story claiming a specific state needs a play() that asserts it) and the Tier C anti-pattern (an empty or optional-chained play() reporting green while asserting nothing).
• General rule: a play() function with no CI wiring is a promise, not a gate — it only protects against regressions a human happens to notice in an open Storybook tab. Verify storybookTest() is actually wired into a Vitest config (not just present in package.json/addons) before treating any repo's play() functions as enforced. (Root cause: OBJ-2919, PR #2667; enforcement shipped: OBJ-3390.)

storybookTest() does not load the root vite.config.ts — only .storybook/main.ts's viteFinal

• A real storybook dev/build-storybook run merges the root ionic_frontend/vite.config.ts in via @storybook/builder-vite's loadConfigFromFile, but @storybook/addon-vitest's storybookTest() vitest-plugin pipeline never loads that file at all — it only resolves .storybook/main.ts's own viteFinal. Anything a story needs that only the root config supplies has to be redeclared in vitest.storybook.config.ts by hand: vue() itself (the vue3-vite framework preset only adds docgen plugins, assuming the project's own Vite config supplies vue() — true for a real Storybook build, not for storybookTest()), resolve.alias/dedupe (the @ → src/ alias every @/... import depends on, plus the dedupe list required for Symbol-keyed inject/provide to resolve through one module graph), and define.__APP_VERSION__ (read directly by components like SideMenu.vue; without it, every story that mounts one throws ReferenceError: __APP_VERSION__ is not defined at setup()). Each was confirmed necessary by a live failure during v4.46 Phase 1 harness bring-up, not spec'd in advance.
• General rule: a Vite alias, define, or plugin added to the root vite.config.ts does not reach a Storybook play()/story test — it has to be redeclared in vitest.storybook.config.ts separately, or the story breaks with no signal pointing back at the config that actually changed. Check vitest.storybook.config.ts's own header comment (ionic_frontend/vitest.storybook.config.ts:10-40) before assuming a root Vite config change is automatically picked up there. (v4.46 Phase 1 deviation 1.)

storybook-test's 4-way sharding cuts variable per-story disk cost, not the fixed per-job cost — disk exhaustion kept recurring and misread as a story flake

• Failed to fetch dynamically imported module: .../setup-file.js (or .../vitest.setup.ts, or any in-flight story file) is the symptom Vite's dev server throws when the runner's disk fills up mid-run — not evidence of a real story or app regression. OBJ-3390 D3 first reproduced this as an outright No space left on device kill on the single-job (unsharded) version of storybook-test and mitigated it with 4-way file-list sharding (ionic_frontend/scripts/partition-storybook-files.mjs), reasoning that spreading 429 story files across 4 jobs would cut each job's disk/Vite-cache footprint to ~1/4. It recurred anyway (OBJ-3480) across unrelated PRs and unrelated story files — always near the tail of an otherwise-healthy shard run (57–76s into a ~75s run), with whatever file(s) happened to be mid-load at that instant failing import, because sharding only reduces the variable cost (each shard's own slice of story-file Vite-cache growth) — it never touched the fixed per-job cost (npm ci + the Playwright Chromium download+cache), which every shard pays in full regardless of how few story files it's assigned. One occurrence (PR #2958, shard 3/4) confirmed the mechanism directly: the job's own "No space left on device" error surfaced through an unrelated write — the runner's own diagnostics-log page — logged immediately after the test failures, in the same job.
• Fix (shipped, OBJ-3480): a "Free disk space" step (.github/workflows/ci.yml, storybook-test job) runs before npm ci, reclaiming space from large preinstalled toolchains the job never uses (/usr/share/dotnet, the Android SDK, GHC/ghcup, Boost, CodeQL) — the same fix already proven for this exact failure class on this runner image in mobile-android-boot-smoke.yml's "Free disk space for Android emulator" step. A df -h / diagnostic runs both before/after cleanup and again (if: always()) right after the test step, so a third occurrence is a one-line log read instead of another full log-archaeology pass through an unrelated write failure to find the real cause.
• General rule: file-list/test-count sharding only scales down a job's variable cost. If a CI job's fixed setup cost (dependency install, browser/tool download) is itself a meaningful fraction of the runner's disk budget, sharding narrower can reduce — but never eliminate — the risk of tipping over; reclaiming unused preinstalled toolchain space (/usr/share/dotnet, /usr/local/lib/android, /opt/ghc, /opt/hostedtoolcache/CodeQL, etc.) is the actual lever for the fixed-cost side, and a plain df -h before/after is worth adding proactively to any disk-heavy job rather than waiting for the failure to surface a number by accident. (Root cause: OBJ-3390 D3, OBJ-3480. Fix: OBJ-3480.)

An unhandled error inside a DOM event callback is invisible to unhandledrejection-based attribution, but still exits non-zero

• HARNESS-FIX-6's afterEach attributor (ionic_frontend/.storybook/vitest.setup.ts) only listens for window's unhandledrejection event — a TypeError thrown synchronously inside a native DOM event callback (e.g. an <img> element's leaked load listener) is not a promise rejection, so it fires with no "current story" to attach to. ImageCropper.stories.ts hit this for real: vue-advanced-cropper renders two independent real-network <img> elements per story, each owned by an internal subcomponent that leaks a native load listener with no matching removeEventListener on unmount; under host contention the listener can fire after Storybook has already torn the story down for the next one, throwing an unattributed error that Vitest can't pin on any test. The result is confusing on first encounter: every individual test in the run still reports "passed," there is no named failing file, and yet the suite-level process still exits non-zero — the failure is real, just invisible to the same-file-name signal every other harness failure gives you.
• Fix (shipped, OBJ-3308, PR #2898): a play() added to the affected story waits for every rendered <img> to report a real, non-empty src and a completed load before the story's own teardown can race the leaked listener — closing the async gap rather than trying to attribute the error after the fact. HARNESS-FIX-6's attributor itself is untouched; this is the same "drain before the next test's teardown can race it" shape as HARNESS-FIX-7/8 above, applied to a DOM-event source instead of a promise rejection.
• General rule: a suite-level non-zero exit with zero individually-failing tests is not proof the harness has a bug — check for an unhandled error from a non-promise source (a DOM event callback, a setTimeout callback, a leaked listener) before assuming the attribution mechanism itself is broken. window.addEventListener('unhandledrejection', ...) alone does not see it; vitest's own unhandled-error reporting is the backstop, same as the existing timing-window gap this file's header comment already documents. (Root cause/fix: OBJ-3308, PR #2898.)

VitePress ignoreDeadLinks regex tests the raw link string, not the source file's directory

• A VitePress ignoreDeadLinks regex entry is tested against the raw relative link string as authored in the markdown source — e.g. ./mockups/auth-screens-mobile-fit/README — never against the resolved route or anything containing the source file's own directory, so a pattern anchored on the containing directory can never match. docs_site/.vitepress/config.ts suppresses 65 intentionally-dead ./mockups/<name>/README links across 28 files under docs/ui-specs/, left dangling after PR #2891 (OBJ-3192) untracked docs/ui-specs/mockups/ (529 files removed from git). Code review asked to tighten the ignore pattern from /\/mockups\/.*\/README$/ to /\/ui-specs\/mockups\/.*\/README$/, reasoning that scoping the match to /ui-specs/ would still let an unrelated broken .planning/**/mockups/*/README link fail the build. Applying that change reintroduced Error: 65 dead link(s) found. — /ui-specs/ never appears in the string VitePress tests, even though every one of the 28 source files genuinely lives in docs/ui-specs/. The build output makes the split visible: (!) Found dead link ./mockups/auth-screens-mobile-fit/README in file /Users/.../docs/ui-specs/auth-screens-mobile-fit.md:15 — the link and its containing file are reported as two separate values, and only the first is matched against. Fix (shipped): /^\.\/mockups\/[^/]+\/README$/ — anchored on the leading ./ and a single path segment. This is strictly tighter than the original unanchored pattern (it can't match an absolute path or a nested path), but it genuinely cannot be scoped by source directory — a broken ./mockups/<x>/README link authored in some other directory would also be silently suppressed. That's the accepted limit of what the API can express, not an oversight. General rule: before trying to narrow a VitePress ignoreDeadLinks regex by the source file's directory, confirm what string is actually being matched (the raw authored link, not the resolved route or file path) — the build's own dead-link error output shows both values on the same line for exactly this reason. (Root cause: OBJ-3192, PR #2891.)

A name-convention spec check can't see a consolidated spec file — specCoverage() flagged real coverage as missing

• scripts/capture-arch-evidence.mjs's specCoverage() only checked name-convention candidate paths (app/graphql/mutations/<ns>/<name>.rb → spec/{requests,graphql}/mutations/<ns>/<name>_spec.rb), so any mutation whose test lives in a spec file that consolidates several mutations by feature area — instead of mirroring each mutation file 1:1 — read as "missing spec" even though it has real coverage. Confirmed in two real domains: social (7 ally-request/lifecycle mutations — accept_ally_request, block_ally, decline_ally_request, invite_ally_to_community, remove_ally, send_ally_request, unblock_ally — flagged during the v4.3 sweep per .planning/milestones/v4.3-keystone-arch-sweep-AUDIT-FINDINGS.md §3/§7, never actioned until now) and integrations (9 of its 10 GraphQL mutations, all really tested in rails_api/spec/requests/mutations/integrations/connect_mutations_spec.rb and habit_integration_mapping_mutations_spec.rb — found scoring the domain for the first time, OBJ-3511). Fix (shipped, OBJ-3519): specCoverage() now checks the name-convention candidate path first and, only when none exists, falls back to a word-boundary-anchored search of the *_spec.rb files already sitting in the mutation's candidate spec directories (plus the confirmed spec/requests/graphql/ consolidation convention) for its camelCase GraphQL field name as a whole word. Every covered interaction/mutation is now recorded with matchedBy: 'path' | 'content' in evidence.json's new specCoverageCovered array, and evidence.md's "Spec coverage" section reports the two counts separately — content-fallback coverage is real, but worth knowing which mutations rely on it rather than a direct file match. Residual limitation: the content fallback only proves the mutation's GraphQL field name appears as a whole word somewhere in a candidate spec file — a spec that genuinely exercises the mutation without ever typing its field name literally still reads as missing. Narrower than the original bug; not a fully general fix. (Root cause: v4.3 sweep (social), OBJ-3511 (integrations, first full score). Fix: OBJ-3519.)

BM25 length normalization loses a short answer-dense chunk on count/table-lookup questions

• BM25's length normalization can rank a short, answer-dense chunk below several much longer chunks that repeat the same generic query vocabulary, keeping the correct answer out of a small top-k even though a larger k reaches it. Found by OBJ-3612's docs-agent retrieval spike (scripts/docs-agent-spike.mjs --corpus docs) on two GraphQL field-count questions ("how many root mutation fields", "how many root query fields"): docs/architecture/graphql/index.md chunks (by heading) into a short ## Contents chunk holding the literal count, plus two much longer category-enumeration chunks. At k=5, BM25 ranked the short chunk outside the retrieved set on both questions (recall@5 = 90.9%, 20/22 — these are the two misses); at k=15 it was inside the retrieved set on both (recall@15 = 100%, 22/22), reproduced identically on a repeat run. This is a retriever-ranking weakness specific to count/table-lookup questions phrased with generic terms — not a chunking defect; the heading-based chunker isolated the answer chunk correctly in both cases. Flagged as inference: the harness doesn't log which competing chunks displaced the answer chunk from the top-k, so the exact ranking margin is unconfirmed.
• ✅ Fixed (v4.55 Phase 5, OBJ-3622), not accepted. DocsCorpusIndexer (rails_api/app/services/docs_corpus_indexer.rb) prefixes each chunk's tokens with its document's H1 title, repeated (TITLE_BOOST_REPEAT = 2) — enough to disambiguate which document a short chunk belongs to without drowning out the chunk's own content. This resolved both known misses with no collateral regression: recall@5 went 90.9% → 100% (22/22), tested against k=5/10/15 and BM25 b-parameter alternatives. The CI regression check (promoted from the spike's own eval set) asserts this floor live and un-stubbed, so a future regression genuinely fails the build rather than silently reverting. If a similar count/table-lookup miss shows up elsewhere: a title/heading boost is the first thing to try before reaching for a larger k or a retrieval-architecture change. Full findings and go/no-go: Embedding an AI knowledge-retrieval assistant into the docs sites.

A verification step dispatched inside a just-merged dependency's own deploy window looks like a pipeline gap but is a dispatch-timing race

• A phase's first live-verification probe against a just-merged dependency can fail for a reason that looks like a pipeline gap (a missing GraphQL field, a route 404) but is actually a race against that dependency's own still-running staging deploy. Found in v4.55 Phase 4 (OBJ-3624): Codi's first live probe of askGuideQuestion found it missing from staging's GraphQL schema entirely. It looked like a deploy pipeline had silently skipped shipping the schema change. Dave diagnosed it as a dispatch-timing race instead — the probe landed inside Phase 3's own still-running ~10–45 minute staging deploy window (merge → migrate → deploy-backend → smoke), before that deploy had actually reached the revision serving the new field. Once the deploy finished, the field was there. Distinguishing the two: a real pipeline gap persists after the dependency's deploy workflow shows a completed, green run for the merge commit; a dispatch-timing race resolves itself once that run finishes — check gh run list/gh run view for the dependency's merge commit before concluding the pipeline dropped something. General rule: a verification step scheduled immediately after a dependency merges should either wait for that dependency's own deploy workflow to reach a terminal state first, or be prepared to treat an early failure as inconclusive rather than diagnostic.

A surface gated by both a feature flag and an interaction can't be evidenced by a bare route capture

• A route capture only ever proves one data/interaction state per route per run — a surface reachable exclusively through both a feature-flag gate and a user interaction (e.g. a modal that only opens once a flag is on) has no single UI-Evidence-Routes/UI-Evidence-Flag-Overrides combination that reaches it, and the naive fix (a second capture of the same route) used to clobber the first, writing over the same <route-slug>--<mode>--<viewport>.png path. OBJ-2516 hit this directly: un-gating CreateCollectiveGoalModal (interaction-triggered open state) and CollectiveProgressCard (a plain landing state), both under /team-home, needed two different pieces of evidence from one PR. Only one was capturable per CI run; the other fell back to Desi's static mockup stills via a Maggie waiver — real evidence, but not live-rendered-code evidence, and the round cost real CI cycles to discover. Fix (shipped, OBJ-2520): reach for the tool that matches which half of the state is gated, instead of spending CI rounds rediscovering the gap:
  • Component state, no live route reaches it at all — UI-Evidence-Stories (shipped v4.49, see Isolated component evidence in the UI-evidence skill). No login, no auth, no GraphQL, no Capacitor branch — captures straight off a built Storybook.
  • Data state a route can reach but only via live backend seeding — UI-Evidence-Mock-Graphql: <fixture>.json (see the same skill). Drives the route off a deterministic fixture instead of live staging data, sidestepping smoke-account provisioning, active-team seeding, and flag-override targeting entirely.
  • Interaction state layered on the same route as its own landing state — a labeled UI-Evidence-Interact: <route> <label> <steps> entry (see the same skill). Adds an additional capture (<route-slug>--<label>--<mode>--<viewport>.png) alongside the route's bare one instead of overwriting it — this is what actually closes the OBJ-2516 clobber.
  • A route reachable at all — even by adjusting UI-Evidence-Flag-Overrides or UI-Evidence-Mock-Graphql — should still use UI-Evidence-Routes; UI-Evidence-Stories is for the residual case a route structurally can't reach. General rule: before falling back to a static mockup waiver on a flag-and-interaction-gated surface, check whether the gap is a component state (Stories), a data state (Mock-Graphql), or an interaction state on an already-reachable route (labeled Interact) — all three now have a first-class CI directive, and only a genuine capture-infrastructure failure still warrants a waiver. (Root cause: OBJ-2516, PR #2336-era evidence-gate rounds. Fix: OBJ-2520.)

Local Development & Docker

Docker naming: frontend service, ionic_frontend/ dir

• Docker naming — docker-compose.yml uses frontend as the service name; the source directory is ionic_frontend/.

Redis in Docker: redis:7-alpine on 6379

• Redis in Docker — redis:7-alpine on port 6379, accessed via REDIS_URL=redis://redis:6379/1.

Last updated: 2026-09-20 (OBJ-4006: added the deferred-reload-races-reconnect-flow gotcha, amended the OBJ-4047 entry (Frontend/Vue-Ionic))

Prior update: 2026-09-19 (OBJ-3953: added the setOffline-vs-WebSocket gotcha (Testing & Smoke Suites) and corrected the ActionCable ConnectionMonitor entry's superseded causal claim)