Smoke-account concurrency gotcha

Concurrent CI smoke suites racing a shared demo account produces intermittent, hard-to-attribute UI assertion failures

Applies to: Any CI workflow that authenticates against a shared, persistent, mutable seed account (fallback-auth or otherwise) from more than one workflow trigger

Post-mortem for: OBJ-1150 — staging Playwright smoke intermittently hard-failed getByText("Today's practice") (Dashboard) and getByText("My Goals") (Goals list) in 03b-habit-create, across five fix attempts before the real root cause was found.

The symptom

TodayActions.vue's <h2>Today's practice</h2> is unconditional inside its <section> — it renders whenever TodayActions mounts at all. So a getByText("Today's practice") miss means Dashboard.vue fell into a different render branch (no-goals / new-goal / mobile streamlined), which only happens when the account's active-goal set looks different than the test expects at that instant. Same story for "My Goals" on the goals-list page. Both symptoms trace back to the shared staging demo account (demo@objectuve.com) having a different goal/habit state than the spec assumed — because another smoke run was mutating it at the same time.

Root cause

journey.spec.ts mutates the shared, persistent demo@objectuve.com account on every invocation (goal/habit create, check-ins, onboarding reset, teardown). Every Playwright smoke suite hitting staging.api.objectuve.com — the Staging Deployment's own post-merge suite, and every open PR's Preview Deployment suite — authenticated against that same account, and .github/workflows/playwright-smoke.yml had zero concurrency: block. Two suites overlapping left the account in a state neither test expected.

Confirmed by cross-referencing every Staging smoke run's job window against every other Staging/Preview smoke job window: every reproduction of the tracked symptom had at least one concurrent overlapping smoke run; the one clean run in the comparison set had none.

Lesson 1 — five fix rounds targeted the wrong layer before the real defect was found

Three separate fix attempts (six CI-fix-loop rounds total) treated this as a single-request cost problem and each looked clean for a run or two before reproducing the identical symptom:

1. PR #1276 — warmed the Sidekiq/Redis dependency path and goalKinds, pinned the smoke checkout to the deployed commit. Merged, CI-green twice, hard-failed on its first live fresh-deploy watch run.
2. PR #1278 — replaced a hand-mirrored warmup query with real in-suite browser navigation (the existing smoke.warmup.ts project), on the theory that a mirror query had silently drifted from the real GOALS_QUERY. Genuinely reduced the failure rate — two clean watch runs — then reproduced on a third.
3. PR #1284 — fixed a real N+1 in Resolvers::GoalQueries#goals and hardened ClearDemoDataJob's residue purge. Also a real, worthwhile fix. Also did not close the gap.

Each of these shipped a correct fix for a real problem it found — but none of them touched the actual defect, because the defect wasn't in any single request's cost. The failure signature moving between browser projects across attempts (chromium+mobile-safari on one watch, mobile-safari-only on the next) was the tell: that pattern reads as request-level variance under concurrent load, not a deterministic per-request query cost. Once a fresh diagnosis targeted "do two smoke suites ever touch the same account at once?" instead of "is this one request too slow?", the correlation was 1:1 on the first pass.

Takeaway: when a fix that passed CI and looked clean for a run or two keeps reproducing the same symptom after shipping, that's a signal the fix is treating a symptom of contention (slow response, stale cache) rather than the contention itself. Check whether the failing surface is shared, mutable state touched by more than one concurrent actor before spending another round on request-level cost.

Lesson 2 — prefer the platform's native primitive over hand-rolling synchronization

Once the concurrency root cause was confirmed, the first fix (PR #1299) serialized access with a hand-rolled bash lock: a step in playwright-smoke.yml that polled gh run list and waited for any earlier in-progress run to clear before proceeding. This was chosen over GitHub Actions' native concurrency: group specifically to avoid the group's cancellation semantics from killing a queued Staging fresh-deploy run — a real risk at the time, since Staging and Preview still shared one account and a shared concurrency: group could let a Preview push cancel the Staging run needed for the closure-bar evidence.

The hand-rolled lock produced three distinct, real, correctly-diagnosed-and-fixed bugs across three review/CI rounds — hitting the crew's 3-round fix cap without shipping:

1. Timeout — the original 60-minute bound was too short for this repo's actual concurrent-PR churn (~1 new smoke run every 10–20 minutes).
2. Livelock — extending the timeout wasn't enough: the "blocker" could rotate to a fresher arrival before the original wait converged, so the wait could run indefinitely under sustained churn. Fixed by making it a monotonic-run-id FCFS queue (only wait on runs that started strictly earlier).
3. TOCTOU — the FCFS check filtered on --status in_progress, missing an older run that was still queued (created, not yet runner-assigned) at query time — a window where two runs could both proceed unlogged, defeating the lock's own "loud, attributable collision" guarantee.

A fourth latent bug was identified (but never hit in practice) before the mechanism was retired: gh run list defaults to --limit 20, so under sustained churn a genuinely-older blocker could sit beyond the fetch window and be invisible — the same silent-collision class as bug 3.

Each of those three bugs was a sound, correct fix for a real problem — this wasn't sloppy work. It was the signature of an inherently edge-case-rich design: a hand-rolled distributed mutex polling an eventually-consistent, paginated CI API. Every fix closed one edge case and the mechanism kept generating more, because the category of problem (build correct mutual exclusion over an API that wasn't designed to provide it) has more edge cases than any one PR account for.

The fix that actually shipped (re-scope #4, PR #1305) retired the mutex entirely:

• Staging keeps demo@objectuve.com, gets a native concurrency: group (smoke-staging-demo, cancel-in-progress: false) scoped to staging.yml's own smoke job.
• Preview gets its own dedicated account (preview-smoke@objectuve.com) via a preview_smoke_email input threaded through playwright-smoke.yml, with its own group (smoke-preview-lane).

Two distinct groups mean GitHub's cancellation semantics never cross lanes — a Preview run can never touch a queued Staging run, which is what made a shared native group unsafe back when the two lanes shared one account. Splitting the account is what made the native primitive safe to use at all.

Takeaway: when a hand-rolled synchronization mechanism keeps producing new, real bugs across review rounds, that is evidence the mechanism is the problem, not any single bug in it. Prefer the platform's native primitive, and where possible remove the shared resource the synchronization exists to protect (giving Preview its own account eliminated the dominant collision source by construction) rather than racing to win access to it.

Lesson 3 — when isolating a shared resource, audit every path that authenticates against it, not just the primary one

Giving Preview its own preview-smoke@objectuve.com account only isolates the race if every code path that authenticates as the fallback identity actually uses the new email. Mid-implementation, a review pass found several direct GraphQL call sites in the smoke suite that were hardcoded to the old shared DEMO_EMAIL regardless of which account the browser session had actually signed in as — helpers/env.ts's buildFallbackToken() defaulted to it unconditionally. Left unfixed, those call sites would have silently defeated the isolation for every request that went through them, even though the primary browser-session auth flow was correctly isolated.

The fix was a single funnel point: getFallbackEmail() in ionic_frontend/tests/smoke-playwright/helpers/env.ts (process.env.SMOKE_FALLBACK_EMAIL ?? DEMO_EMAIL), which every fallback-auth call site — journey.spec.ts, auth.production.setup.ts, and buildFallbackToken() itself — now goes through. Verified end-to-end (grepped the whole smoke suite for DEMO_EMAIL/hardcoded email literals and SMOKE_FALLBACK_EMAIL usage across every file), not just trusted from the implementation report.

Takeaway: when isolating a shared resource (account, credential, seed identity), audit every authenticating code path — not just the primary/browser-session path — before declaring isolation complete. A secondary call site that still resolves to the old shared identity reopens the exact race the isolation was built to close, and it will look identical to the original bug when it reproduces.

Follow-ups tracked separately (not blocking this fix)

• smoke-goal-cleanup.yml shared the same exposure pattern — resolved. This is a daily production+staging cron (0 4 * * *, matrix over both environments since OBJ-1333) that authenticates via fallback token against each environment's demo account (api.objectuve.com / staging.api.objectuve.com, demo@objectuve.com by default) to soft-delete stale seed goals. Same root-cause shape as this ticket (unlocked concurrent access to a shared mutable seed identity), different backend/DB. OBJ-1190's follow-up scope is now closed: production.yml's own smoke suite and smoke-goal-cleanup.yml both join the matching per-environment concurrency: group (smoke-production-demo / smoke-staging-demo, cancel-in-progress: false, queue: max), so the cron can no longer delete a goal mid-assertion under a concurrently-running smoke suite on either environment.
• Preview's concurrency: group used to be intentionally global, not per-PR — see next section for the trade-off that held at the time, the starvation follow-up it caused, and the later milestone that removed the constraint entirely.

Preview's (retired) global concurrency group: expected trade-off, with a real follow-up

Superseded by Milestone v4.4 (OBJ-1322/OBJ-1325/OBJ-1326). Everything in this section describes the design as it stood between OBJ-1150 (PR #1305) and v4.4. Preview's concurrency: group and its fallback-auth account are no longer global or shared — each PR now gets its own preview-smoke+pr-<N>@objectuve.com account and its own smoke-preview-lane-<N> group, so the trade-off analyzed below no longer applies. See Preview Smoke Lane Isolation — decision record and Playwright smoke gate for the current design. Kept here as history — the reasoning is still a useful worked example of the shared-global-resource trade-off.

The Preview lane's concurrency: group (smoke-preview-lane in preview.yml) was deliberately global — unscoped by PR/ref — because every open PR's Preview smoke run still shared the one preview-smoke@objectuve.com account (that account wasn't parameterized per-PR). A per-PR-scoped group was tried and reverted for exactly this reason (OBJ-1094): per-PR scoping let multiple PRs' smoke runs execute concurrently against the same shared account, reopening the exact race this whole fix existed to prevent. The global group was what actually serialized account access — a legitimate design at the time, not a bug.

What did need a follow-up fix: at ship time, that global group used cancel-in-progress: true, on the assumption that a superseded Preview-smoke run is safe to kill outright because the check is advisory-only (not in required_status_checks). In practice, under this repo's normal multi-PR push cadence, cancel-in-progress: true on a global group means any PR unlucky enough to push while another PR's smoke run is in flight gets its own run killed in favor of the newest arrival across any PR — not just its own. This starved PR #1307 (preempted twice in a row, never reached a concluded status) shortly after this fix shipped, and was corrected in OBJ-1175 (PR #1311) by switching to cancel-in-progress: false: still exactly one Preview smoke run touched the shared account at a time (queue, not kill), but every run ran to completion instead of risking indefinite preemption — matching the trade-off staging.yml's smoke-staging-demo group already made for the same underlying reason.

What eventually made the trade-off itself unnecessary: OBJ-1175's fix was durable against preemption but not against contention — a global lane with one shared account still meant a PR queue of N pushes took roughly N times as long to clear as a single push (OBJ-1322). Milestone v4.4 removed the shared resource this whole section is about: once the account is per-PR (OBJ-1325), scoping the group per-PR too (OBJ-1326) is safe again, because OBJ-1094's original failure mode — concurrent PRs racing a shared account — can't recur when there's no longer a shared account to race.

Takeaway: "this check is advisory-only, so cancelling it is safe" is true for the check's gating effect on a PR, but not for the velocity effect on the PR whose run gets killed — a PR that never sees its own smoke check conclude is a real cost even when nothing is blocked on it. If a shared concurrency group must stay global (because the underlying resource is genuinely global), default to cancel-in-progress: false unless there's a specific reason a stale run's result is actively harmful to keep around. And when a "must stay global" resource stops being shared, that's cause to revisit whether the group still needs to be global at all — the platform's native primitive removed the collision by construction here, exactly as it did when the hand-rolled lock was retired for concurrency: groups in the first place (Lesson 2 above).

Related

• Staging deploy race: unscoped concurrency lets an older commit overwrite a newer one — a different concurrency: pitfall (completion-order races within one deploy job), same workflow family
• GitHub Actions Force-push gotcha — a different CI-trigger footgun in the same workflow surface
• Smoke Test Environment Gotchas — other smoke-playwright environment-specific traps
• .github/workflows/playwright-smoke.yml, .github/workflows/staging.yml, .github/workflows/preview.yml — the reusable smoke workflow and its two callers
• ionic_frontend/tests/smoke-playwright/helpers/env.ts — getFallbackEmail(), the single funnel point for fallback-auth account resolution
• Preview Smoke Lane Isolation — decision record — the v4.4 ADR that supersedes this doc's "Preview's (retired) global concurrency group" section with per-PR account + lane isolation
• Playwright smoke gate — current per-PR account provisioning/teardown and concurrency scoping

Last updated: 2026-07-11 (OBJ-1362: marked the smoke-goal-cleanup.yml follow-up as resolved — cron is now production+staging, both closed under OBJ-1190's concurrency: groups)