ADR: Preview smoke lane isolation — shared account to per-PR

Status: Accepted — shipped in Milestone v4.4 (Preview Smoke Lane Isolation), PR #1431 Date: 2026-07-10 Related: Playwright smoke gate, Smoke-account concurrency gotcha

Context

Preview smoke (playwright-smoke.yml, called from preview.yml on every PR push) authenticates as a fallback-auth account rather than a real Clerk sign-up. For most of this account's history, every open PR shared a single account and a single GitHub Actions concurrency: group. As PR volume grew, that single-account, single-lane design became the bottleneck: every open PR's Preview run queued behind every other PR's, so a busy PR queue could leave an individual PR's Preview run pending indefinitely (OBJ-1322).

This decision record captures the shared-account → per-PR isolation pivot and the sequence of concurrency designs that preceded it, three of which were retired.

Retired approaches

1. Hand-rolled cross-run lock (gh run list polling)

The first fix for a shared, unlocked demo account (OBJ-1150) serialized access with a bash step that polled gh run list and waited for any earlier in-progress run to clear. Chosen over a native concurrency: group specifically to avoid killing a queued Staging run while Staging and Preview still shared one account.

Retired after hitting the crew's 3-round fix cap on its own mechanism:

• Timeout — the original 60-minute bound was too short for the repo's actual concurrent-PR churn.
• Livelock — extending the timeout wasn't enough; the "blocker" could rotate to a fresher arrival before the wait converged.
• TOCTOU — the FCFS check filtered on --status in_progress, missing an older run still queued at query time, a window where two runs could both proceed unlogged.

See Smoke-account concurrency gotcha, Lesson 2 for the full post-mortem.

2. OBJ-1094 — early per-PR/ref concurrency scoping (reverted)

Before the Preview account itself was isolated, the smoke-preview-lane group was scoped per-PR (per github.event.number / ref) on the theory that giving each PR its own lane would remove contention. It didn't — every PR was still authenticating as the same preview-smoke@objectuve.com account, so per-PR scoping let multiple PRs' smoke runs execute concurrently against that shared account, reopening the exact OBJ-1150 race the lane isolation existed to prevent. Reverted back to a single global group.

3. Global group + preview-smoke@objectuve.com (OBJ-1150 re-scope #4, then OBJ-1175)

The re-scope #4 fix (PR #1305) retired the hand-rolled mutex in favor of native concurrency: groups and gave Preview a single dedicated account (preview-smoke@objectuve.com, separate from Staging's demo@objectuve.com) with one global smoke-preview-lane group. Because the group stayed global (every open PR's Preview run in the same lane), cancel-in-progress: true meant any PR unlucky enough to push while another PR's run was in flight got its own run killed in favor of the newest arrival across any PR, not just its own. This starved PR #1307 (preempted twice in a row, never reached a concluded status) shortly after the fix shipped, and was corrected in OBJ-1175 (PR #1311) by switching to cancel-in-progress: false — exactly one Preview smoke run touched the account at a time (queue, not kill), and every run ran to completion.

That correction was durable against preemption, but not against contention: it still serialized every open PR's Preview run onto one account. OBJ-1291 (PR #1414) later found that cancel-in-progress: false alone isn't a durable FIFO queue without queue: max (see CLAUDE.md's gotcha of the same name) and added queue: max to smoke-preview-lane alongside the other two smoke lanes. That kept pending runs from being silently evicted, but a global lane with one shared account still means a PR queue of N pushes takes roughly N times as long to clear as a single push — real account-level starvation under this milestone's parallel-PR volume (OBJ-1322).

Decision

Round 4 (this milestone, OBJ-1322): give every PR its own account and its own concurrency lane.

• Per-PR account isolation (OBJ-1325) — each PR's Preview run authenticates as preview-smoke+pr-<N>@objectuve.com, provisioned idempotently by preview.yml's ensure-preview-smoke-account job and torn down (hard delete, cascading Smoke * goal cleanup) by preview_teardown.yml's teardown-preview-smoke-account job on PR close. Two PRs' Preview runs can never touch the same account row.

  Addendum (OBJ-2705, PR #2472). Account-row isolation held, but the provisioning mechanism underneath it didn't: ensure-preview-smoke-account's Cloud Run Job resource (enkidu-bootstrap-preview-smoke-account) is not itself PR-scoped — every PR's Preview run mutated the same job spec via gcloud run jobs update --args <this PR's script> (or create) before calling execute --wait with no args override. Two PRs racing that step in the same window could each overwrite the other's stored --args between their own update and execute calls, so execute could run with a different PR's RUNNER_SCRIPT — silently provisioning the wrong PR's account while the step still reported success (Capture UI Evidence (Preview)'s baseline-goal GraphQL query then failed UNAUTHORIZED against an account that was never actually created for that PR). preview_teardown.yml's teardown counterpart shared the identical shape. Fixed by moving --args off the update/create calls and onto execute itself — a one-off override that doesn't mutate the shared spec — with a CI guard (ci.yml's lint-workflows job) against regression. Full implementation detail: Playwright smoke gate.

• Per-PR concurrency re-scope (OBJ-1326) — smoke-preview-lane's group key is now smoke-preview-lane-${{ github.event.number }}, keyed on the same PR number as the account, so lane-scope and account-scope always agree. cancel-in-progress: true is safe to re-apply because it can now only supersede that PR's own stale run on a rapid re-push, not another PR's run — the OBJ-1175 starvation case doesn't apply to a per-PR group. queue: max is deliberately not re-added: it only protects a global group where unrelated PRs queue behind each other, which no longer happens once the group is per-PR.

This removes the shared-resource invariant (both the account and the lane) that forced every prior round to choose between serialization (OBJ-1150/OBJ-1175/OBJ-1291, safe but N-times-slower under load) and concurrency (OBJ-1094, fast but racy). Full implementation detail: Playwright smoke gate — per-PR smoke account and concurrency isolation.

Milestone deviations

Two deviations from the milestone's original plan surfaced during execution and are recorded here rather than left implicit in issue history:

Phase 1 reframe. The milestone's opening premise — "a cancelled Preview run GitHub-enforces a merge block; Staging is the hard PR gate" — did not hold against live repo state (verified in OBJ-1324): Preview smoke has never been a required status check on master, and Staging smoke isn't a PR check at all (staging.yml only triggers on push: branches: [master], after merge). Phase 1 is accordingly framed as a signal-quality / policy-clarification fix (documenting the true, already-advisory gate behavior), not a "hard-gate unblock." The account-starvation problem (OBJ-1322) this ADR addresses was independent of that premise and real regardless — it's what Phases 2–3 fixed.

Phase 2 close path. OBJ-1325 (per-PR account isolation) closed via co-ship verification inside PR #1431 — Codi's 19 green RSpec examples, Roy's review of PR #1431, and Vicki's independent re-check — rather than a standalone review hop. Teardown was resolved as a hard delete (really_destroy!, not a soft delete) with cascading Smoke * goal cleanup, for the unique-index reason documented in Playwright smoke gate.

Consequences

• Two PRs' Preview smoke runs can now execute genuinely concurrently — no shared account, no shared lane — removing the account-starvation bottleneck at its root instead of tuning the queue around it.
• Every PR close must reliably fire preview_teardown.yml's teardown-preview-smoke-account job, or a closed PR's account/goals become orphaned rows (idempotent re-close and never-provisioned cases are both handled as no-ops, but a workflow that never runs — e.g. a force-deleted branch bypassing the closed event — would leak a row; not observed to date).
• queue: max remains on smoke-staging-demo and smoke-production-demo (still global, single-account lanes by design) but must not be re-added to smoke-preview-lane-<N> without first reverting it back to a global group — doing so silently reopens the OBJ-1094 failure mode alongside a queue that no longer needs to exist.

Related

• OBJ-1094 — early per-PR concurrency scoping, reverted (retired round 2)
• OBJ-1150 — shared dedicated Preview account + native concurrency: groups, replacing the hand-rolled lock (retired round 1's replacement, later itself superseded)
• OBJ-1175 — cancel-in-progress: false correction on the (then-global) lane (retired round 3's fix)
• OBJ-1291 — queue: max added across all three smoke lanes
• OBJ-1322 — Milestone v4.4 (Preview Smoke Lane Isolation), account-starvation problem this ADR resolves
• OBJ-1324 — live branch-protection verification behind the Phase 1 reframe
• OBJ-1325 — per-PR isolated smoke accounts
• OBJ-1326 — per-PR concurrency re-scope
• OBJ-1327 — this decision record
• OBJ-2705 — provisioning-layer race on the shared bootstrap/teardown Cloud Run Job, fixed by moving --args to execute-time (account-row isolation itself unaffected)

---

Last updated: 2026-08-19 (OBJ-2705: added the provisioning-layer Cloud Run Job race addendum)