Objectuve Docs

Appearance

v4.4 — Preview Smoke Lane Isolation ​

Give every PR its own smoke account and its own concurrency lane, and the three-round starvation problem disappears by construction instead of by another queue tweak.

Summary ​

Playwright Smoke Tests (Preview) — the advisory-by-design smoke suite that runs on every PR push — had spent three rounds trying to fix contention under load without ever touching its root cause: every open PR shared one preview-smoke@objectuve.com account and one global smoke-preview-lane concurrency group. A hand-rolled cross-run lock was retired after hitting its own round cap (timeout → livelock → TOCTOU). A per-PR concurrency re-scope (OBJ-1094) was reverted because it let multiple PRs authenticate against the same account concurrently, reopening the exact race it was meant to prevent. The current-at-the-time fix — a single global group with cancel-in-progress: false — stopped runs from being killed but not from queuing behind each other, so a busy PR queue could still starve an individual PR's run indefinitely (hit PR #1409 twice, respec'd as OBJ-1322 after its own round cap was reached).

v4.4 stopped tuning the queue and removed the shared resource instead. Each PR now provisions its own isolated smoke account (preview-smoke+pr-<N>@objectuve.com) at first Preview run and tears it down on PR close; with accounts disjoint, the concurrency group could safely be re-scoped per-PR too (smoke-preview-lane-${{ github.event.number }}), reviving the OBJ-1094 approach without reopening its race. Along the way, Phase 1's live-ruleset check found the milestone's opening premise didn't hold — Preview smoke was never actually a required merge check, so nothing was hard-blocking anyone. All four phases shipped the same day the milestone opened, directly to master — this is CI/infra work with no feature flag and no user-facing surface.

Goal ​

Stop Playwright Smoke Tests (Preview) from starving/blocking PRs under concurrent load by removing the shared-account bottleneck rather than re-tuning the concurrency queue (which has hit its round cap at R3). Two independent tracks: (1) an immediate policy mitigation — de-require the advisory-by-design Preview check from branch protection so a cancelled run can no longer hard-block merge; (2) the durable fix — a per-PR isolated smoke account so per-PR concurrency scoping becomes safe again without reopening the OBJ-1150 shared-account race.

Scope — What Shipped ​

  • rails_api/lib/tasks/smoke_accounts.rake extended for per-PR email (preview-smoke+pr-<N>@objectuve.com) + a unique username per PR, idempotent find-or-create, demo: false
  • .github/workflows/preview.yml's ensure-preview-smoke-account job provisions the PR-N account and threads it through to playwright-smoke.yml as preview_smoke_email
  • .github/workflows/preview_teardown.yml (fires on pull_request: [closed]) hard-deletes the PR-N smoke account and its residual Smoke * goals; idempotent — a re-close or a PR that never provisioned is a clean no-op
  • smoke-preview-lane concurrency group re-scoped from a single global group to smoke-preview-lane-${{ github.event.number }} with cancel-in-progress: true, keyed on the same PR number as the account
  • docs/development/playwright-smoke-gate.md updated with the true PR-merge-gate policy (Preview was never required; Staging isn't a PR check at all) and the new per-PR isolation/concurrency section
  • New ADR at docs/development/preview-smoke-lane-isolation-adr.md recording all four concurrency-design rounds and the shared-account → per-PR pivot
  • Stale-reference sweep: CLAUDE.md's queue: max gotcha and docs/operations/smoke-account-concurrency-gotcha.md both updated to point at the new per-PR design instead of describing the retired global-queue model as current

Phases ​

PhaseNameStatusPlansHighlights
1De-require Preview smoke (policy mitigation) + docsShipped1Live-ruleset check found Preview was never a required check — shipped as a gate-doc clarification instead of a settings change
2Per-PR smoke account isolationShipped1preview-smoke+pr-<N>@objectuve.com, idempotent provisioning, hard-delete teardown with cascading goal cleanup, 19 green RSpec
3Per-PR concurrency re-scopeShipped1smoke-preview-lane-<N> group, live-verified with a concurrent 2-PR run
4Docs & decision record (ADR)Shipped—Gate doc, new ADR, CLAUDE.md + gotcha-doc sweep

Key Decisions ​

  • Remove the shared resource, don't re-tune the queue — three prior rounds (hand-rolled lock, OBJ-1094 revert, OBJ-1175 global-group correction) all treated smoke-preview-lane contention as a scheduling problem. This milestone treated it as an ownership problem: give every PR its own account and its own lane, and there's nothing left to contend over.
  • Lane-scope keyed to the same identifier as account-scope (github.event.number) — this is the structural reason the OBJ-1150 race can't reopen. Two PRs can never share an account or a lane, by construction, not by queue discipline.
  • Phase 2 and Phase 3 co-shipped as one PR — the per-PR concurrency re-scope is unsafe until per-PR account isolation is proven; isolation alone delivers no throughput benefit. Landing them as one PR meant the re-scope never existed on master without the isolation it depends on.
  • Teardown is a hard delete, not soft — really_destroy! with cascading Smoke * goal cleanup, resolved during Phase 2 execution to avoid a unique-index collision if a PR number is reused after a soft-deleted row.
  • queue: max deliberately not re-added to the per-PR lane — it only protects a global group where unrelated PRs queue behind each other; once the group is per-PR, there's nothing left for it to protect. It remains on the two lanes that are still global by design (smoke-staging-demo, smoke-production-demo).
  • No feature flag, no master-flip — unlike flag-gated product milestones (e.g. v4.1), this is CI/infra with no user-facing surface. Each phase PR merged straight to master; the isolation and re-scope have been live in production CI since Phase 2/3 shipped.

Requirements Coverage ​

11 / 11 requirements satisfied — DEREQ 1/1, DOCS 4/4, ACCT 3/3, TEARDOWN 1/1, SCOPE 2/2.

CategoryCountStatus
DEREQ-11Satisfied (resolved as a no-op — never actually required)
DOCS-1 – DOCS-44All satisfied
ACCT-1 – ACCT-33All satisfied
TEARDOWN-11Satisfied
SCOPE-1, SCOPE-22All satisfied

Full requirements list and per-phase detail: v4.4-preview-smoke-lane-isolation-ROADMAP.md on GitHub. No separate REQUIREMENTS.md file exists for this milestone — requirements are defined inline per phase in the ROADMAP.

Outcomes ​

Two or more PRs' Preview smoke runs now execute genuinely concurrently — no shared account, no shared concurrency lane — so a busy PR queue no longer starves an individual PR's run. Preview smoke's real merge-gate status is documented accurately (advisory, never required; Staging remains the hard post-merge gate). The smoke-preview-lane design history (4 rounds across 2+ months) is now captured in a single ADR instead of being reconstructed from workflow comments and issue history each time it resurfaces.

Tech Debt ​

  • (Milestone) No periodic sweep exists for orphaned smoke accounts if preview_teardown.yml ever fails to fire (e.g. a force-deleted branch bypassing the closed event). Teardown itself is idempotent and reliable in the observed cases; this is an unobserved, low-priority residual risk noted in the ADR's Consequences section, not a known incident.
  • faea5b9e6 — docs(ci): Preview smoke is advisory/non-required; Staging isn't a PR check (#1423)
  • 82091bb4b — [Codi] fix(ci): per-PR isolated Preview smoke accounts + per-PR concurrency scope (v4.4, OBJ-1322 round-4) (#1431)
  • 5325f184b — docs(ci): v4.4 per-PR smoke isolation — gate doc + ADR + stale-ref sweep (#1433)

Last updated: 2026-07-10 Version: v3.13.4 (no dedicated milestone tag — CI/infra only)

Loading…