Playwright smoke gate

The Playwright smoke suite runs against deployed environments after every push. Its gate behavior differs by environment. Neither Staging nor Preview smoke gates PR merges today — see PR merge gate status below.

Smoke surface by environment

Environment	Gate severity	Playwright projects	Notes
Staging (staging.app.objectuve.com)	Hard-blocking on the staging deploy pipeline — a failure fails that pipeline run, but this runs after merge, so it cannot block a PR	Full suite (all 9 spec suites, 3 browsers)	Triggered by staging.yml on push: branches: [master] (post-merge, not a PR check)
Production (app.objectuve.com)	Hard-blocking — release is blocked on failure	Full suite	Triggered by production.yml on every version tag
Firebase ephemeral PR preview channels (enkidu-app--pr-*.web.app)	Advisory — PR can merge even if smoke fails	Full suite (runs but gate is advisory)	Triggered by preview.yml on every PR push; not a required status check (see below)

PR merge gate status

As of 2026-07-10 (verified live against objectuve-softworks/enkidu's branch-protection ruleset, "master CI gate", id 17927598, created 2026-06-20):

• Preview smoke has never been a required status check on master. It does not appear in the ruleset's required_status_checks list, and never has since the ruleset was created. A cancelled or failed Preview run does not block PR merge — confirmed on PR #1409 (mergeable: true, mergeable_state: unstable with a cancelled Preview run).
• Staging smoke is not a PR check at all. staging.yml triggers on push: branches: [master] — it only runs after a commit has already landed on master, as part of the post-merge deploy pipeline. It has no mechanism to gate a PR merge.
• The PR-merge-blocking required checks on master include: RSpec + Rubocop (3 shards), Vitest (ionic_frontend), Vitest admin_dashboard + agent_runner, GraphQL Schema + Codegen Drift Check, A11y gate (jest-axe / WCAG 2.1 AA), Cross-App Helper Sync, PostHog flag drift gate, and Lint GitHub Actions workflows. Playwright smoke (Preview or Staging) is not among them. (This list is partial — ruleset 17927598 carries 14 required contexts; the four not enumerated here are the Guard — / Validate secret wiring / Frontend Lint + Guards (ionic_frontend) entries. The canonical list is REQUIRED_CONTEXTS in scripts/check-ruleset-required-contexts.mjs.)

This clears up a premise from OBJ-1322: a cancelled Preview smoke run was suspected to be GitHub-enforced merge-blocking, but it isn't. The real defect was smoke-preview-lane account contention/starvation — every open PR's Preview run shared one preview-smoke@objectuve.com account under a single, global smoke-preview-lane concurrency group in preview.yml, so a busy PR queue could preempt a given PR's run indefinitely. That starvation problem was real and independent of branch protection; Milestone v4.4 (Preview Smoke Lane Isolation) resolved it with per-PR isolated smoke accounts and per-PR concurrency scoping — see Per-PR smoke account and concurrency isolation below.

Re-raised and re-declined (OBJ-2481, 2026-08-12): the objectuve-signal audit (OBJ-2479, gate 38) independently re-flagged this same gap — Preview smoke's real reported check-name (Playwright Smoke Tests (Preview) / Run Playwright Smoke Tests, confirmed live against PR #2304) isn't in ruleset 17927598's required-checks list — and proposed adding it. Not implemented. Doing so would revert the advisory-on-ephemeral design this doc's Why ephemeral preview channels are advisory section documents: OBJ-1024 burned 3 rounds (PRs #1213, #1218) trying exactly this — promote Preview smoke to hard-blocking — and converged on "advisory is the durable design" per Josh's 2026-07-03 decision, precisely because Preview's min-instances=0 cold start has no bounded deadline a required PR check can safely assert on. Staging smoke has the same structural issue from the other direction: it only runs post-merge (staging.yml triggers on push: branches: [master]), so it cannot become a PR-blocking required check without restructuring the deploy pipeline itself. If Preview smoke is ever promoted, it should happen via the @ephemeral-eligible cold-surface-agnostic subset described in When to revisit this design, not a blanket ruleset addition. Future audits: this is a known, reasoned, standing decision — not an open gap.

Per-PR smoke account and concurrency isolation (v4.4)

Every open PR's Preview smoke run now gets its own dedicated fallback-auth account and its own concurrency lane — no two PRs ever share either resource, so the account-starvation problem above cannot recur.

Account provisioning — preview.yml's ensure-preview-smoke-account job resolves a PR-scoped email, preview-smoke+pr-<N>@objectuve.com (username derived from the local-part, + replaced with -, so the User.username unique index never collides across PRs), and idempotently finds-or-creates that user via an inline rails runner script run against whatever image enkidu-api-staging is currently serving — it does not depend on this PR's own Ruby code having reached a deployed image. Runs on every Preview push; cheap and idempotent (rails_api/lib/tasks/smoke_accounts.rake's smoke:ensure_preview_account task documents the same logic for manual/staging use, keyed off PREVIEW_SMOKE_EMAIL).

Teardown — preview_teardown.yml's teardown-preview-smoke-account job runs on pull_request: types: [closed] and hard-deletes the PR's account (really_destroy!), plus any residual Smoke Habit / Smoke Goal goals it created (matched via user.goals.with_deleted.where("name ~ '^Smoke (Habit|Goal) '"), each really_destroy!'d before the user). Hard delete, not soft delete, because User's email/username unique indexes aren't scoped to deleted_at IS NULL — a soft-deleted row would keep blocking re-provisioning of the same PR-N account on a PR reopen. Idempotent: a PR that never provisioned an account, or a re-close, both no-op cleanly.

Concurrency scoping — the playwright-smoke job in preview.yml uses:

concurrency:
  group: smoke-preview-lane-${{ github.event.number }}
  cancel-in-progress: true

The group key is suffixed with the PR number — the same key ensure-preview-smoke-account uses to derive the account email, so lane-scope and account-scope always agree. cancel-in-progress: true now only supersedes that PR's own stale run on a rapid re-push; it can no longer preempt another PR's run, since two different PRs never share a group. queue: max (see CLAUDE.md's cancel-in-progress: false alone is not a durable FIFO queue gotcha) is deliberately not applied on this lane — it only matters for a global group where unrelated PRs queue behind each other, which no longer happens here.

Why ephemeral preview channels are advisory

Firebase ephemeral PR preview channels (per-PR deployments at enkidu-app--pr-<N>-<hash>.web.app) have an unbounded cold-start window. Unlike staging (which has min-instances > 0 on Cloud Run), each preview channel spins up a fresh Cloud Run instance with zero pre-warmed state. In practice this means:

• The CDN may have a cold miss on the bundled JS chunks.
• The Cloud Run Rails API may take 20–30s to process the first GraphQL request on a cold resolver path.
• There is no practical way to pre-warm every cold surface without consuming the entire smoke budget.

OBJ-1024 empirically tested this across 3 warmup-patch rounds (PRs #1213 and #1218, both closed without converging):

Round	Cold surface uncovered	Fix attempted	Result
R1	WelcomeView JS chunk (CDN cold miss)	Removed advisory carve-out (no warmup change)	❌ smoke fail
R2	Onboarding slide rendering (smoke.warmup.ts added /welcome/start networkidle)	Warmup extended to 215s of 240s budget	❌ smoke fail (page crash at slide 5)
R3	resetOnboarding GraphQL resolver cold-start (~25s cold)	Added resetOnboarding pre-warm + test.setTimeout(120_000)	❌ smoke fail

After R3, the 240s warmup budget was 215s consumed with no convergence path visible. Josh's decision (2026-07-03): advisory on ephemeral channels is the durable design. Staging smoke remains the hard gate on the post-merge deploy pipeline — it still can't block a PR merge (see PR merge gate status), but a Staging smoke failure does stop that commit from being considered safely deployed.

What the advisory gate means

A smoke failure on a Firebase preview channel:

• Sets ::warning in GitHub Actions (not ::error)
• Does not block PR merge
• Does not block the staging deploy that follows merge
• Is visible in the PR's Check details and the Step Summary

The next staging push (which happens automatically on merge to master) runs with the full hard gate. A regression that slips through ephemeral advisory will be caught by the staging gate before it can reach production.

When to revisit this design

Re-evaluate the advisory carve-out if and when:

• Firebase preview channels gain min-instances > 0 on Cloud Run (pre-warmed server).
• A bounded subset of specs can be reliably identified as cold-surface-agnostic and filtered to a separate Playwright project (the @ephemeral-eligible tagging approach described in Orion's OBJ-1024 respec).

Until then, do not attempt to re-promote the gate to uniformly hard-blocking. See OBJ-1024 for the full evidence trail.

General rule: don't hard-gate on unbounded async latency

Two independent tickets have now hit the same failure shape from different doors — and the second one burned three fix rounds before the pattern was recognized as a repeat, not a new bug.

The rule: if a hard-blocking deploy gate asserts on the wall-clock completion of an async process running under --min-instances=0 (or any other setup where "how long until this actually runs" has no upper bound), no timeout increase converges. Bumping the number treats a design problem as a tuning problem.

Instance	Async process	Environment	Rounds burned	Resolution
OBJ-872 / OBJ-1024	Firebase ephemeral preview channel cold start (CDN cache + Cloud Run cold resolver path)	Preview (min-instances=0)	3 (PRs #1213, #1218)	Advisory-on-ephemeral is the durable design — see above
OBJ-1391	ActiveStorage::PurgeJob (Sidekiq) deleting a superseded photo blob	Staging (min-instances=0 on the Rails API/worker)	3 (PRs #1523 → #1530 → #1531; PHOTO_PURGE_POLL 90s → 240s, run 29286108978)	Delete-poll moved out of the hard-blocking gate into a non-gating, alerting check

Both times, the property actually under test turned out to be fine — proven by direct, primary evidence in every round (cold-start resilience; the storage delete grant). The defect was never in the thing being tested. It was in asserting a bounded deadline on an unbounded tail. In the OBJ-1391 case, expect.polls on both the initial attempt and the automatic retry hit the exact new ceiling and only resolved shortly after — the same qualitative pattern as before, just at a higher number, which is itself the signal that another bump wouldn't converge either.

Before adding or widening a timeout on a hard-blocking gate, ask first: is the thing being waited on pinned to an environment with min-instances=0, or otherwise free to sit in a queue indefinitely? If yes, don't bump the number — split the assertion instead:

• Keep the synchronous, bounded half of the check (a mutation succeeding, a resource appearing) in the hard gate.
• Move the async, unbounded half to a non-gating check — advisory (::warning, the ephemeral-preview pattern above) or a separately-scheduled check with a realistic window — so it keeps alerting without blocking every deploy behind an unbounded tail.

This generalizes the ephemeral-preview carve-out above: that section is the cold-start instance of this rule, OBJ-1391 is the async-job instance. Expect a third shape to eventually surface — anything polling a queue, webhook, or background job for a persistent-environment side effect — and apply the same split.

A related but distinct failure class is CLAUDE.md's continue-on-error re-raise rule (a swallowed step failure reports green instead of timing out red) — different mechanism, same symptom family: a gate that silently isn't doing what someone thinks it's doing.

Related rule: a readiness probe must inherit the gate severity of the resource it's probing

OBJ-1657 (2026-07-22) surfaced a fourth shape, adjacent to but distinct from the async-latency rule above: not a timeout that never converges, but a pre-flight probe hard-failing on infrastructure this deploy doesn't own.

playwright-smoke.yml's Wait for API step always targets api_url, which on Preview is unconditionally the shared staging API (https://staging.api.objectuve.com — Preview deploys a Firebase frontend only, it never gets its own API). When shared staging's Sidekiq dead-queue crossed SIDEKIQ_DEAD_THRESHOLD (26 > 25, a real backlog — not a flake or a slow-to-warm process), /health correctly reported degraded, and Wait for API's Phase 4 check exit 1'd the whole job — even though the job had already computed, in its own Resolve gate severity step, that this run should be advisory-only. The severity classification existed and was correct; it just wasn't wired into every failure path, only into the two Report smoke result steps that run after Playwright. Confirmed systemic, not a one-off: all four open PRs with app-source changes (#1715–#1718) failed this exact step the same day, from one shared-infra event unrelated to any of their diffs.

The rule: every place a job can fail must consult the same severity classification, not just the "headline" check. If a job resolves "is this environment's outcome advisory or hard-blocking" once, every exit 1 downstream — including pre-flight/readiness probes — needs to route through that same resolution, not just the step whose failure the gate was originally designed around.

Fix (shipped): Wait for API now has id: api and every phase failure calls a shared fail_or_warn helper that consults steps.gate.outputs.is_ephemeral_preview (the same output the Report smoke result steps already consumed): advisory on ephemeral preview (::warning, ready=false output, exit 0, skip the Playwright run itself since it can't meaningfully run against an unready API), hard-blocking (exit 1, unchanged) everywhere else. A new Report API readiness failure (advisory) step surfaces the skip in the Step Summary so it's still visible, just not merge-blocking. See .github/workflows/playwright-smoke.yml's Wait for API step.

Guardrail: none added as CI lint — this is a design/wiring invariant across if: conditions in one file, not a pattern a static grep can reliably catch (unlike the errexit/teardown/hardcoded-hash guards in ci.yml's lint-workflows). When adding a new failure path to playwright-smoke.yml, check whether it should also route through steps.gate.outputs.is_ephemeral_preview rather than assuming only the smoke-test-result steps need it.

Configuration

Gate severity is resolved in .github/workflows/playwright-smoke.yml, Resolve gate severity step:

- name: Resolve gate severity
  id: gate
  run: |
    FRONTEND_URL="${{ steps.urls.outputs.frontend_url }}"
    if [[ "$FRONTEND_URL" == *--pr-*.web.app ]]; then
      echo "is_ephemeral_preview=true" >> "$GITHUB_OUTPUT"
    else
      echo "is_ephemeral_preview=false" >> "$GITHUB_OUTPUT"
    fi

The two Report smoke result steps that follow consume steps.gate.outputs.is_ephemeral_preview to choose between exit 1 (hard-blocking) and ::warning (advisory).

Spec provenance: first real run vs. regression (OBJ-3917)

The hard gate above treats every red spec identically — it has no way to tell "this spec has never had a real passing run" from "this spec used to pass and broke." Two incidents one calendar day apart (OBJ-3868, OBJ-3853/PR #3227) both went merge → release-blocking while reading clean in every surface a human or preflight tool checked, because high deploy velocity on master means most post-merge staging.yml runs are guard_stale-skipped (see gotchas.md's guard_stale entry) — the window where a brand-new spec reads "clean" is the norm, not the exception.

scripts/check-smoke-spec-provenance.mjs closes that gap. It derives a green watermark — the newest commit with a real (non-guard_stale-skipped) passing run of the Playwright Smoke Tests (Staging) / Run Playwright Smoke Tests sub-job on staging.yml push runs — and classifies every spec under ionic_frontend/tests/smoke-playwright/specs/**/*.spec.ts against it:

• proven — the spec's introduction commit (git log --diff-filter=AR) is an ancestor of, or equal to, the watermark. It has had at least one real green run.
• unproven — the spec was introduced after the watermark. It has never had a real passing run — a failure here is a first run, not a regression.
• unknown — the bounded watermark walk (capped at MAX_RUNS = 200 staging.yml runs) found no real green run at all. The classifier fails closed: an exhausted walk, a gh api error, or missing git history always yields unknown, never proven.

No new persisted state or ledger — because the suite is unsharded (single job, workers: 4), a fully-green real run proves every spec present at that commit, so provenance is derivable from git history plus one gh api walk.

Three consumers:

1. In-run Step Summary — playwright-smoke.yml's Classify smoke spec provenance step runs after a hard-blocking failure and before Report smoke result (hard-blocking), splitting this run's failing specs into a "❌ regressions in previously-proven coverage" section and a "🆕 never-proven specs (first real run since merge)" section. A run with only one kind of failure shows only that section.
2. staging.yml's notify-smoke-failure Slack alert — carries the same split, best-effort (drops the line rather than blanking the whole alert if classification fails or produces nothing).
3. node scripts/check-smoke-spec-provenance.mjs report — machine-readable JSON ({ spec, status, introduced_at, watermark_sha } per spec) that a release preflight should read directly, instead of raw gh run list conclusions. See deployment.md's beta release cadence.

The classification step runs continue-on-error: true with a terminal re-raise step immediately after it (the same pattern Roy required at R2 on OBJ-3898's streak-count step) — a broken classifier turns the job red without ever changing the smoke gate's own exit code.

Stated limitations:

• Renames reset a spec to unproven. The introduction-commit query uses --diff-filter=AR, deliberately not --follow — a renamed spec's pre-rename history doesn't carry over.
• The watermark is staging-only. Only staging.yml push runs on master are walked; production only runs on tags (too sparse to be a useful watermark) and Preview is advisory on ephemeral channels (and the classification step doesn't run there at all — see steps.gate.outputs.is_ephemeral_preview above).
• An exhausted walk yields unknown for every spec, not proven. This is a fail-closed default, not a bug — see the classify/report output for the watermark_sha that produced it.

See scripts/check-smoke-spec-provenance.mjs's module header and scripts/check-smoke-spec-provenance.test.mjs for the full fixture suite (including the OBJ-3853 and OBJ-3868 incident replays). docs/development/gotchas.md's OBJ-3879 addendum names this as the mechanism that closes its "Still open" callout.

Smoke suite worker concurrency (OBJ-1942)

ionic_frontend/playwright.config.ts sets workers: process.env.CI ? 4 : undefined — raised from an unset default (~2 concurrent workers on a 4-core runner) to use the full runner. This landed as part of a CI-speedup package (Orion profiled Vicki's release runs at 55+ minutes; the unsharded, uncached smoke job was ~10.4m of that).

Orion's original proposal was CI-job-matrix sharding across the 3 browser projects (chromium, mobile-safari, mobile-android). That was investigated and rejected: each shard would re-run setup's account mutations (which reset the shared demo account) concurrently across separate jobs — the exact bug class in CLAUDE.md's "Concurrent CI smoke suites racing an unlocked shared demo account" gotcha (OBJ-1150 — 5 fix attempts, a hand-rolled lock, and a 3-round fix cap hit before that root cause was found). Raising workers instead increases concurrency within the existing single job/single setup/teardown, so it doesn't reopen that failure mode. True per-shard sharding would need per-shard account isolation first (the OBJ-1322/OBJ-1325/OBJ-1326 per-PR isolation pattern, extended to per-shard) — not done here, flagged as a follow-up.

Playwright browser binaries are also now cached (.github/workflows/playwright-smoke.yml, actions/cache keyed on ionic_frontend/package-lock.json) instead of re-downloaded on every run, and the invocation switched to npx playwright test directly (was npm run smoke:playwright, which unconditionally reinstalled browsers first).

Related

• .github/workflows/playwright-smoke.yml — gate implementation
• docs/development/smoke-test-env-gotchas.md — per-spec and per-environment gotchas (Gotcha 16 covers the advisory gate history)
• docs/development/playwright-smoke-secrets-gotcha.md — required GitHub Actions secrets
• The Preview UI-evidence gate proves the frontend believes a query resolves, never that it actually does — same shared-backend caveat as this doc's advisory-gate design, applied to preview.yml's capture-ui-evidence job: a green check proves the frontend renders, not that a schema change resolves end-to-end (OBJ-2348)
• OBJ-872 — introduced the context-aware gate (hard on persistent, advisory on ephemeral)
• OBJ-1013 — eager-load onboarding slides (Task 2 merged; Task 3 / OBJ-1024 retired after R3)
• OBJ-1024 — attempted uniform hard-blocking promotion; 3 rounds; confirmed advisory is durable
• OBJ-1391 — second instance of the async-latency-in-a-hard-gate rule: 3 rounds bumping PHOTO_PURGE_POLL (90s → 240s) before Orion's respec recognized the repeat and moved the delete-half assertion out of the hard-blocking staging gate
• OBJ-1150 — Preview smoke lane isolation re-scope #4: dedicated preview-smoke@objectuve.com account + native concurrency: groups, replacing a hand-rolled polling lock; later superseded by OBJ-1325/OBJ-1326's per-PR isolation (see the ADR below)
• OBJ-1175 — flipped the (then-global) smoke-preview-lane group from cancel-in-progress: true to false after it starved an unlucky PR's run via repeated preemption
• OBJ-1322 / Milestone v4.4 (Preview Smoke Lane Isolation) — the smoke-preview-lane account-starvation problem this doc's PR merge gate status section clarifies is not branch-protection-enforced
• OBJ-1324 — verified live that Preview smoke was never in master's required-checks list and Staging smoke isn't a PR check at all; no ruleset change was needed
• OBJ-1325 / OBJ-1326 — round-4 resolution: per-PR isolated smoke accounts (OBJ-1325) + per-PR concurrency re-scope (OBJ-1326), replacing the shared account and global group described in the retired-history section above
• Preview Smoke Lane Isolation — decision record — the full shared-account-to-per-PR-isolation ADR, all four retired/superseding rounds, and the v4.4 milestone deviations
• OBJ-1657 — readiness probe must inherit the gate severity of the resource it's probing: Wait for API's Phase 4 (/health redis+sidekiq check) hard-failed every open Preview PR on a shared staging Sidekiq dead-queue backlog unrelated to any of their diffs; fixed by routing the step through the existing is_ephemeral_preview classification
• OBJ-1942 — smoke suite worker concurrency: raised Playwright workers to 4 instead of job-matrix sharding, citing OBJ-1150's shared-demo-account concurrency risk as the reason sharding was rejected
• OBJ-3917 — spec provenance: first real run vs. regression: scripts/check-smoke-spec-provenance.mjs classifies each spec as proven/unproven/unknown against a green watermark, closing the gap in gotchas.md's OBJ-3879 addendum where a brand-new spec could go merge → release-blocking while reading clean the entire time (OBJ-3868, OBJ-3853/PR #3227)

---

Last updated: 2026-09-17 (OBJ-3917: documented the spec-provenance classifier)