Objectuve Docs

Appearance

v4.46 — Storybook play() CI Enforcement ​

Every Storybook play() function — and every story's mere render, whether or not it defines one — now runs as a real Vitest test on every PR, in headless Chromium, as a required check with no continue-on-error.

Summary ​

Before this milestone, a Storybook play() function ran only when a human opened Storybook and clicked that specific story. npm run test:unit, npm run test:cov, .github/workflows/ci.yml, and storybook.yml (which builds and deploys Storybook but never executes a story) all never touched it. That gap was not hypothetical: Tess's OBJ-2919 fix added outro-reachability assertions to FeatureTour.stories.ts after a story silently rendered the wrong state with nothing to catch it — and those very assertions were unenforced from the moment they shipped, because nothing in CI ran play() at all.

This milestone wires @storybook/addon-vitest's storybookTest() plugin into a required CI job. The wiring itself is small; the risk is not. storybookTest() doesn't only run play()-bearing stories — it generates a render-and-assert-no-error smoke test for every story it discovers, so the change takes CI coverage from 0 to 2,543 tests across 429 files in one commit's worth of enforcement, none of which had ever been executed headlessly before. Shipping that safely, with zero tolerance for the soft-launch shortcut (continue-on-error) this repo's own CLAUDE.md explicitly forbids, is the actual difficulty this milestone solves.

Phase 1 built the harness with no CI changes at all and turned "what breaks?" from speculation into a number: 85 of 2,503 stories failing across 22 files, plus four deviations that reshaped everything downstream — storybookTest() never loads the root vite.config.ts, a new dependency was required, the suite hung indefinitely at teardown, and one story file silently vanished from the full run while passing standalone. Phase 2 sized itself against that inventory and grew from a planned two waves into six, not from scope creep but from one causal chain only live execution could surface: seeding a signed-in Storybook session un-gated dozens of previously-inert queries, which exposed dead decorators, which required making unhandled rejections attributable, which surfaced a residue sweep, which led to a real @ionic/core teardown-race fix. Phase 3 made the job required, closed two more pre-requisite flakes first under the rule that "a job born required cannot be born flaky," and — re-measuring on real CI hardware rather than trusting the local baseline — discovered the suite could exhaust a hosted runner's disk before it ever exhausted the runtime budget, shipping a 4-way file-list shard as the fix.

Goal ​

Make every Storybook story's play() function a real, CI-enforced regression test — without turning master red on 2,508 stories that have never once been executed by a machine.

Scope — What Shipped ​

  • Harness: ionic_frontend/.storybook/vitest.setup.ts (setProjectAnnotations) and ionic_frontend/vitest.storybook.config.ts — a standalone Vitest config running storybookTest() in browser mode against real headless Chromium, deliberately separate from the unit-test config so the app's hand-ratcheted test:cov coverage thresholds (80/73/73/78) never move.
  • Harness fixes: the Clerk Vue plugin installed in .storybook/preview.ts's mock setup, a missing mock-router route, a deterministic fix for a story file that silently vanished from full-suite discovery, and a vue module-realm dedupe fix for several defineComponent is not a function failures.
  • Component + story fixes: a genuine infinite reactive-update loop in SupporterUpgradeModal, a 610-event GraphQL retry loop in WidgetSetupSheet that was the suite's teardown-hang root cause, a shared SyntaxError across three settings sections, an Ionic present() timing bug in StreakDetailsModal, plus mechanical story-wiring and selector fixes across a dozen more files.
  • Tier C remediation: all 11 empty play() bodies (CommunityAllyInviteInterstitial ×5, BetterTogetherInterstitial ×6) replaced with real assertions or removed; ~30 optional-chained ?.click() no-ops converted to throwing getBy*/findBy* queries so a wrong selector fails instead of silently passing.
  • CI: a new storybook-test job in .github/workflows/ci.yml, 4-way file-list sharded (ionic_frontend/scripts/partition-storybook-files.mjs), wrapped in both a job-level and command-level timeout, added to the REQUIRED_JOBS continue-on-error guard allowlist. ionic_frontend/scripts/check-storybook-file-count.mjs enforces a discovered-vs-reported completeness invariant per shard.
  • Docs: docs/development/testing.md § Storybook (CI enforcement, local run commands, the semantic coverage policy, the Tier C anti-pattern) and a new docs/development/gotchas.md entry, cross-linked from the pre-existing OBJ-2919 entry; CHANGELOG.md [Unreleased] entry.

Phases ​

PhaseNameStatusPlansHighlights
1Harness + failure inventoryShipped1Wired storybookTest() with zero CI changes; produced a counted 85-failure inventory across 22 of 433 files; found storybookTest() doesn't load the root Vite config and the suite hangs indefinitely at teardown after every test reports.
2RemediateShipped1 (8 PRs, 6 waves)Fixed all inventoried failures plus a fully unplanned Wave C causal chain surfaced by seeding a signed-in session; killed every Tier C structurally-unfailable play(); verified 429/429 files green on three consecutive runs with unchanged coverage thresholds.
3Enforce + documentShipped1 (2 PRs)Made storybook-test a required CI job with no continue-on-error; re-measured on CI hardware and discovered a disk-exhaustion failure mode invisible to the runtime-based sizing, fixed via 4-way sharding; wrote the gotcha and testing docs OBJ-2969 asked for.

Key Decisions ​

  • A separate vitest.storybook.config.ts, not Storybook's scaffolded test.projects entry — merging 2,503 browser-mode story renders into the app's coverage-gated unit-test run would move every one of its 12 hand-ratcheted per-file thresholds and force a Playwright browser install onto every npm run test:unit invocation.
  • No continue-on-error soft-launch at any point — the obvious de-risking move (land the job non-blocking, watch it, then flip it) is forbidden by this repo's own CLAUDE.md gotcha that "a continue-on-error gate is an absent gate." Phase 1 shipped zero workflow changes; the job was born required in Phase 3, only after Phase 2 made it green.
  • Enforce all discovered stories, not just play()-bearing ones — storybookTest() has no supported "only stories with play()" mode, and the render-only smoke test is the exact mechanism that would have caught OBJ-2919's original bug (a story rendering the wrong state). Narrowing scope was explicitly ruled out at kickoff.
  • 4-way explicit file-list sharding, not Vitest's own --shard — partitioning by an explicit, committed file list lets check-storybook-file-count.mjs verify each shard's exact assigned subset, so the discovered-vs-reported completeness invariant survives sharding instead of being skipped for a "scoped" run.
  • Render-only stories are legitimate coverage, not a gap with a numeric target — Phase 3 resolved this explicitly as policy rather than leaving it implied: every story is executed by the harness regardless of whether it defines a play(), catching mount crashes and provider/teardown issues on its own. The rule going forward is semantic, not a percentage: a story that claims a specific rendered state in its name or args must assert that state actually rendered.
  • The master CI gate ruleset promotion is a request, recorded — not a silent gap. Promoting storybook-test into the branch-protection ruleset's required contexts is a repo-admin action outside the crew's authority. The milestone's own acceptance criterion was written as "request it and record the outcome," and that is what shipped — stated honestly here rather than rounded up to a flat "done."

Requirements Coverage ​

5 / 5 milestone success criteria satisfied, 3 / 3 phase acceptance sets satisfied (see .planning/milestones/v4.46-storybook-play-ci-enforcement-MILESTONE-AUDIT.md).

Success criterionStatus
Every play() function executes on every PR✅ (job required in ci.yml's own guard; branch-protection ruleset promotion requested, not yet applied — non-blocking)
OBJ-2919's FeatureTour outro-reachability assertion is enforced✅
Zero play() functions that structurally cannot fail✅
Coverage thresholds byte-identical to pre-milestone values✅
The gotcha is written down where the next person will hit it✅

Outcomes ​

A regression that silently renders the wrong Storybook state — the exact OBJ-2919 shape — now fails a required PR check instead of waiting for a human to notice it in a dev Storybook tab. 2,543 tests across 429 files execute on every PR in real headless Chromium; 133 play() definitions across 50 files run as real Vitest assertions, with zero structurally-unfailable ones remaining. The suite runs in 221–280s locally and completes reliably on hosted CI runners via 4-way sharding, after two host-level failure modes (a teardown hang, a disk-exhaustion kill) were root-caused and fixed rather than retried away. The one thing not yet true: GitHub's own merge gate does not yet block on a red storybook-test result, because the master CI gate branch-protection ruleset hasn't been flipped — the job reports honestly on every PR today, and the promotion is requested and tracked (OBJ-3390), not silently dropped.

Tech Debt ​

  • (Phase 3, ENFORCE-2) The master CI gate ruleset's required-contexts promotion for storybook-test is requested-and-recorded with Josh, not yet applied by a repo admin. Tracked on OBJ-3390; no separate follow-up issue needed since the request itself is the tracked artifact.
  • (Phase 2, play() coverage breadth) 379 of 429 story files define no play() at all. Resolved as policy, not carried forward as a gap: render-only stories are legitimate smoke coverage under the semantic rule documented in docs/development/testing.md.
  • 3f9f99e2b — Phase 1: Harness + failure inventory (PR #2690)
  • 08cc1f70f — Phase 2 final wave: <ion-refresher> async-teardown race fix (PR #2836)
  • 205707be9 — Phase 3: storybook-test required CI job (PR #2917)
  • d5b5536eb — Phase 3: Storybook play() CI enforcement docs (PR #2920)

Last updated: 2026-09-05

Loading…