Test Suite Audit (2026-06-06)
A monorepo-wide evaluation of the test suite — unit through smoke/E2E — capturing the state of every tier, a risk-scored gap register, and a phased remediation roadmap. Tracked as OBJ-920. Wave 1 (CI wiring + coverage floors) shipped alongside this doc; Waves 2–3 remain open.
Companion to testing.md (the how-to guide). This file is the audit snapshot — what exists, what's missing, and what to do about it.
Current state (verified)
| Tier | App | Size | In CI? | Coverage gate |
|---|---|---|---|---|
| RSpec | rails_api | 4,185 examples / ~446 files, 58 factories | ✅ blocking | line ≥ 85% (CI-only, ratchet) |
| Vitest | ionic_frontend | 268 specs | ✅ blocking | global 55/40/45/55 + 8 per-file @80/75/80/80 |
| Vitest | admin_dashboard | 73 specs / 503 tests | ✅ blocking (Wave 1) | 72/62/66/74 (Wave 1) |
| Vitest | agent_runner | 9 specs / 99 tests | ✅ blocking (Wave 1) | 85/78/68/85 (Wave 1) |
| Cypress E2E | ionic_frontend | 3 specs (smoke tier, OBJ-568) | ✅ blocking | n/a |
| Playwright smoke | ionic_frontend | 32 specs / 9 suites | ⚠️ advisory preview; gates staging/prod | n/a |
| Storybook | ionic_frontend | 294 stories, addon-a11y + jest-axe | ❌ a11y not asserted in CI | n/a |
Measured coverage at audit time: rails ~90% line, ionic per-config, admin 75.0/65.7/69.6/77.9, agent_runner 90.5/83.7/75/90.1 (stmts/branches/funcs/lines).
Gap register (risk-scored)
| # | Risk | Gap | Evidence | Status |
|---|---|---|---|---|
| 1 | 🔴 | admin_dashboard + agent_runner unit suites never ran in CI (deps installed + npm audit only) | ci.yml vitest job ran test:unit for ionic only | Fixed (Wave 1) |
| 2 | 🔴 | Committed duplicate test tree admin_dashboard/admin_dashboard/ shadowed the real tree and held the only coverage for ReviewQueueView, RunDetailView, OnboardingFunnelView, ErrorBoundary, sentry, posthog, apollo-error-link; one spec had rotted (missing DialogDescription mock) | double-include via vite globs | Fixed (Wave 1) — relocated 8 unique files, ported rich EmployeeDetailView behavior tests, deleted nested tree |
| 3 | 🔴 | Billing untested (Rails) | no specs for stripe_service, create_checkout_session, cancel_supporter_subscription, get_billing_portal_url | Wave 2 |
| 4 | 🟠 | No backend coverage floor (SimpleCov minimum_coverage unset) | spec/spec_helper.rb | Fixed (Wave 1) — line: 85 in CI |
| 5 | 🟠 | agent_runner internals thin: state/runMap.ts 14% lines; index.ts 66% | vitest run --coverage | Wave 2 |
| 6 | 🟡 | ionic component coverage ~47% (138/263 untested); global gate 55% vs 80% target | per-file thresholds only on 8 surfaces | Wave 3 |
| 7 | 🟡 | No coverage telemetry (no Codecov/JUnit upload; no PR-diff/trend) | CI uploads none | Wave 3 |
| 8 | 🟡 | Rails gaps: 3 interactions (enqueue_clear, flag_review_operations, scope_preview), 6 services, 7 non-base mutations without specs | spec vs app file diff | Wave 2 |
| 9 | 🟢 | No parallelization (4,185 RSpec + 268 Vitest serial) | ci.yml; no parallel_tests/sharding | Wave 3 |
| 10 | 🟢 | Tooling absences: no timecop/vcr (Rails); Storybook a11y not CI-gated; visual-regression manual-dispatch only | Gemfile, workflows | Wave 3 |
| 11 | 🟢 | 8 critical_path specs fail locally on date-sensitive assertions (YYYYMMDD puzzle seed / "played today"); pass in CI | spec/interactions/critical_path/*, spec/requests/critical_path_play_spec.rb | Observed — latent date-fragility, candidate for timecop (see #10) |
Tooling assessment
In place: RSpec 8 + FactoryBot + shoulda-matchers + webmock + database_cleaner + SimpleCov; Vitest 4 + @vue/test-utils + v8 coverage (all four JS apps); Cypress 15; Playwright 1.60 (9 suites, mobile + desktop projects); Storybook 10 + addon-a11y + jest-axe; custom auth-spec gate (bin/check_auth_specs); 8 RSpec shared examples; N+1 query_recorder.
Worth adding (Wave 3): coverage telemetry (Codecov or artifact upload of v8 lcov + SimpleCov HTML); timecop for Rails time-travel (would harden the date-fragile critical_path specs, #11); a CI a11y gate driving the existing jest-axe/addon-a11y; test parallelization (parallel_tests for RSpec, Vitest sharding).
Roadmap
Wave 1 — shipped (this batch). Wire admin_dashboard + agent_runner suites into the ci.yml vitest job; consolidate the duplicate admin test tree (no coverage lost); add ratchet (no-regression) coverage floors to rails, admin, and agent_runner.
Wave 2 — missing critical specs. Stripe/billing services + mutations (#3); agent_runner runMap/index/route-middleware coverage (#5); the 3 untested interactions, 6 services, 7 mutations (#8).
Wave 3 — coverage ramp + infra. Raise ionic global threshold toward 80% as component coverage climbs (#6); coverage telemetry (#7); parallelization (#9); timecop
- CI a11y gate + automated visual regression (#10, #11).
Coverage stance
Ratchet upward, not 80%-now. Floors are set a few points below current passing levels to lock no-regression while absorbing run-to-run variation, and raised incrementally as suites improve. New floors live in: rails_api/spec/spec_helper.rb (CI-gated), admin_dashboard/vite.config.ts, agent_runner/vitest.config.ts.
Last updated: 2026-07-21