Objectuve Docs

Appearance

v4.38 — Clerk Auth-Config Integrity ​

A production auth setting was flipped in a dashboard with no diff, no PR, and no review — this milestone put the setting back, built the guard that makes it impossible to do quietly again, and then checked its own remedy's user impact before calling itself done. That impact was zero.

Summary ​

On 2026-08-15, production Clerk's username.required and password.required were toggled true → false directly in the live dashboard — an improvised remedy for native Google OAuth's missing_requirements dead-end. It bypassed every review gate, was caught five days later only because Roy happened to notice it while reviewing an unrelated PR, and it didn't even solve the problem it was meant to solve: legal_consent_enabled was and remained true, so Google sign-up was still blocked regardless. The change traded away web username enforcement and got nothing for it.

v4.38 restored the invariant, then went further than "put it back." Phase 1 discovered the kickoff's own premise was wrong — staging was assumed to already hold the correct value and didn't, so restoring "parity" required fixing both instances, not just production. Phase 3 shipped this repo's first drift guard for the auth provider, and it proved itself twice during the milestone's own lifetime: once catching the very incident it was built for, and once catching a second, unrelated out-of-band change to three other attributes mid-milestone. Phase 5 was added only after the fix landed, when it became clear username.required also governs sign-in for accounts that already exist without a username — not just sign-up for new ones. Rather than assume the one exposed production account was now broken, the milestone went and checked: live traces on both web and native showed Clerk enforces required on account creation and update, not on sign-in. The account works fine. No fix was needed, and none was invented.

Three separate acceptance criteria in this milestone were written as "observe the broken state being handled" and each became impossible to satisfy the moment its own fix shipped — because the fix removes the very state the criterion needed to observe. The milestone named this pattern explicitly and restated each criterion as a property of the restored system instead, which is now a documented, cross-referenced gotcha for the next person who writes an acceptance criterion the same way.

Goal ​

Restore the username-present invariant that production enforced until 2026-08-15, make its violation visible if it recurs, and install a guard so a third-party auth-config change can never again reach production unobserved.

Scope — What Shipped ​

  • Live config restore (Phase 1, OBJ-2607): username.required = true on both production and staging Clerk instances; password.required deliberately kept false on Josh's written product decision (passkey/OAuth-first, and a required password risks re-triggering the same missing_requirements dead-end). username_gate_enabled (PostHog flag 822166) confirmed serving live at 100% rollout. RESTORE-5 found exactly one exposed production account created during the ~3.4-day exposure window.
  • Observability (Phase 2, OBJ-2608, PRs #2421, #2485): execution-level Sentry capture around RailsServerSchema.execute, closing a gap where a resolver failure surfaced only as a GraphQL errors entry on an HTTP 200 with no alert. RSpec coverage pinning the corrected finding that a blank-username user has no reachable GraphQL crash path. A Playwright smoke spec asserting the restored Clerk-form username requirement directly, across all three browser projects.
  • Drift guard (Phase 3, OBJ-2609, PR #2420): scripts/check-clerk-auth-config.mjs + scripts/clerk-auth-config.expected.json + .github/workflows/clerk-auth-config-drift.yml — the ninth *-drift* guard in this repo and the first covering the auth provider. Runs daily against both instances' live GET /v1/environment, gates on the checked-in expected snapshot, and flags any prod/staging divergence not explicitly recorded with an owner issue.
  • Docs (Phase 4, OBJ-2610, PRs #2422, #2507): split a conflated authentication.md warning that this incident itself disproved; recorded Josh's password.required decision; documented the new governance rule — production Clerk auth-config changes go through a PR against the checked-in snapshot, never the dashboard alone; corrected a stale divergence claim that had been carried, unverified, across three separate artifacts.
  • Sign-in impact assessment (Phase 5, OBJ-2728, PR #2512): live captured traces (Playwright + direct FAPI probes) proving a username-less account signs in cleanly on both web and native under username.required = true. A planned completion-path fix and a planned smoke assertion for a lockout were both closed unbuilt once the evidence showed neither was needed. One CI regression test shipped instead, pinning the observed no-lockout behavior.

Phases ​

PhaseNameStatusPRsHighlights
1Restore the username invariant (live config)Shipped— (live config)username.required = true on both instances; RESTORE-5 found exactly 1 exposed account
2Make the invariant's violation visibleShipped2 (#2421, #2485)Sentry capture on unrescued GraphQL execution failures; Playwright smoke asserting the Clerk-form layer directly
3Clerk auth-config drift guard + staging parityShipped1 (#2420)Ninth *-drift* guard, first covering the auth provider; caught the original incident and a second unplanned one, live
4Docs closeShipped2 (#2422, #2507)Corrected warning, governance rule, incident record
5Sign-in impact on the username-less cohortShipped1 (#2512)Live traces: no lockout, no fix needed, net user impact zero

Key Decisions ​

  • Fix both instances, not just production. The kickoff assumed staging already held the correct value. It didn't — Josh's probe and an independent re-probe both found staging matched production's broken false. Restoring parity meant fixing staging too (RESTORE-6), which is also what made Phase 2's smoke spec meaningful rather than a disjunction that could pass without ever exercising the Clerk layer.
  • Keep password.required = false on production, deliberately. The original plan assumed restoring password requirement too, on a parity argument that turned out not to exist. Josh's written decision keeps it off: Objectuve supports passkey and OAuth as first factors, and a required password plausibly re-triggers the same OAuth missing_requirements dead-end this incident's original change was trying (and failing) to fix.
  • A gate, not a nag. Unlike the PostHog reverse-drift guard, the Clerk auth-config drift check has no legitimate-false-positive class — the expected snapshot is checked in, so any divergence is either a real out-of-band change or a stale snapshot, and both need a human. It runs daily rather than weekly, because a dashboard change is live the instant it's saved.
  • Don't build a fix for a problem that evidence shows doesn't exist. Phase 5 was scoped as a genuine question, not a foregone-conclusion fix. When live traces showed no lockout on either surface, the planned completion path (COHORT-2) and smoke assertion (COHORT-5) were both closed unbuilt, stated explicitly rather than quietly dropped.
  • Write criteria as properties of the restored system, not observations of the broken one. Three criteria across two phases were originally written to be proven by creating or observing a blank-username account — each became unsatisfiable the moment its own fix landed, because the fix is exactly what stops that account from existing. Restated as property assertions each time, and now a named repo gotcha so the next milestone doesn't rediscover the trap.
  • No shared integration branch. Each phase's surface was disjoint enough (a live config op, rails_api/, scripts/ + workflows, docs/) that bundling them into one branch would have gated a one-line observability fix behind a CI workflow review. Every phase PR based directly on master.

Requirements Coverage ​

5/5 — this milestone has no standalone REQUIREMENTS.md; coverage is derived from the ROADMAP's own "Definition of done" section (5 criteria), all fully satisfied. Three were restated mid-milestone from an observation-based form to a property-based form after the original wording became unsatisfiable — each restatement is documented in the ROADMAP with its reasoning at the point it happened. Full criterion-by-criterion detail: MILESTONE-AUDIT.md.

Outcomes ​

Production and staging both enforce username.required = true; a required-attribute divergence between the live Clerk config and its checked-in expected snapshot is caught within a day by a scheduled CI check rather than by a human noticing during an unrelated review. docs/architecture/authentication.md accurately describes the current state, the governance rule for changing it, and the milestone's own finding. The one production account exposed during the ~3.4-day incident window needs no correction — it signs in normally on both web and native, confirmed by live traces rather than assumed. No completion-path code shipped, because none was needed.

Tech Debt ​

  • (Phase 3, non-blocking) The drift guard's live-check job only runs on the daily cron, not on PRs (if: github.event_name != 'pull_request') — a config change made just after a cron run is invisible for up to 24 hours. A deliberate tradeoff (the check needs live network access), stated plainly rather than left to read as near-real-time coverage.
  • (carried forward, filed for Tess, non-blocking) A mid-milestone out-of-band change to staging's first_name.required/last_name.required broke two staging smoke fixtures that don't supply those fields. Diagnosed, not yet fixed — unscheduled work, not a novel mystery.
  • (carried forward, non-blocking) Whether Clerk's email_link-first-factor default (the source of a near-miss during Phase 5's own verification, where a stuck sign-in form looked exactly like a lockout and wasn't) applies to every user on the staging instance, or just the one test subject, is an open question — correctly left unanswered rather than guessed at.
  • (process note) Three ROADMAP-carrier PRs this milestone (#2477, #2507's sibling, #2509) merged with no reviewer assigned. Docs-only, no harm done, named for the retro rather than filed as its own ticket.
  • 06041bcbf64a31f57bdcfab6be515d7c1460fe69 — Phase 3, drift guard (#2420)
  • c5db4f8cb — Phase 2, VISIBLE-3 smoke spec (#2485)
  • 91c883c34 — Phase 4, docs close (#2507)
  • 0fc913bdd — Phase 5, COHORT-3 regression pin (#2512)
  • 3b6575313 — Phase 5 follow-up, authentication.md incident-record completion (OBJ-2768, #2511)
  • 0c636db5c — Phase 5 follow-up, password.used_for_first_factor added to the drift guard (OBJ-2769, #2510)

Last updated: 2026-08-19

Loading…