Deployment & Domain Configuration

Public uptime status: https://status.objectuve.com (BetterStack-hosted). See status-page.md for monitor map and incident workflow.

Environments

Environment	Frontend URL	API URL	Firebase Site	Cloud Run Service
Production	https://app.objectuve.com	https://api.objectuve.com	enkidu-app	enkidu-api-production
Staging	https://staging.app.objectuve.com	https://staging.api.objectuve.com	enkidu-app-staging	enkidu-api-staging
Marketing	https://www.objectuve.com	—	enkidu-marketing	—

Agent Runner (AI Workforce)

Environment	Cloud Run Service	Port	Auth
Production	enkidu-agent-runner-production	4001	OIDC (service-to-service)
Staging	enkidu-agent-runner-staging	4001	OIDC (service-to-service)
Local	docker-compose agent_runner	4001	Shared secret (AGENT_RUNNER_SECRET)

The Agent Runner is a standalone TypeScript Express service (agent_runner/) that executes AI employee runs using the Claude Agent SDK. It communicates with the Rails API via HMAC-authenticated webhook callbacks and routes LLM calls through LiteLLM.

Docker: agent_runner/Dockerfile (3-stage: deps, builder, runtime on Node 24 Alpine). Skills from .claude/skills/ are baked into the image at build time.

Environment variables (see rails_api/.env.example for Rails-side vars; see agent_runner/.env.example for ANTHROPIC_BASE_URL, SKILLS_DIR, MAX_RUN_BUDGET_USD): AGENT_RUNNER_URL, AGENT_RUNNER_SECRET, AGENT_RUNNER_AUTH_MODE, RAILS_WEBHOOK_SECRET, ANTHROPIC_BASE_URL, ANTHROPIC_API_KEY, SKILLS_DIR, MAX_RUN_BUDGET_USD, GITHUB_ACCESS_TOKEN, GITHUB_TARGET_REPO, MAILTRAP_SANDBOX_INBOX_ID, SLACK_WEBHOOK_URL.

Default URLs (always work, no custom domain required):

• Production frontend: https://enkidu-app.web.app
• Staging frontend: https://enkidu-app-staging.web.app
• Production API: https://enkidu-api-production-h26eaxs77q-uc.a.run.app
• Staging API: https://enkidu-api-staging-h26eaxs77q-uc.a.run.app

Branch & Release Model

feature/* → PR → master (auto-deploys to staging)
                    ↓
              git tag v3.5.0 → deploys to production (exact same image)

• master — only long-lived branch. Merges auto-deploy to staging.
• Release tags (v*) — trigger production deployments. The tagged commit's Docker image (already built for staging) is reused for production with no rebuild.
• PRs target master. Frontend-only preview environments are created per-PR.

There is no production branch. Production state is determined by which tag was most recently deployed.

Beta release cadence — weekly release train

During the open beta, production is gated to a weekly release train instead of shipping per-merge. The technical model above is unchanged (a v* tag still deploys production); what changes is when tags get cut:

• Merge stays continuous. Approved PRs merge to master all week and auto-deploy to staging — staging is always current for daily testing.
• Production tags are cut once a week: Wednesdays 08:00 America/Chicago, batching everything merged since the last tag into one release.
• Human-gated. The Multica Vicki — Weekly release train autopilot (Wed 08:00 CT) assembles the manifest (commits since the last tag, gate status, proposed version) and files a "Release train: week of …" issue assigned to Josh for go/no-go. On "go" it routes to Vicki, who runs release (the tag push auto-builds and auto-promotes Android to Play production — see Mobile Builds § Promoting to production), dispatches the manual iOS store build, and posts a second comment carrying paste-ready store copy for both stores — Play "What's new", App Store "What's New in This Version", and the App Review notes (see store-release-notes.md). A Tuesday 14:00 CT pre-flight autopilot surfaces blockers a day early. The train tags whatever is on master at fire time — there is no Tuesday code-freeze.
• Smoke gate status (OBJ-3917). The Tuesday pre-flight and the Wednesday go/no-go must both read node scripts/check-smoke-spec-provenance.mjs report for the current staging smoke picture, not raw gh run list conclusions on staging.yml. High deploy velocity means most post-merge runs are guard_stale-skipped and read identically green to a real pass (see gotchas.md's guard_stale entry) — the report subcommand classifies each spec as proven/unproven/unknown against a real green watermark instead. See Spec provenance: first real run vs. regression.
• Hotfix escape hatch. Issues labeled hotfix (sev-1 / security / prod-down) ship immediately (merge + release), bypassing the train.
• Break-glass, not a second gate. .github/workflows/scheduled-release.yml (Manual Aggregate Release) is workflow_dispatch-only — it no longer runs on a schedule (the weekday cron that duplicated the train's job was removed under OBJ-1368, resolving the "two unreconciled mechanisms" gap OBJ-1349 flagged). It's a manually-triggered recovery path for when the autopilot can't cut a release; RELEASE_TOKEN is optional for it and it never fires on its own, so it can't pre-empt the Wednesday go/no-go.
• Reverting to continuous CD. When the beta ends, delete the Vicki — Weekly release train autopilot and restore Vicki's per-ship release step. The train is the only thing gating production; nothing else in this model changes.

See docs/guides/multica-autopilots.md #7 / #7c and the Vicki section of docs/guides/multica-agent-crew.md for the runbooks.

Versioning & Releases

Version Source of Truth

The canonical version is the git tag name (e.g., v3.5.0). The version in ionic_frontend/package.json is informational and kept in sync by the release command.

Frontend version injection

__APP_VERSION__ is the global the frontend reads to render the side-menu version button (SideMenu.vue's .version-info), the changelog modal heading (ChangelogModal.vue), the Settings page version row (Settings.vue), and Sentry's release tag (ionic_frontend/src/sentry.ts:15). It's defined at build time in ionic_frontend/vite.config.ts:140 as a fallback chain:

__APP_VERSION__: JSON.stringify(
  process.env.VITE_APP_VERSION || process.env.npm_package_version || 'dev'
)

Build path	What sets VITE_APP_VERSION	Resulting value
Production (production.yml)	${GITHUB_REF_NAME#v} from the tag (e.g. v3.8.5 → 3.8.5)	bare tag, e.g. 3.8.5
Staging (staging.yml)	git describe --tags --always | sed 's/^v//' against master	e.g. 3.8.5-12-gabc123, or bare 3.8.5 if HEAD == tag
Local dev (npm run dev / npm run build without override)	unset — falls through to npm_package_version	the version field in ionic_frontend/package.json
No npm context (defensive)	both unset	literal 'dev'

The v prefix is stripped at the source so existing templates that render v{{ appVersion }} continue to display correctly (e.g. tag v3.8.5 → button text v3.8.5). Sentry's frontend release therefore reports the bare value (3.8.5); backend SENTRY_RELEASE is currently set with the prefix (v3.8.5) — see observability.md.

The staging job needs fetch-depth: 0 on its actions/checkout step for git describe to see tags; production reads GITHUB_REF_NAME directly and does not require a deep fetch.

package.json#version is a local-dev fallback only — CI is the source of truth for any deployed build. Engineers running npm run dev see whatever package.json says, and the release command keeps that value in rough sync, but production and staging never read package.json.

A smoke spec at ionic_frontend/tests/smoke-playwright/specs/platform/00-version-tag.spec.ts guards against drift: the production smoke run passes version to playwright-smoke.yml, which forwards it as SMOKE_VERSION_TAG; the spec asserts .version-info text equals v${tag}. Staging and PR-preview smoke runs leave the env var unset, and the spec skips cleanly.

Creating a Release

The canonical release entry point is scripts/release.sh (repo-tracked). A local release CLI alias wraps it for convenience.

Changelog requirement: Before running the release script, ionic_frontend/CHANGELOG.md must have at least one entry under ## [Unreleased]. The script enforces this with a guard.

Usage:

bash scripts/release.sh              # auto-bumps patch: v3.4.72 → v3.4.73
bash scripts/release.sh 3.5.0       # specific version: creates tag v3.5.0
bash scripts/release.sh 4.0.0       # major version: creates tag v4.0.0

Or via the CLI shortcut (if configured locally):

release          # auto-bumps patch
release 3.5.0   # specific version
release 4.0.0   # major version

The release process (5 steps):

1. Validate — Guards:
   • ## [Unreleased] section must have at least one entry (see CONTRIBUTING.md for format)
   • Must be on master branch
   • Local master must be up-to-date with origin/master
   • Tag must not already exist
2. Update changelog — Renames ## [Unreleased] to ## [VERSION] - DATE (e.g., ## [3.9.17] - 2026-04-30)
3. Insert fresh section — Adds a new empty ## [Unreleased] section after the changelog header
4. Commit & push — Commits the changelog change with message chore(release): update CHANGELOG for v$VERSION and pushes to origin/master
5. Tag & trigger deploy — Creates annotated tag v$VERSION and pushes it; tag push triggers production.yml workflow

Alternative: Direct tag without script

If you bypass the script and run git tag && git push directly, the tag will still trigger production deployment, but:

• The changelog will not be automatically updated (debt accumulates)
• CI emits a soft warning (non-blocking, for observability)
• Use the script to keep changelog in sync with releases

Help site changelog (/whats-new)

guide_site/whats-new.md (help.objectuve.com/whats-new) is a public, unauthenticated changelog — Milestone v4.26 Phase 6 shipped the page and linked it from the guide's nav; nothing in the pipeline keeps it current. A public changelog that stops updating is worse than never having shipped one, so this is a standing per-release obligation, not a one-time task:

• Source of truth: the same tagged ## [X.Y.Z] section of ionic_frontend/CHANGELOG.md the mobile store copy drafts from — see Mobile Store Release Notes §2 for the extraction command and the ordering gotcha (never draft before release has renamed [Unreleased] → [X.Y.Z] on the tagged commit).
• Cadence: every weekly release train tag that has at least one user-visible entry gets a new dated section (## Month DD, YYYY, newest first — see the existing sections in guide_site/whats-new.md for the exact shape) added in the same PR that carries any other guide content for that cycle, or as its own small follow-up PR if the train shipped no other guide work. A release with nothing user-visible (pure infra/CI/deps) does not need an entry — match the same "would a user notice" filter the CHANGELOG's own header comment uses.
• Voice: same house rules as the mobile store copy — .claude/skills/writing-release-notes/SKILL.md and docs/brand/brand.md. Lead with the user outcome, one bullet per change, merge related entries, no "AI" (say Coach). Unlike the store copy there is no hard character limit, but keep it scannable — this page has no pagination.
• Owner: Dori drafts the entry as part of the same docs pass that lands other v-next guide content; if a release has no other guide work queued, Maggie can route a standalone /whats-new update issue to Dori directly off the weekly release train issue. Vicki verifies the entry landed (or was deliberately skipped as a no-user-visible-change release) before closing out the train issue — same checkpoint spirit as the mobile store copy in Mobile Store Release Notes §1, just without a hard store-submission deadline forcing it.
• Not automated. No script generates or checks this page against the CHANGELOG — if that ever changes, update this section and store-release-notes.md together so the two don't drift apart on source-of-truth rules.

What Happens on Tag Push

1. Validate: The workflow resolves the tagged commit's SHA and waits for rails-api:{sha} to appear in Artifact Registry — polling every 20 seconds for up to 5 minutes (300 seconds). If the staging build is still in flight when the tag lands, validate waits it out instead of failing fast. See Staging build and production validate below.
2. Tag image: Adds the version tag to the existing image (gcloud artifacts docker tags add) — no rebuild
3. Deploy backend: Runs migrations, deploys the validated image to Cloud Run
4. Deploy frontend: Builds the frontend from the tagged commit, deploys to Firebase
5. Smoke tests: Runs desktop and mobile viewport tests
6. GitHub Release: Creates a release with auto-generated notes

.github/workflows/mobile-android-release.yml also triggers on the same tag push, independently of production.yml — it builds a signed direct-install APK, uploads it to a private GCS bucket, and posts a 12-hour signed download URL to the GitHub Release notes. See Mobile Builds for details.

Production deploy safeguards

The production workflow includes two structural safeguards (introduced in OBJ-349) to prevent silent deployment failures:

1. Tag must be on master

The validate job verifies that the tagged commit is an ancestor of origin/master before attempting any GCP operations. This prevents a class of failures where tags are placed on pre-squash feature-branch commits (which have no Docker image) and cascade into silent deploy failures.

Error output (if tag points to an off-master commit):

::error::Tag v3.5.0 points to commit abc1234..., which is not an ancestor of origin/master.
::error::staging.yml only builds images for master pushes, so no Docker image exists for this SHA. The deploy cannot succeed.
::error::Likely cause: tag was placed on a pre-squash feature-branch commit instead of the squash-merge commit on master.
::error::Fix: delete the tag, identify the correct master SHA (usually the squash-merge commit for the PR), and re-tag using scripts/release.sh (which guarantees the tag lands on a master commit).

Workflow: .github/workflows/production.yml, lines 37–46 (the Verify tag is on master step in the validate job).

2. Fail-fast image polling with clear recovery path

The image validation step now times out after 300 seconds (5 minutes) instead of silently camping on the concurrency slot for 15 minutes. When an image is missing, the error message points operators to the recovery runbook instead of leaving them guessing.

Error output (if image is missing after 300s):

::error::Image us-central1-docker.pkg.dev/.../rails-api:abc1234... not found after 300s.
::error::Likely cause: staging.yml build for abc1234... is still in flight, failed, or was cancelled. The 'tag must be on master' guard already ran, so this SHA *is* on master — the image just isn't published yet.
::error::Recovery: 1) Check the staging.yml run for this SHA in GitHub Actions. 2) If failed/cancelled, re-run staging via workflow_dispatch with commit_sha=abc1234... to rebuild. 3) Once the image lands in Artifact Registry, delete and re-push tag v3.5.0 to retrigger this workflow. See docs/operations/deployment.md for the full runbook.

Workflow: .github/workflows/production.yml, line 90 (the TIMEOUT=300 setting) and lines 109–112 (the error block) in the Resolve and validate image step.

Context: Both safeguards address failure modes discovered in OBJ-348, where 12 production deploys failed silently because of off-master tags and long-running image polls that held concurrency slots. See Staging build and production validate for the manual recovery runbook if safeguards trigger.

3. Pre-deploy secrets lint (scripts/validate-secrets.sh)

The validate job in .github/workflows/production.yml and the build-rails-api job in .github/workflows/staging.yml both run bash scripts/validate-secrets.sh before any image push or deploy step. The script catches the OBJ-452 class of bug: a Rails ENV[…] reference wired into one --set-secrets block (e.g., the main service deploy) but forgotten in a parallel block (e.g., the db-migrate job). When that happens, the deploy succeeds for some Cloud Run resources and fails mid-rollout for others, leaving production on a stale revision and forcing manual recovery (see CHANGELOG v3.9.187 for the original incident).

What it checks:

• Cross-block consistency (always runs, no auth required): every env var present in any --set-secrets block targeting the rails-api image must be present in every rails-api --set-secrets block in the same workflow. The litellm-proxy block is excluded — it targets a different image and a different secret surface.
• Secret Manager existence (runs only when gcloud is authed): every secret name referenced on the right-hand side of = in any --set-secrets block must exist in GCP Secret Manager. When gcloud is not authed, the script prints a warning and the cross-block check still runs.

How to extend the allowlists: the script has four lists near the top:

• ALLOWLIST — Rails-internal env vars supplied via --set-env-vars (e.g., RAILS_ENV, DATABASE_URL, REDIS_URL). These never appear in --set-secrets and are exempt from all checks.
• ALLOWLIST_PREFIXES — prefix matches for runtime-only vars (BUNDLE_*, RUBY_*, VITE_*, etc.).
• DEPLOY_ONLY — env vars that are intentionally only on the main HTTP-serving gcloud run deploy block and NOT on the db-migrate / demo-reset job blocks. These back request-path code (e.g., STRIPE_SECRET_KEY, STRIPE_WEBHOOK_SECRET, RAILS_WEBHOOK_SECRET) that never executes during rails db:migrate or rake demo:reset. See docs/operations/ci-secrets-checklist.md for the rationale. Adding a name here exempts it from the cross-block consistency check; the Secret Manager existence check still applies.
• WIRING_ONLY (OBJ-1339) — env vars consumed exclusively by rake stripe:wire_prices (rails_api/lib/tasks/stripe.rake), never by request-path code: STRIPE_PRICE_TEAMS_MONTHLY, STRIPE_PRICE_TEAMS_ANNUAL, STRIPE_PRICE_SUPPORTER_MONTHLY, STRIPE_PRICE_SUPPORTER_YEARLY. These are wired only into the dedicated enkidu-stripe-wire-prices job's --set-secrets block (see §4 below), so cross-block presence is meaningless for them — the opposite pattern from DEPLOY_ONLY (present on the main block, absent from job blocks). Do not overload DEPLOY_ONLY with these; the two lists mean different things.

All four lists are case-sensitive, UPPER_SNAKE_CASE only.

If the validate step fails: the error message names the env var and the workflow line(s) missing it. Fix by either (a) adding the env var to the omitted --set-secrets block, (b) creating the missing GCP secret with gcloud secrets create <name> --project=enkidu-488723 --data-file=-, or (c) if the var is intentionally deploy-only or wiring-only, adding it to DEPLOY_ONLY/WIRING_ONLY with a comment explaining why.

Workflow: .github/workflows/production.yml (validate job); .github/workflows/staging.yml (build-rails-api job).

Zero-presence gap, closed (OBJ-1340 T2): the cross-block check used to only catch a var missing from some but not all --set-secrets blocks; a var wired into zero blocks anywhere was invisible to it — exactly how STRIPE_PRICE_TEAMS_* shipped unwired to production undetected the first time (see §4 below). scripts/validate-secrets.sh now closes that blind spot with check_zero_presence(), called from main(). It scans rails_api/app and rails_api/lib (BARE_FETCH_DIRS) for bare ENV.fetch('VAR') calls with no default and no fallback block (collect_bare_env_fetches()), then fails the lint for any such var that is wired into neither a workflow --set-secrets/--set-env-vars payload nor a deploy/*.yaml manifest.

A repo-wide deploy lint cannot be tightened ahead of the wiring it enforces. validate-secrets.sh lints both production.yml and staging.yml in one run and is a hard gate on both deploys (production.yml's validate job, staging.yml's build-rails-api job). A lint change that starts requiring a var before that var is actually wired into every block it now checks will fail the very next deploy of either environment — including staging, even if only production's wiring was the actual gap. This took staging down on 2026-07-11 (PR #1448, reverted 5 minutes after merge): a lint tightening that correctly demanded STRIPE_PRICE_TEAMS_*/STRIPE_PRICE_SUPPORTER_* in production's blocks shipped in the same PR as the demand itself, with production not yet wired — the very next staging deploy hit the same unmet demand and failed. Any future lint tightening must land in the same PR/commit as the wiring that satisfies it, never ahead of it — run bash scripts/validate-secrets.sh locally against the branch before opening the PR, not just in CI.

4. Stripe price wiring — the burned-migration trap (OBJ-1339)

Plan#stripe_price_id (Supporter monthly/yearly, Teams monthly/annual) is populated from STRIPE_PRICE_* env vars by the rake task stripe:wire_prices (rails_api/lib/tasks/stripe.rake) — the sole supported wiring mechanism. It is slug-scoped and idempotent:

# Wire only the given slugs — required for production (see below)
bundle exec rake 'stripe:wire_prices[teams_monthly,teams_annual]'

# Omit the slug list to wire all four (monthly, yearly, teams_monthly, teams_annual)
bundle exec rake stripe:wire_prices

For each requested slug it reads the matching STRIPE_PRICE_* var, retrieves the price from Stripe, and update!s the Plan row — but only after every requested slug has passed two checks up front, so a run either fully succeeds or writes nothing:

• Fails loudly, not silently. An unset var or a missing Plan row exits non-zero naming the var/slug, instead of falling back to a stale value.
• Cross-mode guard (F5). Every price is retrieved from Stripe and its livemode is checked against the active STRIPE_SECRET_KEY's mode before any row is written. A test-mode price can never land on a Plan row read by a live key, or vice versa — this is what makes the shared-Secret-Manager footgun below non-catastrophic even if a secret is ever mis-wired.
• Out-of-scope plans are never touched — not read, not written. Requesting [teams_monthly,teams_annual] never looks at the Supporter rows at all, which is what makes it safe to run against production without the Supporter env vars ever being set there.

Why a rake task and not a migration. The wiring originally shipped as a migration (20260708000000_wire_test_mode_stripe_prices.rb), using an ENV.fetch(var, existing_value) pattern. A Rails migration only ever executes once per recorded version: if it runs before the STRIPE_PRICE_* secret is wired into the executing job's --set-secrets block, ENV.fetch falls back to the existing (nil) value, the migration still records as applied, and no amount of fixing the env vars afterward makes it re-apply — a new migration version becomes the only way to retry. This happened once already on staging. Both original migrations are now permanently neutralized to explicit no-ops (rails_api/db/migrate/20260708000000_wire_test_mode_stripe_prices.rb, .../20260708130000_retry_wire_test_mode_stripe_prices.rb) — their up/down bodies are single-line comments, verified by a dedicated spec that instantiates each migration class directly with all four vars set and asserts nothing writes. They are not "fixed" or "reactivated" as a future troubleshooting step; stripe:wire_prices is the only path that ever writes Plan#stripe_price_id now.

Staging and production share one GCP Secret Manager project, with no environment: scoping. Both workflows read from the same enkidu-488723 project via a single repo-level GCP_PROJECT_ID var. This means an environment-specific value needs an environment-distinct secret name — there is no other boundary:

Secret	Mode	Read by
stripe-price-teams-monthly / stripe-price-teams-annual	test-mode	staging
stripe-price-teams-monthly-live / stripe-price-teams-annual-live	live-mode	production

Never put a live-mode price ID into the unsuffixed secrets — staging reads those, and doing so would make staging checkout charge real cards. This is a live-money footgun, not a style preference: the cross-mode guard above is the code-level backstop for exactly this mistake, but the secret-naming discipline is the first line of defense.

Production execution. A dedicated Cloud Run job, enkidu-stripe-wire-prices, is created/updated on every production deploy (.github/workflows/production.yml's deploy-backend job) but is never auto-executed — there is no gcloud run jobs execute step for it in the workflow. Wiring live Teams prices is a one-time op; auto-running it on every deploy would make the production deploy path depend on Stripe API reachability for no ongoing benefit. Run it manually after a deploy that includes this job has completed:

gcloud run jobs execute enkidu-stripe-wire-prices --region us-central1 --wait

The job's --set-secrets carries STRIPE_SECRET_KEY=stripe-secret-key:latest plus the two live-only Teams price secrets above — it does not wire STRIPE_PRICE_SUPPORTER_* anywhere; production's Supporter prices are already live via the earlier 20260428232539_update_plan_prices migration and must never be overwritten by this job. Its --args uses the ^|^ alternate-delimiter syntax (--args "^|^exec|rails|stripe:wire_prices[teams_monthly,teams_annual]") because gcloud splits --args on commas by default, which would otherwise mangle the rake slug list's internal comma into two broken arguments.

Ordering constraint. The teams_monthly/teams_annual Plan rows are seeded by 20260704000010_seed_team_plans.rb, which merged after the production tag active at the time this job shipped (v3.13.4, cut 2026-07-03). Until a production tag runs db:migrate past that point, those Plan rows don't exist in production and stripe:wire_prices will exit 1 cleanly (missing Plan row) rather than corrupt anything — safe to attempt early, just won't succeed until seeding has happened. Run the job only after a deploy has both shipped the job definition and run db:migrate past the seeding migration.

See Phase 7 Teams § Billing & provisioning for the full incident history and the Teams billing runbook for the on-call procedure if a price-wiring gap ever surfaces as a live incident.

Viewing Releases

• GitHub UI: Repository > Releases
• CLI: gh release list or gh release view v3.5.0
• Tags: git tag --sort=-v:refname | head -10
• What's deployed: deployed (fish function)

CI/CD Pipelines

Workflow	Trigger	File
CI (Tests + Lint)	PR to master, push to master	.github/workflows/ci.yml
Staging Deployment	Push to master	.github/workflows/staging.yml
Production Deployment (API + consumer app)	Tag push (v*)	.github/workflows/production.yml
Production Rollback	Manual dispatch	.github/workflows/rollback.yml
Admin Dashboard Deployment	Push to master (admin_dashboard/**)	.github/workflows/admin.yml
Marketing Site Deployment	Push to master (marketing_landing/**)	.github/workflows/marketing.yml
Docs Deployment	Push to master (docs/**, docs_site/**)	.github/workflows/docs.yml
Storybook Deployment	Push to master (ionic_frontend/src/**)	.github/workflows/storybook.yml
Design System Deployment	Push to master (design_system/**)	.github/workflows/design-system.yml
API Docs Deployment	Push to master (rails_api/app/graphql/**)	.github/workflows/api-docs.yml
Preview (frontend-only)	PR open/sync/reopen	.github/workflows/preview.yml
Preview Teardown	PR closed	.github/workflows/preview_teardown.yml
Android Build & Deploy	Manual dispatch	.github/workflows/mobile-android.yml
Android Direct-Install Release	Tag push (v*), manual dispatch	.github/workflows/mobile-android-release.yml
iOS Build & Deploy	Manual dispatch	.github/workflows/mobile-ios.yml
Smoke Tests (desktop + mobile viewports)	Called by production.yml	.github/workflows/playwright-smoke.yml

See Mobile Builds for full mobile CI/CD documentation.

Deployment Flow (Staging)

1. Code is pushed to master
2. Docker image is built and pushed to Artifact Registry (rails-api:{sha} + rails-api:staging-latest)
3. Database migrations run via Cloud Run job
4. Rails API deployed to Cloud Run (enkidu-api-staging)
5. Frontend built with staging env vars, deployed to Firebase (enkidu-app-staging)
6. release-health-watch job (staging.yml) polls the Sentry Issues API for new, unresolved, release-attributed errors — see Release-health watch coverage window below

Run-level serialization (OBJ-2751): the pipeline above now queues at the run level, not just per job — a workflow-level concurrency: { group: staging-pipeline, cancel-in-progress: false, queue: max } (staging.yml:13-31) means a new push's migrate/deploy-* jobs can't start until the previous push's playwright-smoke job reaches a terminal state. This closes a race where a later push's environment-mutating jobs could corrupt an earlier push's still-running smoke validation — see the gotcha for the incident. Cadence implication: on a high-merge-cadence day, pushes now queue instead of overlapping — each one waits for the full previous pipeline (nominal ~10–15m, up to ~45m degraded) rather than only its own deploy. build-rails-api/build-agent-runner gained a guard_stale step so a burst of queued pushes still drains cheaply for every commit except the tip.

Migrate-to-deploy exposure window (OBJ-3779): even within one hard-sequenced run (deploy-backend: needs: [..., migrate]), db:migrate finishing and deploy-backend reaching 100% traffic are two different timestamps — a real gap (~90s–5m depending on what else runs in between) where the schema has changed but the previous, still-warm revision keeps serving on connections holding server-side prepared plans for the old shape. This is not a guard_stale cross-run race; it's a structural property of any migrate-before-deploy pipeline, present on production too. verify-migration-seed-data (staging.yml:565) was moved off the migrate job's critical path specifically to shrink this window on staging; the app-level fix that actually closes it for users is the bounded retry in GraphqlController#execute_schema — see the full gotcha write-up.

Release-health watch coverage window

After a staging deploy, staging.yml's release-health-watch job runs scripts/check-release-health.mjs on a poll loop to catch a new, unresolved, release-attributed Sentry error before it's discovered live. Its coverage is deploy+0 → deploy+RELEASE_HEALTH_WINDOW_MINUTES (default 90 minutes), anchored to the workflow run's own start time — not to when the job itself starts, which only happens once the upstream needs: chain (build → migrate → deploy → smoke) finishes. Anchoring to deploy time (rather than job-start time) keeps the window a stated, tested property instead of an emergent one that silently shrinks if the smoke suite gets faster (OBJ-2267). A job that starts after its own deadline logs ::warning:: and exits 0 rather than reading as a silent clean; see the gotcha for the full incident history and the workflow-level assertion that enforces this. Live coverage remains blocked on provisioning SENTRY_ISSUES_API_TOKEN — until then the job fail-softs on every run.

Deployment Flow (Production)

1. Developer pushes a version tag (git tag v3.5.0 && git push origin v3.5.0 or release)
2. Workflow waits for rails-api:{sha} to land in Artifact Registry (20s poll, 5 min timeout (300s)) — if staging is still building, validate waits rather than failing fast
3. Version tag added to existing image — no Docker rebuild
4. Database migrations run via Cloud Run job
5. Rails API deployed to Cloud Run (enkidu-api-production)
6. Consumer app built from tagged commit, deployed to Firebase (enkidu-app only)
7. Smoke tests run (desktop + mobile)
8. GitHub Release created

A release tag ships the Rails API and the consumer app, and nothing else. The admin dashboard, marketing site, docs, Storybook, design system, and API docs all have their own lanes — see Independent surface lanes.

production.yml:169-217 has the identical deploy-backend: needs: [..., migrate] shape as staging, so production carries the same migrate-to-deploy exposure window described in Deployment Flow (Staging) above — with real users on it. Nothing in this lane shrinks the window the way staging's verify-migration-seed-data move does; the app-level retry in GraphqlController#execute_schema is what covers production, since it ships everywhere the code does. See the gotcha write-up.

Independent surface lanes

Not everything in this monolith is the consumer app, and not everything needs the consumer app's cadence. Production tags are batched to a weekly release train; a docs correction or an admin-only table column has no reason to wait a week for it.

Eight surfaces therefore deploy on merge to master, path-filtered, independent of any v* tag:

Surface	Workflow	Hosting target	Trigger paths	Gate before deploy
Admin dashboard	admin.yml	enkidu-admin	admin_dashboard/**, design_system/**	Schema compat vs live production API, then production Playwright smoke after deploy
Marketing site	marketing.yml	enkidu-marketing	marketing_landing/**	Contrast + PNG legibility guards on PR; HTTP 200 check after deploy
Docs	docs.yml	enkidu-docs	docs/**, docs_site/**	PR build/lockfile validation + schema compat vs live production API
Guide	guide.yml	enkidu-guide	guide_site/**	PR build/lockfile validation
Feedback board	feedback.yml	enkidu-feedback	feedback_site/**, design_system/**, firebase.json, .firebaserc	PR build/lockfile validation + codegen drift check. No live-schema-compat gate (unlike Admin/Docs) — see the follow-up note below.
Storybook	storybook.yml	enkidu-storybook	ionic_frontend/src/**, .storybook/**	—
Design system	design-system.yml	enkidu-design	design_system/**	PR build validation + PR-preview deploy (pr-{number} channel)
API docs	api-docs.yml	enkidu-api-docs	rails_api/app/graphql/**, schema.graphql	PR generation validation

Two rules hold this together.

1. One Hosting target, one writing workflow. Firebase Hosting has no notion of commit ordering — the deploy that finishes last wins, even if it built an older commit. Two workflows publishing the same target race by construction, and no staleness guard can fix it because each lane only knows about its own runs. This is why production.yml no longer deploys enkidu-admin or enkidu-marketing, and why re-adding either would be a regression. ci.yml's lint-workflows job enforces it ("Guard against two workflows deploying one Firebase Hosting target"). rollback.yml is the one documented exemption: it is break-glass, triggered by a human instead of the forward lanes, never alongside them.

2. Every lane guards against stale commits. Each lane re-checks origin/master's tip against its own github.sha immediately before calling firebase-tools deploy, and skips if superseded (the guard_stale step, same pattern as staging.yml — see Staging deploy race). Without it, two merges landing a minute apart both build and deploy, and the older one can land last. Any new lane needs the same step.

The admin schema-compat gate

Admin was the first independent surface with a real gate for its API dependency — Docs later gained the same class of gate (see below), and Feedback board currently has neither (see The feedback site has no schema-compat gate). scripts/check-admin-schema-compat.mjs extracts every gql document under admin_dashboard/src/, introspects the live production schema at https://api.objectuve.com/graphql, and validates each operation against it. If admin reaches for a field or argument production does not serve yet, the lane fails and the change has to ride the release train instead — which deploys the API first.

Note it validates against what production serves, not against rails_api/schema.graphql at HEAD: HEAD's schema routinely contains merged-but-unreleased fields, and shipping admin against those is precisely the v4.0.0 skew failure this gate exists to prevent.

Two implementation details worth knowing before you touch it:

• The introspection query is hand-written and depth-reduced. graphql's stock getIntrospectionQuery() nests ofType seven deep for a depth-15 query, which RailsServerSchema's max_depth 10 rejects outright. Four ofType levels put the deepest path at exactly 10. Adding a level will start failing every run against production; assertNoTruncatedTypeRefs catches the opposite problem (a wrapper type nested deeper than the query can follow) rather than silently building a wrong schema.
• --offline falls back to rails_api/schema.graphql as of the last v* tag, for when introspection is unreachable. It is opt-in and prints a warning, because the last tag is a proxy for what's deployed — a hotfix deployed outside the train wouldn't show up.

On PRs the check runs advisory (continue-on-error), because an admin change may legitimately land in the same PR as the API change it needs. On master pushes it hard-blocks, because that is the run that would actually deploy.

The docs schema-compat gate

Docs gained the same class of dependency once docs_site's "Ask the docs" widget started calling askDocsQuestion — and the same class of incident: PR #3097 (f58e1ba43, OBJ-3623) merged the widget and the GraphQL field it calls together, docs.yml deployed the widget to production within minutes on its path-filtered master-push lane, but the field itself only ships with the Rails API on the next weekly v* tag. Production was still v4.8.3 when the widget went live, so every question returned Field 'askDocsQuestion' doesn't exist on type 'Query' (extensions.code: undefinedField) — see Frontend/API deploy skew and migration-model coupling for the full incident writeup.

scripts/check-docs-schema-compat.mjs closes this the same way admin's gate does: it introspects the live production schema and validates every docs_site GraphQL operation against it, hard-blocking deploy-docs (needs: schema-compat) on master push and running advisory (continue-on-error) on PRs, for the same reason as admin — a docs change may legitimately land in the same PR as the API change it needs.

It is a near-duplicate of check-admin-schema-compat.mjs, adapted rather than generalized into a shared utility — this repo's pattern is a per-surface guard script, and two consumers doesn't justify an abstraction. The one real difference: docs_site has no graphql/Apollo runtime dependency, so its one operation (docs_site/.vitepress/theme/askDocsQuestion.ts:51-59, AskDocsQuestion) is a bare backtick template rather than a gql-tagged one. The extractor handles both shapes — a gql-tagged document, or a bare template whose body opens with query/mutation/subscription/fragment — so it finds AskDocsQuestion without askDocsQuestion.ts needing to add a graphql/graphql-tag dependency it otherwise has no use for. Like admin's gate, it resolves the graphql package itself out of admin_dashboard's node_modules (both CI jobs run npm ci --legacy-peer-deps in admin_dashboard/, not docs_site/, for exactly that reason), and exit code 2 (could-not-run — unreachable schema, zero documents found, a parse failure) fails the gate the same way it does for admin: never treated as a pass.

The feedback site has no schema-compat gate (a known gap)

feedback_site/ (Feedback board row above) consumes the API the same way admin and docs do — it calls publicFeedbackPosts/publicFeedbackPost and the authenticated createFeedbackPost mutation via Apollo — but feedback.yml's validate job only runs a build + npm run codegen:check (codegen drift against rails_api/schema.graphql at HEAD, not against what production actually serves). There is no scripts/check-feedback-schema-compat.mjs. This is the exact shape the general rule in the gotchas doc warns against — "any independent surface lane that consumes the API needs its own schema-compat gate before it deploys on master push" — and, in practice, low-risk today only because the read path it depends on is still behind the PUBLIC_FEEDBACK_READ_ENABLED kill switch: a schema-skew bug here would currently ship silently dark rather than to real traffic. That stops being true the moment the kill switch flips on. Closing this gap (a check-feedback-schema-compat.mjs mirroring admin's/docs') is Codi/Riley work, not resolved here.

Rolling back an independent lane

These lanes are not covered by rollback.yml, which handles the API and consumer app. Roll one back by re-running its workflow via workflow dispatch with the ref input set to the last-good master SHA:

gh workflow run admin.yml -f ref=<previous-master-sha>

gh workflow run marketing.yml -f ref=<previous-master-sha>

The guard_stale step deliberately skips itself for workflow_dispatch runs — deploying a ref that is not master's tip is the entire point of a rollback. Note this rebuilds from source rather than re-publishing a stored artifact, so the rollback is only as reproducible as that commit's dependency tree.

Staging build and production validate

The staging workflow splits the Docker build from the deploy steps so that a release tag pushed right after a merge can never race a still-running staging build.

How staging.yml is structured

• build-rails-api — builds and pushes rails-api:{sha} to Artifact Registry. Uses concurrency: { group: staging-build-${{ inputs.commit_sha || github.sha }}, cancel-in-progress: false }. Per-SHA grouping plus non-cancellable means an in-flight build for SHA-A is never cancelled when SHA-B lands; both images get published. Between build and push, a boot-smoke step (OBJ-3427) runs the freshly built image's own config/puma.rb against a trivial rack app (rails_api/config/ci/puma_boot_smoke.ru, no DB/secrets needed) and blocks the push if Puma can't get past config-file evaluation — this is what would have caught OBJ-3424's .present? regression. On failure, the step dumps docker logs for the smoke container into the job log before exiting 1.
• migrate — runs rails db:migrate via the enkidu-db-migrate-staging Cloud Run job. Split out of deploy-backend into its own job so a migration failure blocks the deploy via needs: instead of failing mid-job. Gated with if: github.event_name == 'push', needs: [build-rails-api, build-agent-runner], and its own concurrency: { group: staging-migrate, cancel-in-progress: true } plus a guard_stale step (mirroring deploy-backend's) so a superseded commit never kicks off a redundant concurrent db:migrate run — cancel-in-progress: true only cancels the local runner, not an in-flight Cloud Run Job execution already kicked off on GCP's side.
• deploy-backend, deploy-litellm, deploy-frontend — gated with if: github.event_name == 'push' and grouped on staging-deploy-{backend,litellm,frontend}. deploy-litellm (single gcloud run deploy) keeps cancel-in-progress: true. deploy-backend and deploy-frontend switched to cancel-in-progress: false (OBJ-1992) because each publishes more than one resource per run — deploy-backend runs 3 sequential gcloud run services replace calls, deploy-frontend runs one firebase-tools deploy across 2 hosting targets — and a mid-sequence SIGTERM from cancel-in-progress: true could leave those resources split across two commits' builds. An in-flight deploy on either job now always finishes as one internally-consistent publish; staleness is instead handled by each job's own guard_stale step plus GitHub's default single-pending-slot queue. See Staging deploy race for the full history.
• deploy-backend declares needs: [build-rails-api, build-agent-runner, migrate], so it only starts deploying once migrations have completed. deploy-frontend declares needs: deploy-backend (previously unblocked), so a migration or backend deploy failure now blocks the frontend rollout instead of leaving it live against a schema the API hasn't migrated to yet.
• seed-staging-data (OBJ-1942) — runs the three data-seeding steps (Reset Demo Account, Seed Goal Templates, Ensure Preview Smoke Account) that used to run serially inside deploy-backend, adding ~7.2m to its critical path. Declares needs: [build-rails-api, deploy-backend] (only needs the built image and a live API to seed against, same as deploy-backend itself), so it runs off deploy-backend's critical path instead of inside it, letting deploy-frontend start sooner. playwright-smoke's needs: and stale-guard were updated to include it, so smoke can't start before seeding finishes.

The staging-latest tag is only re-pointed on push events, never on workflow_dispatch rebuilds.

How production.yml validates

The Validate Release Image step polls Artifact Registry for rails-api:{sha} every 20 seconds for up to 5 minutes (300 seconds):

• Image already there (fast path) — loop exits on the first check, no extra log noise.
• Image not yet there — emits a single ::notice::Waiting for rails-api:{sha} and keeps polling. On success, logs the elapsed wait so the timeout can be tuned over time.
• Image still missing after 5 min (300s) — emits ::error:: with a recovery instruction pointing at the workflow_dispatch rebuild path below.

Tagging a release immediately after merging to master is now safe — production validate will wait for staging's build to finish.

Manual rebuild path

If a staging build never produced an image (cancelled, failed, or otherwise), trigger a rebuild without a no-op commit:

1. Open the Staging Deployment workflow in the GitHub Actions UI.
2. Click Run workflow and provide commit_sha — the SHA you need rebuilt.
3. The dispatched run executes only build-rails-api. It checks out the requested SHA, builds the image, runs the same boot-smoke check described above, and pushes it as rails-api:{commit_sha}. Deploy jobs are skipped, and staging-latest is not touched. If the smoke check fails, the rebuild stops before push — check the job log's docker logs dump for the crash.
4. Once the image lands, re-tag the release: git tag -d vX.Y.Z && git push origin :refs/tags/vX.Y.Z && git tag -a vX.Y.Z -m "Release X.Y.Z" <commit_sha> && git push origin vX.Y.Z.

Background

This structure was introduced in OBJ-73 after the v3.8.3 (Apr 27, 2026) and v3.8.4 (Apr 28, 2026) tags landed unbuildable. In both cases, a release was tagged within seconds of a master merge, and the next merge to master cancelled the in-flight rails-api:{sha} build via the workflow-level cancel-in-progress: true policy. The image never got pushed, the production validate step failed fast on the missing image, and the only recovery was to retag at a later SHA.

PR Previews

PR previews cover two independent surfaces, each with its own site and workflow:

• Consumer app (enkidu-app) — .github/workflows/preview.yml. The frontend is built against the staging API (staging.api.objectuve.com) and deployed to a Firebase preview channel (pr-{number}, expires in 14 days). No per-PR Cloud Run service or database is created.
• Design system (enkidu-design) — .github/workflows/design-system.yml's deploy-design-system-preview job (OBJ-2158). Any PR touching design_system/**, firebase.json, or .firebaserc builds design_system and deploys it to a Firebase preview channel on enkidu-design, also pr-{number} with a 14-day expiry, gated behind the same validate-design-system build check and VITE_DOCS_PASSWORD_HASH gate as production. A PR that doesn't touch those paths never triggers the job, so no channel is created.

Both lanes reuse the existing GCP_SERVICE_ACCOUNT / GCP_WORKLOAD_IDENTITY_PROVIDER secrets — no separate credentials per site.

Preview Channel Lifecycle

Channel Creation:

• enkidu-app — When a PR is opened, synchronized, or reopened, .github/workflows/preview.yml runs:
  1. Builds the frontend with staging API URL and Clerk test keys
  2. Authenticates to Google Cloud via OIDC workload identity
  3. Deploys to Firebase preview channel pr-{number} with --expires 14d
  4. Comments the preview URL on the PR
• enkidu-design — When a PR touching design_system/** (or firebase.json/.firebaserc) is opened, synchronized, or reopened, .github/workflows/design-system.yml's deploy-design-system-preview job runs the same build → authenticate → deploy → comment sequence, scoped to --only enkidu-design.

Channel Teardown — When a PR is closed, .github/workflows/preview_teardown.yml runs:

1. Authenticates to Google Cloud via OIDC
2. Deletes the pr-{number} channel on enkidu-app via firebase hosting:channel:delete
3. Deletes the pr-{number} channel on enkidu-design the same way
4. Treats "not found" errors (channel already deleted, or never created because the PR never touched design_system/**) as success, not failure

Hourly Cleanup Sweep — A scheduled workflow (.github/workflows/cleanup-preview-channels.yml) runs hourly (0 * * * *, also available via workflow_dispatch) as a backstop, looping over both enkidu-app and enkidu-design independently:

1. For each site, queries firebase hosting:channel:list for all channels on that site
2. For each channel matching pr-<number>:
   • Checks if the GitHub PR is closed/merged (404 = PR deleted, treat as closed)
   • Checks if the channel's 14-day TTL has expired
   • Marks the channel for deletion if either condition is true
3. Deletes marked channels
4. Logs a per-site summary of deletions, then fails the run with ::error:: if that site's post-sweep pr-* channel count is still ≥40 (80% of its quota) — a loud, specific signal that teardown/cleanup isn't keeping pace for that site, instead of a silent creep into the next 429 outage. Each site's threshold is checked independently, and a failure on one site never masks or is masked by the other (OVERALL_FAIL is only evaluated after both sites' loops complete).

This sweep prevents quota exhaustion if the on-close teardown workflow fails or misses a PR (e.g., GitHub event delivery issues). It was daily until OBJ-1337 — a full day's backlog of closed-PR stragglers was enough on its own to exhaust enkidu-app's quota and 429 every open PR's preview deploy before that day's single run had fired. Hourly caps the backlog at ~1 hour of churn instead of ~24.

Firebase Quota — Firebase Hosting has a 50-channel-per-site limit, and it applies per site — enkidu-app and enkidu-design each get their own 50-channel budget, not a shared pool. The hourly cleanup sweep keeps each site's quota well below its own cap by pruning stale pr-* channels automatically, and fails loudly (see above) if either site's count still creeps toward it.

Runbook: "Deploy Frontend Preview" Workflow Fails with HTTP 429

Symptom: The GitHub Actions Deploy Frontend Preview step (enkidu-app, preview.yml) or Deploy Design System Preview step (enkidu-design, design-system.yml) fails with:

HTTP Error: 429, Couldn't create channel on `projects/.../sites/enkidu-app`: channel quota reached.

(or the same error with sites/enkidu-design). The quota guard applies per site — enkidu-app and enkidu-design each have their own independent 50-channel budget, so one site 429ing doesn't imply the other is anywhere near its cap.

Resolution:

1. Immediate: Trigger the cleanup sweep manually:

   • Go to Actions → Cleanup Preview Channels workflow
   • Click Run workflow → toggle dry_run to false → Run workflow
   • Wait for the run to complete; it sweeps both enkidu-app and enkidu-design. Channels matching closed PRs and expired TTL will be deleted on whichever site(s) need it.

2. Verify quota cleared: Once the sweep finishes, the affected site's channel count should drop below 30 (of its own 50/site quota). The next preview deploy on any PR touching that surface will succeed.

3. Post-incident: No further action needed. The hourly cron will run again within the hour to catch any channels created after the manual prune, on both sites.

Prevention: The hourly cleanup sweep automatically prevents this, and fails its own run (with a ::error:: annotation naming the site) if either site's post-sweep channel count is still ≥40 (of that site's 50-channel quota) — check that run's logs first if this recurs despite the sweep. If the sweep itself is failing (not just failing its quota check), escalate.

Rollback

Tier 1: Emergency (seconds, API only)

Instantly routes Cloud Run traffic to the previous revision. No CI, no rebuild, no frontend change.

rollback --emergency

Or directly via gcloud:

gcloud run revisions list --service=enkidu-api-production --region=us-central1 --limit=5
gcloud run services update-traffic enkidu-api-production \
  --region=us-central1 --to-revisions=PREVIOUS_REVISION=100

Tier 2: Planned (minutes, full stack)

Re-deploys a previous version's Docker image and rebuilds the frontend from that tag's commit. Uses the existing image — no Docker rebuild.

rollback v3.4.55          # re-deploy version v3.4.55
rollback 3.4.55           # same thing (v prefix added automatically)

Or via GitHub Actions UI: manually trigger the rollback.yml workflow with the version input.

Tier 3: Staging Rollback

For staging, revert the commit on master (triggers a new staging deploy):

rollback staging          # revert last commit on master
rollback staging abc1234  # revert specific commit

Database Migrations During Rollback

Rollbacks run db:migrate forward (idempotent). They do not undo migrations. This is safe when:

• Migrations only added columns/tables (old code ignores them)
• Use skip_migrations=true (the default) for most rollbacks

If a migration removed a column the old code needs, you must write a new forward migration and deploy it as a new release.

CLI Tools Reference

Command	Description
release [version]	Create a version tag and trigger production deploy
rollback <version>	Re-deploy a previous version to production
rollback --emergency	Instant Cloud Run revision switch (API only)
rollback staging [sha]	Revert commit on master (triggers staging deploy)
deployed	Show what's currently running on staging and production

These are fish shell functions in ~/.config/fish/functions/.

Read-only staging console checks

There is no general Rails console access on staging for service accounts / agents — a general "rails console" job was deliberately ruled out (OBJ-1225 Task 2) because it widens the blast radius of an executable-with-DB-access surface far beyond what any single check needs. Instead, narrow, single-purpose, read-only Cloud Run Jobs and direct read-only API calls cover the two things that recurred as access gaps: evaluating a backend feature flag for a real user, and inspecting a flag's PostHog configuration (rollout %, cohort targeting).

Checking a feature flag against staging (enkidu-flag-check-staging)

FeatureFlagService.enabled? (rails_api/app/services/feature_flag_service.rb) is PostHog-backed, not DB-backed — it calls POSTHOG_CLIENT.is_feature_enabled(flag_name, user.public_id). A direct Cloud SQL query can't verify it; only exercising the actual code path can. enkidu-flag-check-staging is a parameterized, on-demand Cloud Run Job wired in .github/workflows/staging.yml ("Provision Flag-Check Console Job" step) around the verify:feature_flag rake task (rails_api/lib/tasks/verify.rake) — read-only, no writes. It is created/updated on every staging deploy but never auto-executed; run it on demand:

# By public_id
gcloud run jobs execute enkidu-flag-check-staging --region us-central1 \
  --update-env-vars FLAG_NAME=<flag_name>,USER_PUBLIC_ID=<public_id> --wait

# By email (added OBJ-1225 P2 — useful when you know the account's email but not its public_id)
gcloud run jobs execute enkidu-flag-check-staging --region us-central1 \
  --update-env-vars FLAG_NAME=<flag_name>,USER_EMAIL=<email> --wait

The job's own output isn't in the gcloud run jobs execute command output — read it from Cloud Logging:

gcloud logging read \
  'resource.type="cloud_run_job" AND resource.labels.job_name="enkidu-flag-check-staging"' \
  --limit 5 --format=json

Expected jsonPayload: {"flag_name": "...", "user_public_id": "...", "posthog_client_nil": false, "enabled": true|false}. posthog_client_nil: true means POSTHOG_CLIENT never initialized (missing POSTHOG_API_KEY on the job) — that reads as a fail-closed false, not a real evaluation, so always check it alongside enabled. An unrecognized USER_PUBLIC_ID/USER_EMAIL fails fast with no user found for ... rather than silently evaluating for no one.

See Feature Flags § Reference for the frontend registry and PostHog rollout lifecycle this job is verifying against.

Querying PostHog flag/cohort definitions directly (posthog-personal-api-key)

Whether a flag exists and whether it's actually scoped to a cohort are different questions — a flag that exists but has rollout_percentage: 0 and no properties filter looks identical to a missing flag from the app's /decide/ call, both evaluating false for everyone. The posthog-personal-api-key GCP Secret Manager secret holds a PostHog personal API key (distinct from the project key used by the app) scoped for read access to PostHog's management REST API — it is provisioned once by a human (Josh) via the PostHog UI (Settings → Personal API keys) and is not injected into any Cloud Run service; query it directly:

POSTHOG_KEY=$(gcloud secrets versions access latest --secret=posthog-personal-api-key)

curl -s "https://us.posthog.com/api/projects/368400/feature_flags/" \
  -H "Authorization: Bearer $POSTHOG_KEY" | jq '.results[] | {key, id, active, rollout_percentage: .filters.groups[0].rollout_percentage, properties: .filters.groups[0].properties}'

This is the same PostHog project (368400) and personal-key concept the feature-flag sync tool uses locally for --apply/--promote — the difference is this secret lets a service account (dave-autopilot) run the same kind of read against staging/production verification work without a human pasting a key into an ad hoc shell each time.

Gotcha: roles/run.invoker doesn't cover --update-env-vars on a parameterized job

gcloud run jobs execute --update-env-vars (or --args) requires the run.jobs.runWithOverrides permission — this is not included in roles/run.invoker (whose includedPermissions are only run.jobs.run and run.routes.invoke). Granting run.invoker to a service account and expecting it to run a job with per-invocation overrides fails with PERMISSION_DENIED: Permission 'run.jobs.runWithOverrides' denied, even though the same account can run the job with zero overrides just fine — a confusing partial-success signal if you don't already know the permission split.

run.jobs.runWithOverrides is otherwise only bundled into the much broader roles/run.developer (which also grants create/update/delete on jobs — more than a verification-only service account should have). The fix used here: a least-privilege custom IAM role scoped to exactly run.jobs.get, run.jobs.run, run.jobs.runWithOverrides, bound to the service account on the specific job resource, not project-wide:

gcloud iam roles create flagCheckStagingRunner \
  --project=enkidu-488723 \
  --permissions=run.jobs.get,run.jobs.run,run.jobs.runWithOverrides

gcloud run jobs add-iam-policy-binding enkidu-flag-check-staging \
  --region=us-central1 \
  --member="serviceAccount:dave-autopilot@enkidu-488723.iam.gserviceaccount.com" \
  --role="projects/enkidu-488723/roles/flagCheckStagingRunner"

Apply the same pattern for any future parameterized Cloud Run Job a service account needs to invoke with runtime overrides — don't reach for run.invoker (silently insufficient) or run.developer (unnecessarily broad).

No production equivalent exists — the read-access gap (OBJ-2607)

Everything above is staging-only. There is no production counterpart to enkidu-flag-check-staging or the PostHog management-API read, and no sanctioned path for a one-off read-only production query (e.g. counting rows matching a WHERE clause).

This isn't an oversight — it's the same OBJ-1225 Task 2 judgment applied consistently: general Rails-console/DB access was deliberately ruled out for service accounts to cap blast radius, and that reasoning applies with at least as much force to production as to staging. gcloud run jobs list confirms no production job mirrors enkidu-flag-check-staging. gcloud sql connect (used in the Database Point-in-Time Recovery procedure) is a direct raw-SQL connection to the production DB — exactly the surface OBJ-1225 ruled out — and must not be used as a workaround for an ad hoc read-only query.

Sanctioned route today: none for a service account or agent. A one-off read-only production question (e.g. counting affected rows for an incident) needs a human with production access to run it directly. Codi ran into this on RESTORE-5 (OBJ-2607) — the exposure-window user count for the username.required restoration — and had a ready-to-run read-only query with no sanctioned way to execute it:

SELECT COUNT(*) FROM users
WHERE username IS NULL
  AND deleted_at IS NULL
  AND created_at >= '2026-08-15T15:19:40Z'
  AND created_at <  '2026-08-19T00:35:27Z';

Hand a query like this to a human with production access rather than reaching for gcloud sql connect or improvising tooling to answer it.

Outcome: this exact query was run by a human with production access (a throwaway, read-only Cloud Run job mirroring enkidu-db-migrate's runtime config, deleted after use) and returned 1. The access gap itself is unchanged — this was a one-off human run, not a new sanctioned route — see docs/architecture/authentication.md's Clerk Dashboard Configuration incident record for what the count means.

If this recurs often enough to justify a job: that's separately scoped work — a narrow, single-purpose, read-only Cloud Run Job mirroring enkidu-flag-check-staging, with its own IAM review (see the roles/run.invoker gotcha above for the permission split it would need). Not built here — this section documents the gap, not a fix for it.

Gotchas: GitHub Actions Concurrency

Batch tag pushes silently drop runs

Behavior (pre-fix — see "Fixed (OBJ-2481)" below): When production.yml declared concurrency: { group: production-deploy, cancel-in-progress: false } with no queue: key, GitHub Actions enforced a hard limit: at most 1 running + 1 queued workflow per concurrency group. If more workflows arrived while both slots were full, they were silently dropped without error or notification — they never queued, and they never executed.

Impact (OBJ-348, May 14, 2026): Pushing 12 version tags in rapid succession (via script or GitHub API batch) triggered 12 production.yml runs, but only the first 2 were executed. Runs 3–12 were discarded. No error message. No alert. The deployments simply never happened, and the mismatch between git tags and deployed versions only surfaced during manual verification.

How it happens:

1. Tag v3.8.1 lands → production.yml starts (running: 1, queued: 0)
2. Tag v3.8.2 lands → queued (running: 1, queued: 1)
3. Tag v3.8.3 through v3.8.12 land → silently dropped (slots full)
4. When run 1 finishes, run 2 dequeues and executes; no subsequent runs exist

Fixed (OBJ-2481): production.yml's production-deploy concurrency group now sets queue: max alongside cancel-in-progress: false (.github/workflows/production.yml:8-21), switching it from GitHub's default queue: single (1 running + 1 pending, rest dropped) to a real FIFO queue (up to 100 pending runs) — batch tag pushes now queue and execute in order instead of being silently discarded. Enforced going forward by ci.yml's lint-workflows guard ("Guard smoke-lane concurrency groups against pending-run cancellation"), which now includes production.yml:production-deploy in its checked pairs. See gotchas.md's matching entry and the cancel-in-progress: false alone isn't a FIFO queue gotcha for the underlying mechanism.

Mitigation (no longer required, but still fine):

Sequential tag pushes were the only safeguard before this fix and still work — there's just no longer a need to wait for each production.yml run to finish before pushing the next:

# Before the fix this dropped 2 of 3 runs; now all 3 queue and execute in order:
for v in 3.8.1 3.8.2 3.8.3; do git tag "v$v" && git push origin "v$v"; done

# Sequential pushes still work too, just no longer necessary:
git tag v3.8.1 && git push origin v3.8.1
git tag v3.8.2 && git push origin v3.8.2
git tag v3.8.3 && git push origin v3.8.3

When deploying the latest version only (the common recovery case after an incident), push only the latest tag — older tags' code is already subsumed:

# Recover by deploying the latest prod-ready version
# Do NOT re-push intermediate tags
git tag v3.8.5 && git push origin v3.8.5  # all fixes are in this commit

Staging deploy race: unscoped concurrency lets an older commit overwrite a newer one

Behavior (as of OBJ-1144, July 5, 2026): staging.yml's deploy-backend, deploy-litellm, and deploy-frontend jobs each used a bare concurrency group (e.g. staging-deploy-frontend) with cancel-in-progress: true, scoped only by job name — not by commit. cancel-in-progress: true only cancels a run that is genuinely still in-flight when a newer run starts. It does nothing if an older commit's job has already survived past the cancellation point and simply finishes its publish step after a newer commit's job has already finished and deployed. GitHub Actions gives no ordering guarantee on which of two concurrent runs' jobs completes first — whichever one physically finishes last wins the deploy, regardless of which commit is actually newer. deploy-litellm still runs this way; deploy-backend/deploy-frontend no longer do — see the OBJ-1992 follow-up below.

Impact (OBJ-1144, July 5, 2026): Two commits landed on master ~2 minutes apart (PR #1265, then PR #1266). Both Staging Deployment runs executed concurrently. PR #1266's deploy-frontend job finished first and deployed; PR #1265's deploy-frontend job — despite building the older commit — finished afterward and silently overwrote it on Firebase Hosting. No job failed and no error was surfaced anywhere; the only signal was tests/smoke-playwright/specs/platform/00-version-tag.spec.ts failing on a git-describe version-string mismatch, reproduced identically across two independent CI runs 3+ hours apart (ruling out a caching fluke).

Fix (guard step, all three deploy jobs): Each job now re-checks origin/master's actual tip immediately before its publish action (gcloud run services replace / gcloud run deploy / firebase-tools deploy) and skips the publish if a newer commit has since landed:

- name: Guard against stale commit
  id: guard_stale
  run: |
    LATEST_MASTER_SHA="$(git ls-remote origin refs/heads/master | cut -f1)"
    if [ "$LATEST_MASTER_SHA" != "${{ github.sha }}" ]; then
      echo "::notice::${{ github.sha }} is no longer the tip of master (origin/master is now $LATEST_MASTER_SHA) — a newer commit's Staging Deployment run will supersede this one. Skipping the deploy to avoid overwriting it with a stale build."
      echo "is_stale=true" >> "$GITHUB_OUTPUT"
    else
      echo "is_stale=false" >> "$GITHUB_OUTPUT"
    fi

Each job exposes is_stale as a job output, every subsequent publish/side-effect step is gated on steps.guard_stale.outputs.is_stale != 'true', and playwright-smoke skips entirely when any deploy job reports is_stale=true — otherwise a superseded run would spuriously fail the version-tag smoke assertion for a build it never actually published. See .github/workflows/staging.yml (deploy-backend, deploy-litellm, deploy-frontend jobs).

Known residual gap: This is a point-in-time check, not atomic with the publish call that follows it — a much narrower race (two runs whose guard checks both land before either's origin/master update is visible to the other) is theoretically still possible, though far tighter than the original job-duration-wide window. The guard step also adds a new network call (git ls-remote origin) that can itself fail transiently, which fails the job rather than silently misdeploying — a safer failure mode, but a new flake surface to be aware of.

Applies to any future workflow reusing this pattern: a bare, job-name-scoped concurrency group with cancel-in-progress: true protects against overlapping runs but not against completion-order races. If you add a new deploy job (or any job where "only the newest commit's result should count") behind this pattern, add the same staleness guard before its side-effecting step.

Follow-up (OBJ-1992, August 1, 2026) — cancel-in-progress: true itself was killing multi-step deploys mid-publish: guard_stale stops a stale build from being the one left live, but it doesn't protect an in-flight, not-yet-stale build from being interrupted partway through its own publish. deploy-backend runs 3 sequential gcloud run services replace calls (rails-api, worker, crono) and deploy-frontend runs one firebase-tools deploy across 2 hosting targets (enkidu-app-staging, enkidu-admin-staging) — cancel-in-progress: true sends SIGTERM the instant a newer push's same-named job starts, with no guarantee of where in that sequence it lands. A kill between calls/targets can leave those resources split across two commits' builds simultaneously. This was confirmed happening on deploy-frontend (run 30710897912, killed mid npm run build when a newer push's deploy-frontend started ~2 minutes later) during a 3-pushes-in-3-minutes burst that produced OBJ-1992's chromium smoke failures.

Fix: deploy-backend and deploy-frontend's concurrency groups flipped cancel-in-progress: true → false. An in-flight deploy on either job now always finishes as one internally-consistent publish; a superseded run is instead evicted from the queue before it starts (GitHub's default queue: single, which only keeps the newest pending run) or lands after this guard and skips its own deploy. deploy-litellm, staging-migrate, and staging-seed-data were deliberately left on cancel-in-progress: true — none of them publish more than one resource per job run (deploy-litellm is a single gcloud run deploy; the other two invoke gcloud run jobs execute --wait, where cancel-in-progress: true only cancels the local runner step, not an already-kicked-off Cloud Run Job execution on GCP's side), so there's no matching mid-flight-interruption risk to fix. A lint-workflows guardrail (.github/workflows/ci.yml, "Guard multi-step deploy jobs against mid-flight cancellation") now fails CI if deploy-backend or deploy-frontend regresses back to cancel-in-progress: true without first splitting its multi-resource publish into single-resource jobs. See the guard_stale step comments on both jobs in .github/workflows/staging.yml for the full reasoning.

Distinct from: Batch tag pushes silently drop runs above — that's production.yml's single global production-deploy group dropping excess runs outright (1 running + 1 queued, rest discarded); this is staging.yml's per-job groups racing on completion order (OBJ-1144, fixed by guard_stale) and, for multi-resource jobs, on mid-flight interruption (OBJ-1992, fixed by cancel-in-progress: false) among runs that all do execute.

Assumed secrets silently become empty strings

Behavior: GitHub Actions does not validate that secrets exist when a workflow references them. When a workflow step references ${{ secrets.SECRET_NAME }} and the secret is not provisioned in Settings → Secrets and variables → Actions, GitHub Actions silently substitutes an empty string. No error. No warning. The workflow continues.

Impact (OBJ-519, May 14, 2026): The smoke test workflow referenced SMOKE_CLERK_EMAIL in playwright-smoke.yml, but the secret was never provisioned. For 14 hours, all CI runs substituted an empty string. The error only surfaced when the first real PR ran the smoke tests — the auth setup threw an error, and CI blocked the entire PR.

Root cause: The acceptance criteria for the phase that wired the secret did not include "confirm secret exists AND a live CI run passes." The planner had marked it [ASSUMED] in planning notes, but the assumption was never verified before merge.

Mitigation & Prevention: See CI Secrets Checklist for the full guide on GitHub Actions secret management, current required secrets, and the rule: Any new secret-dependent workflow step needs an acceptance criterion: "Confirm secret X is present in Settings and a live CI run (PR or tag push) passes."

Off-master tags never build Docker images

Behavior: staging.yml only builds Docker images on push: branches: [master] (.github/workflows/staging.yml:3-6). If you tag a commit that is not reachable from master (e.g., a squash-merge feature-branch tip, or a commit directly on an old release branch), staging.yml never runs, and no rails-api:{sha} image lands in Artifact Registry.

Impact: When production.yml runs and reaches the Validate Release Image step, it polls for the missing image for 300 seconds (5 minutes). After timeout, the deploy fails. The image was never built because staging.yml didn't run.

Detection:

# Check if a commit is reachable from master
git merge-base --is-ancestor <commit-sha> origin/master
echo $?  # 0 = reachable (safe to tag), 1 = not reachable (risky)

Mitigation:

Always tag from master (i.e., ensure the commit is reachable from master). The recommended approach is to use scripts/release.sh, which enforces this as a guard (scripts/release.sh:40-53):

# scripts/release.sh enforces: must be on master, in sync with origin/master
scripts/release.sh 3.9.0

# Alternatively, ensure the commit is on master:
git checkout master && git pull origin master
git tag v3.9.0  # Only tag commits that are on master
git push origin v3.9.0

Recovery: If you've already tagged an off-master commit and production.yml is timing out waiting for the image, trigger staging.yml to build the missing Docker image:

1. Open the Staging Deployment workflow in the GitHub Actions UI.
2. Click Run workflow and provide the commit SHA via the commit_sha input.
3. Wait for the dispatched build-rails-api job to complete and push rails-api:{commit_sha} to Artifact Registry.
4. Once the image lands, re-tag from a commit reachable from master: git tag -d <version> && git push origin :refs/tags/<version> && git tag <version> <new-commit> && git push origin <version>.

This procedure is documented in the recovery guidance embedded in production.yml's timeout error message (lines 109–112 of the Resolve and validate image step).

Structural safeguard (OBJ-349): A CI-level check was added to production.yml (the Verify tag is on master step at production.yml:37) to fail fast if the tagged commit is not reachable from master, preventing timeout delays and providing immediate feedback.

Cloud Run secrets must be duplicated across all services and revisions

Behavior: Environment variables passed via --set-secrets on a Cloud Run service deploy are not inherited across service revisions and not shared between separate Cloud Run services (api, sidekiq, cron). Each gcloud run deploy invocation specifies --set-secrets independently. If you add a new secret to one service's deploy step in the CI workflow but forget the others, those services will launch without the secret — and the application will treat ENV['SECRET_NAME'] as nil.

Impact (OBJ-452, May 18, 2026): Two secrets were missing from all 10 --set-secrets blocks (5 production + 5 staging) in .github/workflows/production.yml and .github/workflows/staging.yml:

1. MAILTRAP_API_TOKEN — used as the SMTP password in rails_api/config/environments/production.rb (line ~106: password: ENV['MAILTRAP_API_TOKEN']). Missing this secret = SMTP auth fails with nil password, and email delivery fails silently.
2. FRONTEND_URL — used in rails_api/app/mailers/user_mailer.rb (line 38: ENV.fetch('FRONTEND_URL', 'http://localhost:5173')). Missing this secret = magic code emails link to localhost in production, making them unclickable.

Both secrets were added to the Cloud Run service definition during initial setup, but the CI workflow steps were never updated. On the next production deploy, the missing secrets caused email delivery to fail, breaking the entire sign-in flow. This cascaded into "Sign in is taking longer than expected" errors on the frontend.

Mitigation:

Whenever you add a new environment variable to Rails and want it available in production Cloud Run, you must:

1. Add the secret to GCP Secret Manager (one-time):

   echo "your-secret-value" | gcloud secrets create your-secret-name --data-file=-

2. Add the secret to every --set-secrets block in the workflow file:

   • Every gcloud run deploy step for enkidu-api-production (db-migrate-update, db-migrate-create, main service deploy)
   • Every gcloud run deploy step for enkidu-sidekiq-production (db-migrate-update, db-migrate-create)
   • Every gcloud run deploy step for enkidu-cron-production (db-migrate-update, db-migrate-create)
   • Same for staging: all 5 blocks in staging.yml

3. Example additions to .github/workflows/production.yml:

   # Line ~145 (api db-migrate-update):
   --set-secrets DATABASE_PASSWORD=production-db-password:latest,MAILTRAP_API_TOKEN=mailtrap-api-token:latest,FRONTEND_URL=frontend-url:latest
   
   # Line ~153 (api db-migrate-create):
   --set-secrets DATABASE_PASSWORD=production-db-password:latest,MAILTRAP_API_TOKEN=mailtrap-api-token:latest,FRONTEND_URL=frontend-url:latest
   
   # etc.

Prevention:

A pre-deploy validation script (scripts/validate-secrets.sh) can check that all secrets referenced in rails_api/config/environments/production.rb and rails_api/app/mailers/ exist in both GCP Secret Manager and the CI workflow --set-secrets blocks. Proposed structure:

# Check that all ENV['VAR'] / ENV.fetch('VAR') calls in Rails code
# are present in GCP Secret Manager and in all CI workflow --set-secrets lines
grep -rh "ENV\[.*\]" rails_api/ | ... (extract var names)
gcloud secrets list | ... (check they exist)
grep "--set-secrets" .github/workflows/production.yml | ... (check they're in all blocks)

This check is not yet implemented. For now, remember: if you add a new secret-backed env var to Rails, you must update all 10 --set-secrets blocks (5 prod + 5 staging).

Cloud Run silently retains env vars across a service topology change

Behavior: gcloud run deploy --set-env-vars (and --update-env-vars) does not always clear a previously-set environment variable when the service's container topology changes — e.g. when a service goes from single-container to multi-container (sidecar) layout. A variable set on the prior revision can silently carry forward into the new revision even when the new deploy command omits it, because the per-container env merge does not treat "absent in this invocation" as "remove."

Impact (OBJ-565, staging outage): The rails-api service was migrated from single-container to multi-container. The previous revision had AGENT_RUNNER_AUTH_MODE=oidc; the new multi-container deploy did not pass it, on the assumption it would default back to secret. Instead Cloud Run retained oidc from the old revision, so the rails-api → agent-runner health probe tried OIDC auth against a service expecting a shared secret and failed /health for the entire window. There was no error at deploy time — the stale value just persisted.

Mitigation:

• When a variable's intended value changes between revisions, set it explicitly every time: --set-env-vars AGENT_RUNNER_AUTH_MODE=secret,... — never rely on omission to reset it to a default.
• When a variable should be absent in the new revision, remove it explicitly with --remove-env-vars=AGENT_RUNNER_AUTH_MODE.
• Prefer pinning topology-sensitive values in the declarative manifest (deploy/service.*.yaml), which is fully authoritative — the manifest does not inherit from the prior revision the way incremental --set-env-vars flags can.

This is the same silent-persistence failure family as the secret-duplication gotcha above: the deploy succeeds, but the running config differs from the workflow's apparent intent.

Cloud Run multi-container memory limits are an instance-wide sum — a backgrounded process pays full Rails-load cost, not a fraction of it

Behavior: each container in a multi-container Cloud Run manifest declares its own resources.limits.memory, but Cloud Run enforces the ceiling as the sum across every container in the instance — not per-container isolation. If any one process pushes the combined total over that sum, the whole instance OOMs, including containers that were nowhere near their own limit.

The enkidu-worker-* services compound this with a process-model foot-gun: rails_api/bin/start-worker.sh runs three independent processes inside the single rails-worker container —

bundle exec sidekiq -t 25 &     # backgrounded — still a full, separate Rails boot
bundle exec crono &             # backgrounded — still a full, separate Rails boot
exec bundle exec puma -C config/puma.rb   # foreground

Backgrounding a process with & in a shell script does not make it cheap. Each bundle exec here is an independent process that loads all of Rails, every gem (GraphQL, Sentry, PostHog, Firebase Admin, Stripe, ActiveStorage, APNs, LiteLLM client, etc.), and opens its own DB connection pool — there is no copy-on-write memory sharing between them, because they were never forked from a common parent. That's the opposite of a Puma cluster's forked workers, which do share pre-fork memory pages. Three independent full-Rails boots inside one container's memory limit is a 3x tax, not a rounding error.

Impact (OBJ-1481, diagnosed 2026-07-16): enkidu-worker-staging OOM-restarted 99 times over 72 hours (time between restarts ranged 12 seconds to 7.4 hours). Dave's log-correlated diagnosis found this was steady-state over-allocation, not a leak — three lines of evidence: OOM magnitude was flat regardless of instance age (1024–1197 MiB, median 1032, no growth curve), 12 of 99 instances OOM'd within 60 seconds of a fresh boot (fastest: 12.3s — no window for a leak to have grown), and long quiet gaps (up to 7.4h) existed too, consistent with a near-full baseline that only tips over under load rather than a leak that eventually blows up regardless of load.

Container attribution: rails-worker's three-process boot kept it within ~0–50MB of its 512Mi ceiling from cold start alone — the deterministic, load-independent driver. The agent-runner sidecar (its own 512Mi) added a variable ~100–170MB tax on top whenever 2+ AI Workforce runs executed concurrently — the three worst historical spikes (1197/1196/1176 MiB) all coincided, within 1–2 seconds, with two concurrent AiWorkforce::ExecuteRunJob dispatches landing at the same instant.

Fix (OBJ-1483): resized both deploy/worker.staging.yaml and deploy/worker.production.yaml (production carried the identical split): rails-worker 512Mi → 1024Mi, agent-runner 512Mi → 768Mi. New combined ceiling 1792Mi, up from 1024Mi — roughly 30–45% headroom over the worst observed spike (1197 MiB). This is a stopgap, not a structural fix: the underlying architecture — three independent full-process boots sharing one container — is unchanged, and Crono's minScale=maxScale=1 singleton constraint (documented at the top of deploy/worker.staging.yaml) still means this container can never scale horizontally to spread that load. Moving Crono (and possibly Sidekiq) to its own Cloud Run service is tracked separately as a lower-urgency structural follow-up.

Rule of thumb: when sizing a Cloud Run multi-container manifest, size for number of independent process boots × per-boot memory cost, not one app's footprint. A command: override that backgrounds N bundle exec (or any full-runtime) invocations in the same container pays N× the baseline load cost — & does not fork, share memory, or make a process a lightweight worker the way a preloaded app server's forked children do.

Staging silently read production's live-mode Stripe secrets by name collision

Behavior: A Cloud Run secretKeyRef.name is just a string — nothing stops two services in two different environments from pointing at the identical Secret Manager secret. deploy/service.staging.yaml and deploy/worker.staging.yaml referenced STRIPE_SECRET_KEY / STRIPE_WEBHOOK_SECRET via the same secret names as deploy/service.production.yaml / deploy/worker.production.yaml (stripe-secret-key, stripe-webhook-secret). Both environments were reading the same Secret Manager secret and version — there was no Stripe environment separation at all, unlike Clerk, which already used the correct pattern (clerk-secret-key-dev for staging vs clerk-secret-key for production, see Secrets below).

Impact (OBJ-1220 P6 → OBJ-1225, discovered 2026-07-09): Dave's T4b pre-flight audit found staging's deployed STRIPE_SECRET_KEY resolved to a live-mode key (sk_live_…), identical to production's. It fails closed today only by accident: the Teams price ID secrets staging also reads (stripe-price-teams-monthly/-annual) are genuine test-mode objects, so GET /v1/prices/{id} against them with the live-mode key returns resource_missing — Stripe's own diagnostic even names the mismatch ("a similar object exists in test mode, but a live mode key was used"). That accidental protection disappears the moment anyone "fixes" checkout in the wrong direction — repointing the Teams price IDs to their live-mode equivalents without first giving staging its own test-mode key would make staging create real live charges, and staging's URL had no live-mode webhook registered in Stripe, so a real charge would have no provisioning callback (a charged customer with nothing provisioned).

Root cause: the same secret name was hardcoded into both environments' secretKeyRef at manifest-authoring time, with no structural check that a staging service and a production service never share a secret. Nothing about the YAML, the deploy workflow, or code review made the collision visible — both services deployed cleanly and both resolved a non-nil STRIPE_SECRET_KEY, so every "does staging have this env var" check passed.

Fix:

1. Provisioned two new staging-only secrets holding genuine test-mode credentials: stripe-secret-key-staging (sk_test_…) and stripe-webhook-secret-staging (whsec_…, paired with a test-mode webhook endpoint registered against staging's /webhooks/stripe URL).
2. Repointed deploy/service.staging.yaml and deploy/worker.staging.yaml's secretKeyRef.name to the -staging secrets. deploy/*.production.yaml untouched.
3. Added a CI regression guard, scripts/check-deploy-secret-drift.mjs (run by .github/workflows/deploy-secret-drift.yml on any PR touching deploy/**): it parses all four deploy/*.yaml manifests and fails if any STRIPE_* env var resolves to the same secretKeyRef.name on a staging file as on a production file.

The correct pattern going forward — mirror Clerk's naming split: any secret whose value must differ between environments (API keys, webhook signing secrets — anything that isn't itself environment-agnostic, like rails-master-key) needs its own environment-suffixed secret name (<name>-staging / bare <name> for production, matching Clerk's existing clerk-secret-key-dev / clerk-secret-key convention) — never a shared name gated only by "staging happens to use test-mode values right now." A shared name means the next rotation, migration, or "just copy prod's config to unblock staging" fix silently re-collapses the environments. If you add a new secret-backed integration with per-environment credentials, extend check-deploy-secret-drift.mjs's prefix list (currently STRIPE_* only) rather than relying on manual review to catch a future collision.

Staging wrote live uploads into production's GCS bucket by literal-value collision

Behavior: the Stripe gotcha above is about a shared secretKeyRef.name; this is the same failure class but through a plain literal value: instead of a secretKeyRef. deploy/service.staging.yaml and deploy/worker.staging.yaml pinned GCS_BUCKET_NAME: enkidu-storage — byte-identical to deploy/service.production.yaml / deploy/worker.production.yaml. GCS bucket names are a global namespace, so this wasn't two same-named buckets in two projects — it was one bucket. At the time this was discovered, neither manifest declared a serviceAccountName, so both services also ran as the project's default compute SA, meaning staging had exactly production's GCS permissions with no IAM boundary to fall back on. Staging has since gained its own least-privilege service accounts — deploy/service.staging.yaml and deploy/worker.staging.yaml now set serviceAccountName: enkidu-run-api-staging@... / enkidu-run-worker-staging@... respectively (OBJ-1355 Phase 2). deploy/service.production.yaml and deploy/worker.production.yaml now set the equivalent enkidu-run-api-production@... / enkidu-run-worker-production@... (OBJ-1356 Phase 3) — but a manifest change only takes effect once a v* tag is cut and deploys; merging to master reaches staging only.

Impact (OBJ-1341, discovered 2026-07-11): confirmed live, not theoretical — staging was writing real user uploads (user_photo, goal_media, community_media, community.cover_image_attachment, design_asset) into production's bucket. ActiveStorage keys are random 32-character identifiers, so staging did not overwrite or corrupt any production blob, and there was no live data-loss event. The damage was accumulation: staging and production objects were interleaved under random keys with no prefix and no distinguishing metadata, so there was no way to tell them apart from the bucket alone.

Root cause: both staging manifests carried a comment asserting the sharing was intentional ("staging intentionally shares the same bucket as production... pinned explicitly to prevent drift from the provisioned bucket"). That rationale predated the pin — it referred to storage.yml's config default, which the explicit manifest value: had already overridden. The comment described a constraint the pinning itself had removed, and its presence is exactly what would have made a future agent re-share the bucket on its authority.

Fix:

1. Provisioned a new staging-only bucket, enkidu-storage-staging, with the same allUsers:objectViewer public-read grant as production's enkidu-storage (required — rails_api/config/storage.yml sets public: true, which generates unsigned URLs).
2. Repointed deploy/service.staging.yaml and deploy/worker.staging.yaml's GCS_BUCKET_NAME to enkidu-storage-staging, and rewrote the stale "intentional sharing" comment to state staging is deliberately isolated, citing this issue and the Stripe precedent (OBJ-1225). deploy/*.production.yaml untouched. gcp_setup/02_create_resources.sh parameterized to provision and public-bind both buckets.
3. Existing staging attachments were not migrated — staging media is disposable test data, so pre-fix uploads 404 after the repoint. This was an accepted tradeoff, not an oversight.

Regression guard: the Stripe drift guard (scripts/check-deploy-secret-drift.mjs) only matched STRIPE_* env vars wired through secretKeyRef — it could not see this bug, because GCS_BUCKET_NAME is an arbitrary name carrying a literal value:, not a secretKeyRef. Extended the guard with a second check, findLiteralDrift, that compares every literal-valued env var across each staging↔production manifest pair and fails on any byte-identical match — gated by an explicit LITERAL_VALUE_ALLOWLIST for env vars that legitimately match across environments (shared Cloud SQL instance, shared GCP project, auth-mode-by-design, etc.). GCS_BUCKET_NAME is permanently excluded from that allowlist, with an inline comment explaining why. Verified the guard fails on the pre-fix manifests and passes on the post-fix ones — a guard that only passes on the known-bad state proves nothing.

Not part of this fix: a related but separate finding surfaced while investigating this bug — gcp_setup/02_create_resources.sh grants allUsers:objectViewer at the bucket level (required for public media), and docs/operations/disaster-recovery.md's manual DB export procedure wrote production database dumps into that same public bucket's backups/ prefix. Tracked separately as OBJ-1342 and now resolved: the allUsers:objectViewer grant was confirmed live, enkidu-storage/backups/ was confirmed empty (no dump had ever been written, so no disclosure occurred), and the DR runbook was repointed at a private enkidu-backups bucket provisioned by gcp_setup/07_create_backups_bucket.sh — see Disaster Recovery §4.3 for the current procedure.

Public media Cache-Control: how long a deleted photo stays fetchable, and why

Behavior: rails_api/config/storage.yml's google: ActiveStorage service sets public: true (see above) but, before OBJ-1442, never set cache_control: — so GCS applied its own default, public, max-age=3600, to every uploaded object. Because staging.rb requires production.rb, both environments shared this default. Net effect: after a photo was deleted or replaced, the old object could keep being served from GCS's edge cache for up to an hour — a real deletion-completeness gap in production, not just a test artifact. All five attachments backed by the google: service are affected: user_photo.image_file, goal_media.image_file, community_media.image_file, community.cover_image_attachment, design_asset.image.

Fix (OBJ-1442, PR #1534): storage.yml's google: block now sets cache_control: "public, max-age=300" explicitly — one line, covering both environments via the same staging.rb → production.rb inheritance. This drops the post-delete exposure window from 60 minutes to 5 minutes. It only takes effect for newly uploaded objects: anything already in the bucket keeps whatever Cache-Control GCS wrote at upload time (max-age=3600, or nothing before that) until it's superseded or someone runs the optional metadata-only backfill below.

Why 300 seconds, not no-cache: ActiveStorage blob keys are unguessable — a 28-character random base36 string, ~144 bits of entropy — so exploiting the post-delete cache window requires already holding the URL, at which point the image is already accessible regardless of caching. The content at a given key never changes (each blob gets its own key on write), so caching remains semantically correct and worth keeping for avatars and goal media that load on nearly every screen. max-age=300 keeps most of the caching benefit while shrinking the exposure window to something reasonable; no-cache would force an origin revalidation on every image load to close a window the entropy argument says isn't worth paying for. If a future privacy/legal requirement demands a hard zero-second window, swapping to no-cache is the same one-line change in storage.yml.

Why this is safe (no upload-path regression): no direct-to-GCS uploads exist anywhere in the app — no @rails/activestorage in the frontend, no direct_upload route — every upload goes through Rails' server-side ActiveStorage::Service::GCSService#upload path (e.g. via UserIdentity::UpdateUserPhoto). The GCS service gem also has a cache_control:-triggered GCS v2 → v4 URL-signing switch, but that switch lives inside url_for_direct_upload, a method this app never calls — so setting cache_control: only changes the header GCS attaches to newly stored objects, not any signing behavior this app actually exercises.

Optional backfill for pre-existing objects (ops action, not code): objects uploaded before this fix keep their old Cache-Control metadata forever unless explicitly rewritten. A metadata-only update (bumps metageneration, does not rewrite object data) can bring them in line:

gcloud storage objects update 'gs://enkidu-storage-staging/**' --cache-control="public, max-age=300"

Run staging first and verify before touching the production bucket (enkidu-storage), and only after the code fix above is deployed — backfilling first would leave a gap where new uploads still got the old 3600s default. This is genuinely optional: new uploads are covered from day one by the code fix, and the pre-fix population shrinks naturally as media is replaced over time.

A command: pin only controls the container entrypoint — it can't stop an initializer from starting a background process

Behavior: deploy/service.staging.yaml and deploy/service.production.yaml pin the API's container command: to ["bundle", "exec", "puma", "-C", "config/puma.rb"], with a comment asserting this makes the web/worker split "explicit and immune to image CMD drift" — implying the pin was sufficient to guarantee the API never runs Sidekiq. It wasn't. rails_api/config/initializers/sidekiq_embedded.rb ran in every Rails process where Sidekiq was defined and Rails.env was production/staging, including the Puma-only API service — it spun up a real Sidekiq::Launcher on a background thread from after_initialize, entirely independent of what command started the container. A command: pin constrains the process's entrypoint; it has no say over what a Rails initializer does once the app finishes booting.

Impact: the worker's own bundle exec sidekiq process (started via its own manifest, deploy/worker.*.yaml) already subscribed every queue in rails_api/config/sidekiq.yml, so the embedded launcher was pure redundancy — it produced roughly 5 duplicate Sidekiq instances in staging, all consuming the same 7 queues (critical, ai_workforce, mailers, default, low, active_storage_analysis, active_storage_purge). The gotcha wasn't just wasted concurrency: the embedded launcher subscribing all 7 queues on the API service inflated the API's apparent required IAM surface to match the worker's, which is what makes this relevant to the v4.5 least-privilege effort below.

Why it wasn't caught sooner: the deploy manifest comment, a senior code review, and a first diagnostic pass all independently treated the command: pin as proof the API couldn't run Sidekiq — none of them checked whether a config/initializers/*.rb file started one anyway. Non-obvious enough to bite three separate lines of defense.

Fix (OBJ-1445, PR #1536): deleted sidekiq_embedded.rb outright — the worker's explicit bundle exec sidekiq process is the only Sidekiq consumer now. Corrected the manifest comments in both deploy/service.*.yaml files to state the real invariant: the web/worker split is real because no initializer starts a consumer, not because of the command: pin. Added a regression guard, rails_api/spec/config/no_embedded_sidekiq_spec.rb, asserting the initializer file doesn't exist and that no application code outside the worker CLI constructs Sidekiq::Launcher.

Decision record — the API service account's storage.objects.delete grant stays, permanently: gcp_setup/08_runtime_service_accounts.sh grants enkidu-run-api-*'s service account roles/storage.objectAdmin (which includes delete) on the app's media bucket. That grant is not a consequence of the Sidekiq bug and must not be narrowed now that it's fixed: GoalTracking::UpdateGoal#handle_media (rails_api/app/interactions/goal_tracking/update_goal.rb:244-246) calls .purge synchronously, in the web request path, whenever a goal's image is explicitly cleared — no Sidekiq job involved. The API process itself deletes the blob directly, so the API SA needs delete permission on the bucket regardless of what the worker does. This is documented inline, directly above the grant, in 08_runtime_service_accounts.sh.

Least-privilege lesson for v4.5: a grant's true minimum is defined by which processes actually execute which code — not by an entrypoint pin, and not by which manifest a service's command: happens to reference. The embedded initializer had inflated the API's apparent required surface to the union of all 7 Sidekiq queues; closing that gap is what makes an honest least-privilege enkidu-run-api-* service account possible under the v4.5 runtime least-privilege milestone (see gcp_setup/08_runtime_service_accounts.sh).

GCP Infrastructure

Project

• Project ID: enkidu-488723
• Region: us-central1

Cloud SQL

Staging and production use separate Cloud SQL instances — no shared blast radius (OBJ-3416), the same isolation pattern already applied to Memorystore (OBJ-468, see below).

• Production instance: enkidu-postgres (PostgreSQL 15, db-f1-micro)
  • Database: objectuve_production
  • User: enkidu
  • Also backs preview environments and one-off Cloud Run jobs.
• Staging instance: enkidu-postgres-staging (PostgreSQL 15, db-f1-micro, us-central1-a, 10GB SSD zonal — dedicated as of OBJ-3416)
  • Database: objectuve_staging
  • User: enkidu_staging (holds cloudsqlsuperuser by default — this closes the shared-credential gap with production, but is not a privilege reduction)
  • Connection name: enkidu-488723:us-central1:enkidu-postgres-staging
  • Public IP, no private network/authorized networks configured — matches enkidu-postgres's existing posture (parity, not a new exposure, and not a hardening either)

Why the split — the connection-ceiling consequence: before OBJ-3416, staging shared enkidu-postgres with production. That instance's db-f1-micro tier caps out at 25 max_connections, and production alone held it at roughly 52% utilization at rest (13/25, per /health). A shared instance's connection ceiling is consumed by deploy-time revision overlap — the old revision draining while the new one warms its connection pool briefly doubles demand — and with that little headroom left, concurrent deploys across environments could fail outright. This is what caused four consecutive staging deploy failures traced by Orion (OBJ-3154, 2026-09-01). Splitting staging onto its own instance removes the cross-environment coupling entirely and hands production the full 25 connections back. See Database Connection Exhaustion and Connection-Pool Cliff in the incident-response runbook for the operational angle.

Staging's pre-split data was deliberately abandoned, not migrated (Josh's call) — enkidu-postgres-staging's objectuve_staging database starts empty and is reseeded via the existing enkidu-demo-reset-staging job. The old objectuve_staging database on the shared enkidu-postgres instance still exists but is unused; decommissioning it is a separate, gated ticket. Provisioning of the new instance/database/user/secret is performed by gcp_setup/11_staging_cloudsql.sh, which executes the real gcloud sql/gcloud secrets mutations — a human runs it (not an agent), since no runtime service account can create a Cloud SQL instance under the v4.5 least-privilege model.

Known discrepancy — do not use as a provisioning reference for production: gcp_setup/02_create_resources.sh creates a database literally named enkidu_production, but that name has never matched the real production database. rails_api/config/database.yml's production block defaults to objectuve_production when DATABASE_NAME is unset — and no deploy/*.yaml manifest or .github/workflows/*.yml job ever sets DATABASE_NAME — so objectuve_production (confirmed live via /health and the restore drill in disaster-recovery.md) is what every real deploy connects to. Fixing the script's literal string is a separate follow-up.

Memorystore Redis

Staging and production use separate Memorystore instances — no shared blast radius (OBJ-468).

• Production instance: enkidu-redis (Redis 7.0, Basic tier, 1GB)
  • Internal IP: 10.234.96.43 (VPC-accessible only)
• Staging instance: enkidu-redis-staging (Redis 7.0, Basic tier, 1GB, us-central1, reserved range 10.79.34.40/29)
  • Internal IP: 10.79.34.43 (VPC-accessible only)
• URL format: redis://<internal-ip>:6379/<db-index>
• DB-index convention: Rails API uses /1, LiteLLM uses /2 (per instance)
• Used by: ActionCable (WebSocket pub/sub), Sidekiq (background jobs)
• Cloud Run reaches Redis via VPC egress (private-ranges-only)

Cloud Run Services

• enkidu-api-production — Production Rails API
• enkidu-api-staging — Staging Rails API
• Both use Cloud SQL Auth Proxy (--set-cloudsql-instances) with Unix socket connection
• VPC egress: private-ranges-only

Cloud Run Jobs

• enkidu-db-migrate — Production migrations
• enkidu-db-migrate-staging — Staging migrations
• enkidu-flag-check-staging — On-demand, read-only staging feature-flag check (OBJ-1225 P2) — see Read-only staging console checks

Runtime service accounts (least-privilege, v4.5)

Every Cloud Run workload runs as a dedicated, per-workload-class, per-environment service account — never the project's default compute SA (388634525804-compute@developer.gserviceaccount.com). This replaced a prior model where all 6 Cloud Run services, 22 Cloud Run jobs, and the enkidu-preview-db VM ran as the default compute SA, which held the broad roles/editor primitive role project-wide. Migration and strip shipped across milestone v4.5 — Runtime Least-Privilege (parent OBJ-1353); as of 2026-07-18 the default compute SA holds zero project-level IAM roles. The enkidu-preview-db VM was migrated too, then deleted on 2026-07-17 — the project has no GCE instances (gcloud compute instances list and compute disks list both return zero rows; re-verified 2026-09-05). Its enkidu-preview-db-vm@ service account was itself deleted on 2026-09-05 as zero-blast-radius orphan cleanup (zero IAM bindings, no impersonation grants, no user-managed keys, no referencing Cloud Run services/jobs, zero Asset Inventory IAM-policy hits) — recoverable for 30 days (until ~2026-10-05) via gcloud iam service-accounts undelete 105943331296695590995 --project=enkidu-488723.

Provisioning lives in gcp_setup/08_runtime_service_accounts.sh — idempotent, resource-scoped grants only, no service account keys. Roles by workload class:

Class	SA name pattern	Roles
api	enkidu-run-api-{env}@...	cloudsql.client (project-level, no sub-project scope exists); secretmanager.secretAccessor per secret referenced in deploy/service.*.yaml; storage.objectAdmin on the env's own GCS bucket (enkidu-storage / enkidu-storage-staging); run.invoker on enkidu-litellm-{env}; storage.objectViewer on enkidu-gdpr-exports-{env}; iam.serviceAccountTokenCreator on itself (signBlob fallback)
worker	enkidu-run-worker-{env}@...	Same cloudsql.client / secrets / storage.objectAdmin / run.invoker grants as api, per deploy/worker.*.yaml's secret list, plus storage.objectCreator (write-only) on enkidu-gdpr-exports-{env}
jobs	enkidu-run-jobs-{env}@...	cloudsql.client; secretmanager.secretAccessor per secret only
litellm	enkidu-run-litellm-{env}@...	secretmanager.secretAccessor on its own secrets only — no other GCP access
preview-db-vm (VM deleted 2026-07-17; SA deleted 2026-09-05)	enkidu-preview-db-vm@...	None — the VM's startup script made zero GCP API calls

Explicitly not granted to any runtime SA: roles/firebase.admin, roles/storage.admin, roles/run.admin, roles/editor.

Why the API keeps storage.objectAdmin (delete included) permanently: GoalTracking::UpdateGoal#handle_media (rails_api/app/interactions/goal_tracking/update_goal.rb) purges media synchronously in the web request path, not via a background job — the API process itself needs delete access on the bucket. Don't narrow this grant when reasoning about what the worker needs; the two are independent.

Two firebase.admin bindings — do not conflate them. github-actions-deployer@... (the CI deployer identity, not a runtime SA) still legitimately holds roles/firebase.admin (gcp_setup/03_service_accounts.sh) — it needs it for the Firebase Hosting deploy step in production.yml / staging.yml / preview.yml. The default compute SA's roles/firebase.admin was removed as part of the v4.5 strip; it was verified unused because FCM push delivery (rails_api/app/services/fcm_delivery.rb) builds its own credentials from the static-key FIREBASE_SERVICE_ACCOUNT_JSON secret via Google::Auth::ServiceAccountCredentials, and never consults the runtime SA's IAM roles at all. Remove firebase.admin only from the default compute SA — touching github-actions-deployer's binding breaks every frontend deploy.

services replace is declarative; services update/deploy/jobs update are imperative — they reset on omission in opposite directions.

• gcloud run services replace <yaml> — the whole service spec is replaced by what's in the file. Omitting spec.template.spec.serviceAccountName silently resets the service back to the default compute SA on the next replace. production.yml and staging.yml deploy enkidu-api-{env} / enkidu-worker-{env} this way, from deploy/service.*.yaml / deploy/worker.*.yaml — both manifests pin serviceAccountName explicitly.
• gcloud run deploy / gcloud run jobs create / gcloud run jobs update — imperative. An unspecified --service-account flag preserves whatever identity the resource already has; only a resource being created fresh falls back to the default compute SA. Every such invocation across .github/workflows/ carries an explicit --service-account flag.
• gcloud run services update-traffic (used by rollback.yml's emergency rollback) creates no revision and carries no identity — it's out of scope for both checks above.

CI guard: scripts/check-runtime-service-account.mjs (+ .test.mjs) fails the build if any tracked deploy/*.yaml lacks serviceAccountName, or any gcloud run jobs create/update / gcloud run deploy in any tracked .github/workflows/*.yml lacks --service-account. Every file is discovered via git ls-files — never a hardcoded filename list — so a newly added manifest or workflow is covered automatically. Wired into its own workflow, runtime-service-account-guard.yml (PR #1579). The preview-DB VM's creation script, gcp_setup/04_preview_db_vm.sh, now also passes --service-account=enkidu-preview-db-vm@... and --no-scopes on gcloud compute instances create, so a re-provision can't silently default back onto the compute SA — the guard script covers Cloud Run only, this one's enforced by the script itself.

Key decision — legacy services deleted, not migrated: enkidu-agent-runner-{production,staging} were standalone Cloud Run services left running on the default compute SA. Reading the deploy workflows before migrating them overturned the original plan (provision dedicated enkidu-run-agent-runner-{env}@ SAs) — both production.yml and staging.yml already documented these as legacy standalone services, nothing in .github/workflows/ deploys them, and the actual live agent-runner workload runs as a sidecar container inside enkidu-worker-{env} (deploy/worker.*.yaml, Sidekiq → localhost:4001 → agent-runner → localhost:3000/webhooks), already on the dedicated enkidu-run-worker-{env}@ SA. The two standalone services showed 0 HTTP requests over 30 days and were deleted rather than migrated, along with two dormant one-off jobs (enkidu-clerk-migrate, enkidu-make-admin). Lesson for any future SA/workload migration: verify a workload is actually deployed and live before migrating it — reading the code that deploys (or doesn't deploy) it is cheaper than provisioning infrastructure for something already dead.

dave-autopilot diagnostics role — IAM posture decisions (OBJ-3232)

dave-autopilot@enkidu-488723.iam.gserviceaccount.com powers Multica's Dave autopilot (staging health checks, OBJ-3160/OBJ-3171). It authenticates via a non-expiring USER_MANAGED JSON key (~102 days old as of OBJ-3232; OBJ-3218 tracks migrating it to Workload Identity Federation, no timeline yet), loaded into the multica daemon via launchd GOOGLE_APPLICATION_CREDENTIALS on the runner host — any process on that host with that env var can use it. Its project-level bindings: the custom role projects/enkidu-488723/roles/daveAutopilotDiagnostics (9 permissions as of OBJ-3232, widened to 11 by OBJ-3956; IaC record at gcp_setup/dave-autopilot-diagnostics-role.yaml, applied via gcp_setup/12_dave_autopilot_diagnostics_role.sh and gcp_setup/21_dave_autopilot_monitoring_read_role.sh), plus roles/cloudsql.client, roles/logging.viewer, roles/run.viewer.

Given the long-lived, broadly-accessible static key backing this SA, its permission set matters more than usual. Two posture questions carried forward from OBJ-3160's closure were decided on OBJ-3232:

Decision 1 — resourcemanager.projects.getIamPolicy removed (10 → 9 permissions). This permission reads the entire project IAM policy — every principal and every binding on enkidu-488723 — which is high-value privilege-escalation reconnaissance for anyone holding the key. A sweep of all 32 autopilot definitions in the workspace (not just Dave's 7) found none read any IAM policy; Dave's actual GCP surface is gcloud run services describe, gcloud run revisions list, gcloud logging read, curl /health, and Sentry REST. The one recorded live use (OBJ-3158) was Codi checking "is my own PERMISSION_DENIED real?" after a run.services.update denial — a fluke-check fully served by the testIamPermissions REST method (needs no IAM grant to run — see the gotcha below). No substitute permission was added: iam.serviceAccounts.getIamPolicy answers a different question (who can impersonate the SA, not what it holds — see the companion gotcha) and would serve neither the fluke-check nor a hypothetical incident-escalation lookup, and the latter has never been exercised in this crew's history. Applied live 2026-09-06; validated by a Dave rerun — all 3 OBJ-3171 probes unchanged, getIamPolicy confirmed denied, testIamPermissions confirmed working, monitoring.alertPolicies.list confirmed still denied (no regression).

Decision 2 — no grant of monitoring.alertPolicies.create to dave-autopilot. This SA's entire grant is read-only today, bounding worst-case blast radius to information disclosure; alertPolicies.create would be its first write permission, on the credential OBJ-3218 exists to retire. The cost of "no" is concrete, not hypothetical: infra/monitoring/alert-policy-cloudsql-connections-enkidu-postgres.json (OBJ-3156, merged 2026-09-02; renamed from its original misleading -staging filename under OBJ-3419) is a reviewed, ready Cloud SQL connection-saturation alert that stays unprovisioned without this permission granted somewhere. Rather than grant it to this SA — or fall back to a human hand-running gcp_setup/14_create_cloudsql_connection_alert.sh, which had already gone unrun for 80+ minutes after merge by the time this decision was made — the grant goes instead to github-actions-deployer@enkidu-488723.iam.gserviceaccount.com, the already-keyless, WIF-federated CI deploy SA every .github/workflows/*.yml already authenticates as, via a new tightest-fit custom role (monitoring.alertPolicies.create only) invoked from a workflow step on merge to master. That grant and workflow step are tracked separately (OBJ-3232 Task 6) — this SA's own posture doesn't change. The role holds exactly two permissions, monitoring.alertPolicies.create and .list (the latter for gcp_setup/14_create_cloudsql_connection_alert.sh's pre-create idempotency check, also used unmodified by its OBJ-3419 staging sibling gcp_setup/15_create_staging_cloudsql_connection_alert.sh) — IaC record at gcp_setup/github-actions-alert-policy-role.yaml, applied via gcp_setup/13_github_actions_alert_policy_role.sh (hand-run by Josh, not wired into CI) and invoked from the provision-cloudsql-alert-policy job in staging.yml. No resource-level IAM condition is available to scope either grant further: Cloud Monitoring's lowest grantable level for any role carrying alertPolicies.create is Project, and Monitoring is absent from the services supporting resource.type/resource.name IAM conditions — verified via the Monitoring v3 API discovery document, the full gcloud monitoring command tree (no get-iam-policy/set-iam-policy subcommand anywhere), and gcloud iam list-testable-permissions.

Decision 3 — alert-policy read granted, roles/monitoring.viewer rejected (9 → 11 permissions, OBJ-3956). Dave's diagnosis of the staging DatabaseConnectionError incident (OBJECTUVE-API-2Z) hit a verification gap: it couldn't confirm whether the Cloud SQL Connection Saturation alert policy (Decision 2, above) is actually live for staging, because this role held no alert-policy permission at all. monitoring.alertPolicies.list and monitoring.alertPolicies.get were added — the minimum needed to list and inspect a policy. The predefined roles/monitoring.viewer was rejected: granting it would silently reverse Decision 1's deliberate 10-to-9 narrowing (it bundles far more than alert-policy read, including resourcemanager-adjacent monitoring surface this role was specifically trimmed away from). monitoring.notificationChannels.list/.get were considered and dropped: the AlertPolicy resource returned by alertPolicies.get already carries its own notificationChannels field (the array of channel resource names it's bound to — see gcp_setup/15_create_staging_cloudsql_connection_alert.sh, whose NOTIFICATION_CHANNEL_ID is optional and, unset, leaves a policy that exists but pages nobody), so confirming a policy is attached to a channel needs no separate notification-channel read. Only inspecting a channel's own configuration (e.g. is it a valid, enabled destination) would need that permission, and no Dave workload does that today. Not yet applied live as of this PR — requires a human with Project IAM Admin (or equivalent) on enkidu-488723 to hand-run gcp_setup/21_dave_autopilot_monitoring_read_role.sh (same --add-permissions surgical-mutation convention as Decision 1's script); this PR lands the IaC record only.

Full decision record and evidence trail: OBJ-3232 (Decisions 1–2), OBJ-3956 (Decision 3).

Key rotation procedure (OBJ-3560). This SA's non-expiring USER_MANAGED key (above) has no auto-rotation, so any rotation is a manual sequence. Do the steps in this order — verify before you delete:

1. Create the new key on the service account (GCP Console or gcloud iam service-accounts keys create).
2. Write it to ~/.config/dave-autopilot/key.json on the Multica daemon host (the path GOOGLE_APPLICATION_CREDENTIALS points at for the daemon's launchd job — see above).
3. Kickstart the daemon so it picks up the new file.
4. Run scripts/multica-host/ensure-gcloud-sa-credential.sh on that host to re-sync the gcloud CLI's own cached credential to the new key — the daemon's env var and gcloud's on-disk credential store are two independent caches (see the gcloud CLI credential cache gotcha), and only this step re-syncs the second one.
5. Verify with a real call — gcloud run services describe enkidu-api-staging --region us-central1 --project enkidu-488723 --format 'value(status.url)' should return a URL, not an invalid_grant: Invalid JWT Signature error.
6. Only then delete the old key from the service account.

The ordering is the fix, not a formality: OBJ-3346's rotation deleted the old key before confirming step 5, which turned a recoverable cache-staleness gap into a hard invalid_grant outage across every gcloud-shelling Dave/Vicki autopilot (OBJ-3560) — deleting first removed any way for the stale cached credential to keep working while the mismatch got fixed.

Limit of this procedure. Step 4's preflight script only re-syncs the credential cache on the host it's run on. It self-heals autopilot runs that call scripts/multica-host/ensure-gcloud-sa-credential.sh at the top of their own runbook (see docs/guides/multica-autopilots.md) — it does not self-heal a human, or any other agent runtime, invoking gcloud directly under this SA's key on a different host or outside an autopilot run. Anyone rotating this key needs to re-activate it (step 4, or a bare gcloud auth activate-service-account) on every host or process that uses the CLI under this SA, not just the daemon host.

Rotating this key is a recurring chore tracked on OBJ-3346 — that issue should link back to this section rather than rediscovering the procedure from scratch on the next rotation.

Firebase Hosting

• enkidu-app — Production frontend (app.objectuve.com)
• enkidu-app-staging — Staging frontend (staging.app.objectuve.com)
• enkidu-marketing — Marketing site

Artifact Registry

• Repository: enkidu-registry
• Location: us-central1-docker.pkg.dev/enkidu-488723/enkidu-registry
• Image tags: rails-api:{sha} (immutable), rails-api:staging-latest (pointer), rails-api:v3.5.0 (version alias)

Secrets (GCP Secret Manager)

• production-db-password — Production database password
• staging-db-password — Staging database password
• rails-master-key — Rails credentials encryption key (shared)
• clerk-secret-key — Clerk secret key (production)
• clerk-secret-key-dev — Clerk secret key (staging)
• stripe-secret-key — Stripe secret key, live mode (production only)
• stripe-webhook-secret — Stripe webhook signing secret, live mode (production only)
• stripe-secret-key-staging — Stripe secret key, test mode (staging only, OBJ-1225)
• stripe-webhook-secret-staging — Stripe webhook signing secret, test mode (staging only, OBJ-1225)
• posthog-personal-api-key — PostHog personal API key for read access to the management REST API (flag/cohort definitions); not injected into any Cloud Run service — see Read-only staging console checks

Staging and production must never share a STRIPE_* secret name — see Staging silently read production's live-mode Stripe secrets by name collision.

GitHub Actions Secrets

• GCP_PROJECT_ID — enkidu-488723
• GCP_SERVICE_ACCOUNT — github-actions-deployer@enkidu-488723.iam.gserviceaccount.com
• GCP_WORKLOAD_IDENTITY_PROVIDER — Workload Identity Federation provider path

DNS Configuration (Namecheap)

Domain: objectuve.com

Type	Host	Value	Purpose
CNAME	api	ghs.googlehosted.com.	Production API (Cloud Run)
CNAME	staging.api	ghs.googlehosted.com.	Staging API (Cloud Run)
CNAME	app	enkidu-app.web.app.	Production frontend (Firebase)
CNAME	staging.app	enkidu-app-staging.web.app.	Staging frontend (Firebase)
CNAME	www	enkidu-marketing.web.app.	Marketing site (Firebase)
CNAME	help	enkidu-guide.web.app.	Public user guide (Firebase), OBJ-1852
TXT	@	google-site-verification=...	Domain verification

SSL certificates are automatically managed by Google for all custom domains.

Note: admin, support, and docs subdomains, if they exist, are not yet tracked in this table — pre-existing gap, not part of this update.

Note: feedback (intended feedback.objectuve.com → enkidu-feedback.web.app.) has no row here yet because it isn't mapped — the underlying GCP Firebase Hosting site was never provisioned. See Independent surface lanes above and the gotcha write-up (OBJ-3864).

GCP Setup Scripts

Located in gcp_setup/:

Script	Purpose
01_enable_apis.sh	Enable required GCP APIs
02_create_resources.sh	Create Cloud SQL, Redis, Artifact Registry, GCS buckets (production enkidu-storage + staging enkidu-storage-staging, OBJ-1341)
03_service_accounts.sh	Set up Workload Identity Federation for GitHub Actions
04_create_mobile_builds_bucket.sh	Create GCS bucket for mobile build artifacts
04_litellm_iam.sh	Set up IAM permissions for the LiteLLM service
04_preview_db_vm.sh	Create preview environment database VM
05_staging_resources.sh	Create staging secrets and Firebase hosting site (Cloud SQL staging resources moved to 11_staging_cloudsql.sh, OBJ-3416)
06_custom_domains.sh	Map custom domains to Cloud Run and Firebase
07_create_backups_bucket.sh	Create private enkidu-backups GCS bucket for DB export dumps (uniform bucket-level access + public-access-prevention, 90-day lifecycle, Cloud SQL SA granted roles/storage.objectAdmin; OBJ-1342)
07_create_gdpr_exports_buckets.sh	Create enkidu-gdpr-exports-{production,staging} GCS buckets for user data-export bundles (OBJ-1446)
08_runtime_service_accounts.sh	Provision the 9 dedicated least-privilege runtime service accounts (4 workload classes × 2 envs + preview-DB VM) — see Runtime service accounts
11_staging_cloudsql.sh	Provision the dedicated staging Cloud SQL instance: enkidu-postgres-staging, its objectuve_staging database, enkidu_staging SQL user, and staging-db-password secret version (OBJ-3416) — executes real gcloud sql/gcloud secrets mutations, see Cloud SQL

Troubleshooting

Database connection errors

The Cloud Run services connect to Cloud SQL via the Auth Proxy (Unix socket at /cloudsql/enkidu-488723:us-central1:enkidu-postgres). If connections fail:

• Check that --set-cloudsql-instances is set on the service/job
• Ensure --vpc-egress=private-ranges-only (not all-traffic, which blocks the proxy from reaching Google APIs)
• Verify the DATABASE_HOST env var is set to /cloudsql/enkidu-488723:us-central1:enkidu-postgres

Production deploy fails at validation

The Validate Release Image step polls Artifact Registry for rails-api:{sha} for up to 5 minutes (300s timeout, 20s interval) before failing. If it does fail with Image ... not found after 300s, the staging build for that SHA was cancelled or failed and never pushed an image.

Recovery uses the workflow_dispatch rebuild path on staging.yml — no no-op commit on master required:

1. Open the Staging Deployment workflow in the GitHub Actions UI.
2. Click Run workflow and supply the missing SHA via the commit_sha input. Only build-rails-api will run; deploys are skipped and staging-latest is left alone.
3. Wait for the dispatched build to push rails-api:{commit_sha}.
4. Re-tag the release at the same SHA: git tag -d v3.5.0 && git push origin :refs/tags/v3.5.0 && git tag -a v3.5.0 -m "Release 3.5.0" <commit_sha> && git push origin v3.5.0.

See Staging build and production validate for the full mechanics. Background on why this recovery path exists: OBJ-73, and the v3.8.3 / v3.8.4 incidents (Apr 27–28, 2026) that surfaced the original race.

SSL certificate issues

Managed SSL certificates are provisioned automatically once DNS records point to the correct targets. Provisioning can take:

• Cloud Run: 15-30 minutes
• Firebase Hosting: up to 24 hours

Check Cloud Run certificate status:

gcloud beta run domain-mappings describe --domain=api.objectuve.com --region=us-central1

Viewing Cloud Run logs

# Production API logs
gcloud run services logs read enkidu-api-production --region=us-central1 --limit=50

# Staging API logs
gcloud run services logs read enkidu-api-staging --region=us-central1 --limit=50

# Migration job logs
gcloud logging read 'resource.type="cloud_run_job" AND resource.labels.job_name="enkidu-db-migrate"' --limit=20

Observability

After deploying, verify the observability stack is working:

1. Health check: curl https://api.objectuve.com/health (or staging URL) — should return 200 with all checks ok
2. Structured logs: Check Cloud Logging for JSON-formatted request logs with graphql_operation, user_id, and request_id fields
3. Error tracking: Verify Sentry is receiving events — check the Sentry dashboard for the appropriate environment
4. Performance traces: Check Sentry Performance tab for transaction traces

Observability Environment Variables

Variable	Service	Purpose
SENTRY_DSN	Rails API (Cloud Run)	Sentry DSN for backend error tracking
SENTRY_RELEASE	Rails API (Cloud Run)	Version tag (e.g., v3.5.0) for Sentry release tracking
VITE_SENTRY_DSN	Frontend (Firebase Hosting build)	Sentry DSN for frontend error tracking
VITE_SENTRY_ENVIRONMENT	Frontend (Firebase Hosting build)	Sentry environment label; overrides import.meta.env.MODE. Injected by CI per-deploy (staging / production)
CSP_REPORT_URI	Rails API (Cloud Run) only	Sentry CSP endpoint for security header violations. Optional — omit to skip CSP violation reporting. Not currently set in any environment

CSP_REPORT_URI setup:

To enable Content Security Policy violation reporting to Sentry:

1. Log into Sentry
2. Navigate to Project Settings → Security & Privacy → CSP Reports
3. Copy the CSP report URI endpoint (e.g., https://o12345.ingest.sentry.io/api/67890/security/?sentry_key=xxxxxxxx)
4. Set CSP_REPORT_URI to this value in Cloud Run secrets
5. The API's Content-Security-Policy header will include the report-uri directive — browsers will POST violation reports to Sentry. The policy is enforcing either way (OBJ-486); this variable only controls whether you find out when something is blocked

This variable has never been provisioned, so the 2026-05-21 → 2026-08-03 report-only window produced no data. Verify the directive actually appears in a live response (curl -sI https://api.objectuve.com/health | grep -i content-security) after setting it — an unset variable is silently omitted, which is indistinguishable from "no violations" at the Sentry end.

The SPA (app.objectuve.com) does not use this variable. Its CSP is a Firebase Hosting static header (firebase.json), not a Rails middleware response — firebase.json has no environment interpolation, so its report endpoint is a literal URL committed to the file instead of an env var. Setting or changing CSP_REPORT_URI has zero effect on the SPA's policy. See Security Headers § SPA Content Security Policy for the full mechanism and why hardcoding the URL there is safe.

See Security Headers for the full CSP policy and current enforcement mode on both origins.

PostHog key in marketing site: The marketing landing site (marketing_landing/index.html) contains a hardcoded PostHog project key (phc_CcuNbgW2uNpBrK2yMs8mcezGbC3aJ9CDwCinM76idWEd). PostHog client keys are public-facing by design — this key is the same value as VITE_POSTHOG_KEY from ionic_frontend/.env and sourced from .github/workflows/staging.yml. No substitution step is required at deploy time, but the key must remain in sync if the PostHog project ever changes. See PostHog Acquisition Funnel Events for event details.

For full observability documentation, see Observability, Alerting, and Dashboards.

Deep Link Configuration (iOS & Android)

Objective supports deep links for accountability-partner invites (/invite/{token}) to bootstrap partnerships at signup.

iOS: Apple App Site Association (AASA)

The AASA file (marketing_landing/.well-known/apple-app-site-association) declares which app-domain links should route to the native app (vs. the web):

{
  "applinks": {
    "apps": [],
    "details": [
      {
        "appID": "APPLE_TEAM_ID.com.objectuve.ionic",
        "paths": ["/goals/*", "/communities/*", "/sign-in", "/sign-up", "/public_goal/*", "/invite/*"]
      }
    ]
  },
  "webcredentials": {
    "apps": ["APPLE_TEAM_ID.com.objectuve.ionic"]
  }
}

Deployment requirement: Before the partner-invite-enabled flag ramps on iOS production, the placeholder APPLE_TEAM_ID must be replaced with the actual Apple Developer Team ID. This is a critical ops action because:

1. iOS checks the AASA file when a user has the app installed
2. If appID contains the literal string APPLE_TEAM_ID, iOS will silently fail to recognize /invite/* links as native app links
3. Users will open invite links in Safari (not the app), breaking the signup flow

Steps:

1. Get the Apple Team ID from the Apple Developer account
2. Replace APPLE_TEAM_ID in marketing_landing/.well-known/apple-app-site-association
3. Commit and redeploy the marketing site (or the main app if AASA is served from the app domain)
4. Wait 24 hours for Apple's CDN to refresh the cached AASA file
5. Notify the team that iOS deep links are now live; coordinate the feature-flag ramp

The AASA file is served at https://app.objectuve.com/.well-known/apple-app-site-association (exact domain from the applinks section).

Android: Firebase Dynamic Links

Android deep links are configured via Firebase Dynamic Links. The routing logic is wired in Capacitor (capacitor.config.ts) and the native Android app's URI handlers.

Current paths claimed:

• /goals/*, /communities/*, /sign-in, /sign-up, /public_goal/*, /invite/*

Firebase Dynamic Links are managed via the Firebase console; no deployment action needed in the codebase (already configured). Verify routing by:

1. Generating a short link in the Firebase console
2. Testing on Android with the app installed (should open native app)
3. Testing on Android without the app installed (should route to Play Store)

For detailed Capacitor + Firebase Dynamic Links setup, see native-mobile-exploration.md.

PWA Service Worker Deployment & Rollback

The Progressive Web Application layer (v1.15+) includes a service worker (sw.js) and web app manifest (manifest.webmanifest) distributed via Firebase Hosting alongside the frontend SPA.

Service Worker Update Flow

The service worker uses registerType: 'prompt' and injectRegister: false (manual registration in src/main.ts). This means:

• Users must refresh the browser to activate a new SW version
• A persistent toast (usePwaUpdateToast.ts) prompts: "Update ready → Refresh"
• Workbox's skipWaiting is triggered on user click, activating the new SW immediately
• On next page load, the new SW serves updated assets

Emergency: Kill-Switch Service Worker

If a broken SW version is deployed:

Option 1: CDN Cache-Control Header (fastest)

Set Cache-Control: no-store on https://app.objectuve.com/sw.js at the Firebase Hosting CDN edge. This forces browsers to always fetch a fresh sw.js:

firebase hosting:channel:deploy <CHANNEL> --expires 1h

Then update sw.js in the build (code fix), redeploy, and remove the no-store header after confirming new SW propagates.

Option 2: Workbox skipWaiting Escape Hatch (immediate)

Build and deploy a new version containing self.skipWaiting() at the top of the SW entry. This forces any already-installed SW versions to immediately retire themselves. The new version then serves corrected Workbox config and precache manifest.

Workbox Precache Invalidation

Workbox's precache manifest (generated at build time by vite-plugin-pwa) includes hashes of all app shell assets (JS, CSS, HTML, fonts, images). If assets change:

• A new build regenerates the manifest with new hashes
• On next visit, browsers detect hash mismatches
• Outdated cache entries are purged automatically
• New assets are fetched

No manual cache clearing is needed unless precache is corrupted. To manually verify:

// In browser console
caches.keys().then(names => console.log(names))
// Look for 'workbox-precache-v2' — entries should be <hash>:<resource>

Testing PWA Deployment

1. Local build: npm run build in ionic_frontend/, then npm run preview to serve dist/ locally
2. Verify artifacts: dist/sw.js and dist/manifest.webmanifest should be present
3. Staging deploy: Push to master — Firebase auto-deploys to enkidu-app-staging.web.app
4. Staging validation:
   • Open DevTools → Application → Service Workers → should show one registration
   • Offline simulation via DevTools → Network → Offline → app should show offline shell
   • Manual refresh trigger: toast should appear; click "Refresh" → page reloads with new SW

Last updated: 2026-09-17 (OBJ-3917: release preflight reads spec-provenance report, not raw run-list conclusions)