Operations

Deploying, testing, monitoring, and maintaining Objectuve across staging and production environments.

Sections

• Deployment — GCP Cloud Run + Firebase Hosting infrastructure, CI/CD pipelines, environment management, versioning, rollbacks, and CLI shortcuts (release, rollback, deployed)
• Security Headers — HTTP security headers (HSTS, CSP, X-Frame-Options, etc.), Content Security Policy directives, and violation reporting to Sentry
• Testing — RSpec (backend, 80% coverage target), Vitest (frontend unit), Cypress (E2E + smoke), coverage reporting, and CI integration
• Mobile Builds — iOS/Android builds via Capacitor, code signing, hot reload, App Store/Play Store submission, and pre-merge release-build safety gates (cap-sync drift, minified/Release boot smoke, plugin-registration check)
• Store Release Notes — per-release mobile store copy (Google Play "What's new", App Store "What's New in This Version", App Review notes), sourced from the tagged ionic_frontend/CHANGELOG.md; posted by Vicki as the weekly release train's last act
• Game Balance — Game-balance parameter tuning: XP cost formulas, progression curves, gamification settings (Streak Repair costs, etc.)
• Observability — Sentry error tracking + APM, Lograge structured JSON logging, GCP Cloud Logging, health check endpoint
• Rails Concurrency & Capacity — Puma thread / ActiveRecord pool / containerConcurrency / maxScale sizing model, deploy/capacity-budget.json, the check-deploy-capacity.mjs CI guard, and the disclosed known limitation (production capped below its measured connection peak)
• Alerting — Slack-based budget alerts, Sentry alert rules, uptime monitoring
• Dashboards — Admin dashboard overview, GCP metrics, Sentry dashboards, PostHog product analytics (event taxonomy + weekly metrics review)
• AI Runbook — AI service operational guide: LiteLLM proxy, rate limiting, caching, kill switches, budget monitoring
• Docs & Guide Assistant Operations — v4.55 "Ask the guide" / "Ask the docs" runtime controls: the docs_assistant_enabled kill switch, the enforcing per-feature budget ceiling and what happens when it trips, per-surface rate limits, and the docs corpus refresh rake task (monorepo-checkout-only, never the deployed container)
• Agent Runner Ops — Agent Runner deployment, monitoring, troubleshooting, scaling considerations
• Incident Response — Solo-operator runbook: diagnosis flow, rollback tiers, common failure modes, recovery procedures
• Teams Billing Runbook — 3-step Teams (v4.0) billing incident procedure: kill teams_billing_v1, inspect the Stripe dashboard, refund (7-day window, manual)
• Stale Payment Record Reconciliation Runbook — response procedure for the personal-Supporter Billing::ReconcileStalePaymentRecordsJob alert: confirm livemode, look up the record and supporter state, then reconcile (resend the Stripe webhook event) or dismiss (soft-delete a smoke/test artifact)
• Production Test-Account Cleanup — manual procedure to fully remove a one-off account created directly against production Clerk + Rails (hard-delete via Gdpr::DeleteSelfServiceAccount + really_destroy!, then delete the Clerk-side account)
• Sidekiq Dead Set Runbook — self-serve workflow_dispatch procedure to diagnose and clear a stuck Sidekiq dead set (the shared-infra /health degraded state that blocks the Playwright smoke pre-flight for every open PR) — no local gcloud required
• Disaster Recovery — DR plan: RTO/RPO targets, failure scenarios, data backup strategy, PITR procedure, regional outage response
• AI Employee Discovery — Discovery analysis for the AI Workforce feature (employee roles, skill mapping, shadow mode criteria)
• AI Workforce Runbooks — Visual process maps for run lifecycle, review queue, delivery routing, autonomy promotion, failure recovery, and operator cadence
• Vendors — Third-party services inventory, costs, and contract terms
• GCP Cost Analysis — August 2026 GCP cost breakdown for enkidu-488723: measured Cloud Monitoring usage x Cloud Billing Catalog list prices, a ~$135/mo reconciliation gap now partially resolved against 7 complete days of BigQuery billing-export data (2026-09-11 → 2026-09-17, Phase 6b.2b-i / OBJ-3936) — ~$94/mo (~70%) measured and explained (Firebase Hosting, Secret Manager, the Cloud SQL understatement), ~$41/mo still unaccounted for — and v4.54 remediation dispositions on each recommendation — Artifact Registry cleanup policy applied and documented operationally, production minScale drops declined with evidence, Crono throttling mode recommended but not implemented, staging's 24/7 mirror of production still pending
• Billing Export Runbook — operational how-to for the billing_export BigQuery dataset: table/project/location reference, re-provisioning via gcp_setup/19_billing_export.sh, the permanent no-August-2026-data consequence of the us-central1 (not multi-region) dataset location and why it stands, sourced backfill/propagation thresholds, and the 1–3 day row-settling querying hazard
• GitHub Actions Cost Analysis — August 2026 GitHub Actions cost breakdown for objectuve-softworks/enkidu: measured org billing-feed usage x derived per-minute rates, one ci.yml job (Vitest (ionic_frontend)) accounting for 15.8% of the month's Linux minutes, a modeled attribution that overshoots ground truth by ~1.1% (the billing feed aggregates by SKU and repository only, never by workflow), and a reconciliation that closes at $922.16 Actions-only net + $89.00 base subscription = $1,011.16
• Rollouts (rollouts/) — Feature flag rollout runbooks (per-version guides for staged PostHog releases)
• CI Secrets Checklist — GitHub Actions secret setup and the "assumed secrets" gotcha: silent empty strings when secrets are missing
• Dependency Security Exceptions — documented, permanent, risk-accepted exceptions to the "keep dependencies patched" rule; currently: the image-size peer chain pulled in by @clerk/clerk-js's bundled Solana wallet-login in docs_site/design_system (OBJ-2455 → OBJ-2458, includes a general RN/Metro-family Dependabot triage pattern), and the grunt-transitive brace-expansion instance in rails_api's spectaql doc toolchain (OBJ-1877 → OBJ-1878)
• Planning Canon — .planning/ROADMAP.md (milestone index) vs. .planning/milestones/v<X.Y>-ROADMAP.md (live plan) vs. .planning/STATE.md (single-milestone scalar); the do-not-run GSD command list and why; the deferred workstreams-adoption question
• Multica Daemon Disk Hygiene — root cause + fix for the crew daemon host's task-dir disk exhaustion: native GC eligibility gates on the linked issue reaching done server-side (not on task completion or a TTL alone), so a task pinned to an issue that never closes leaks forever; 7-day retention config via MULTICA_GC_* env vars, a fail-closed dirty/mirror-ref-aware backstop sweeper, a launchd disk+GC-health monitor, and the stale git-lock repair procedure
• Gotchas & Tips — Common gotchas and solutions:
  • Firebase Preview Channels — Quota exhaustion when teardown automation is added retroactively
  • gcloud Multi-container Cloud Run — SDK regression with --port flags in multi-container services
  • GitHub Actions Force-push — Rapid force-pushes silently break CI synchronize triggers; throwaway-branch iteration workflow
  • Smoke Account Concurrency — Concurrent CI smoke suites racing an unlocked shared demo account produce intermittent UI assertion failures; prefer native concurrency: groups + resource isolation over a hand-rolled lock; audit every auth call site when isolating a shared account
  • Smoke Test Environment Gotchas — *.web.app preview URLs route to smoke_env=production; demo user onboarding state persists across runs; demo account goal-seeding guard in auth.production.setup.ts; coach persona redirect on /enneagram-assessment
  • Health Check Informational Sub-checks — INFORMATIONAL_CHECKS in Rails code does not produce a neutral JSON signal; informational sub-checks still emit "down" and trigger monitors; return not_applicable instead; Time.zone contamination in sharded RSpec
  • Sentry Environment-Filter Gotcha — the issues-list endpoint's environment= filter selects issues by event match but surfaces issue-level title/lastSeen/unresolved aggregated across all environments; a monitor reading that as environment-scoped evidence false-fires on issues shared between staging and production; query the events endpoint scoped to environment:<env> instead
  • AiUsageEvent FEATURES Allowlist Drift — a validation allowlist duplicated between a model and its distributed callers drifts silently and permanently kills background jobs; guard with a caller-behavior scan, not an enumerated directory list
  • Multica Runtime Identity (daemon.id hijack) — Multica keys runtime identity on ~/.multica/daemon.id, not hostname; a second machine with a copied daemon.id silently claims the crew runtime on its next daemon start and every agent run fails 401 OAuth access token has been revoked; check runtime ownership before debugging credentials
  • Multica Shared-Runtime Quota — all 12 crew agents share one Claude Code login (bf3c31e7-…), which carries both a 5-hour rolling session window and a separate weekly cap; exhaustion is fleet-correlated (one event can fail several unrelated autopilots at once), not per-autopilot, and there is no per-autopilot token attribution today; any cadence change must be evaluated against total fleet load, not just the one autopilot being changed
  • Personal Mac Doubles as Shared Crew Host — the crew daemon host is also Josh's personal Mac; a large majority of df -h /System/Volumes/Data usage is TCC-protected Photos/Mail/Messages/iCloud data no agent process can see or measure — don't burn an investigation cycle trying to account for it from stray checkouts alone; retry once, then route capacity decisions to a human
  • Multica gc/Maintenance Silent Failure — the daemon's per-cycle git maintenance on a shared repo mirror can fail every cycle for days while /health and runtime status stay green; the only trace is a WRN gc: git maintenance failed line; signal: killed is the daemon's own timeout kill, not a crash; the gc-maintenance-watchdog autopilot escalates after 3 consecutive failures
  • Multica Autopilot Run-Status Gotcha — a run's status/failure_reason isn't reliable fault ground truth: issue blocked runs never update after the block clears, last_run_status hides fail-then-succeed, and real faults arrive correlated across many autopilots at once; the Dave — Autopilot run failure sweep autopilot corrects for all three
  • Hallmark Gate A7 Spacing Allowlist Gotcha — gate A7's "4px grid" comment doesn't match its allowlist implementation; the 44px tap-target floor and container widths are correctly exempted (v4.51 Phase 5), but the remaining hits are real off-allowlist findings, not false positives — except two documented deliberate keeps in LifeAreaSlide.vue
  • Multica Task-Dir GC Issue-Status Gate — native task-dir GC only fires once the linked issue's server-side status reaches done; task dirs pinned to issues that never close leak silently forever (found at 82/88/96 days old) despite GC running normally every cycle and reclaiming other dirs
  • launchd Silent-Failure Forensics on joshuas-mbp-2 — log show --predicate 'subsystem == "com.apple.xpc.launchd"' is a dead predicate on this host's macOS build (0 rows, always) — use process == "launchd"; that predicate's retention is only ~22h, far shorter than the multi-day lag typical of a silent-failure discovery

Quick Reference

Environment	API URL	Frontend URL	Deploy Trigger
Local	http://localhost:3000	http://localhost:5173	Manual
Staging	Cloud Run enkidu-api-staging	Firebase enkidu-app-staging	Push to master
Production	Cloud Run enkidu-api-production	Firebase enkidu-app	Push version tag (v*)

Key Commands

# Deploy
release [version]        # Create version tag, deploy to production
rollback <version>       # Re-deploy a previous version
rollback --emergency     # Instant Cloud Run revision switch (API only)
deployed                 # Show what's running on staging and production

# Test
./test.sh                # Run all tests (RSpec + Vitest)
cd rails_api && bundle exec rspec          # Backend tests
cd ionic_frontend && npm run test:unit     # Frontend unit tests
cd ionic_frontend && npm run test:e2e      # E2E tests
cd ionic_frontend && npm run smoke:playwright    # Smoke tests (deployed env)

# Monitor
GET /health              # Component-level health check (DB, Redis, Sidekiq)

Last updated: 2026-09-18 (OBJ-3944: synced GCP Cost Analysis summary with gcp-costs.md's partially-resolved reconciliation-gap status)