Incident Response Runbook

Solo-operator runbook for Objectuve production incidents. Currently a one-person operation — no on-call rotation. This document covers common failure modes, diagnosis steps, and recovery procedures.

Quick Reference

Service	Cloud Run Name	Health Check	Rollback
Rails API	enkidu-api-production	GET /health	rollback --emergency
Frontend	Firebase enkidu-app	N/A (static)	rollback <version>
Agent Runner	enkidu-agent-runner-production	GET /health	Redeploy previous image
LiteLLM	enkidu-litellm-production	GET /health/liveliness	Redeploy previous image

Diagnosis Flow

User reports issue
  │
  ├─ Check /health endpoint → component-level status
  │   ├─ database: failed → See "Database Issues"
  │   ├─ redis: failed → See "Redis Issues"
  │   ├─ sidekiq: failed → See "Sidekiq Issues"
  │   └─ litellm: failed → See "LiteLLM Issues"
  │
  ├─ Check Sentry → recent exceptions, error trends
  │
  ├─ Check GCP Cloud Logging
  │   └─ gcloud logging read 'jsonPayload.status >= 500' --project=enkidu-488723 --limit=20
  │
  └─ Check Cloud Run metrics → request latency, error rate, instance count

Tier 1: Emergency Rollback (Seconds)

For production-breaking deployments. Instantly switches Cloud Run traffic to the previous revision.

rollback --emergency

Manual equivalent:

# List recent revisions
gcloud run revisions list --service=enkidu-api-production --region=us-central1 --limit=5

# Switch to previous revision
gcloud run services update-traffic enkidu-api-production \
  --region=us-central1 \
  --to-revisions=PREVIOUS_REVISION=100

Note: Database migrations are forward-only. Rollbacks do NOT undo migrations. This is safe when migrations only add columns/tables (old code ignores them).

Tier 2: Planned Rollback (Minutes)

Re-deploys a specific version. Use when the issue is identified and a known-good version exists.

rollback v3.5.0          # Specific version
rollback staging          # Revert last staging commit

Common Failure Modes

Clerk Outage (Authentication Down)

Symptoms: Users can't sign in, 401 errors on all GraphQL requests, Sentry flooded with JWT verification failures.

Diagnosis:

# Check Clerk status
curl https://status.clerk.com/api/v2/status.json

# Check Sentry for ClerkJwtVerifier errors
# Sentry → Issues → filter: "ClerkJwtVerifier"

Recovery:

1. Enable fallback auth on staging (if needed for admin access):

   gcloud run services update enkidu-api-staging \
     --set-env-vars="FALLBACK_AUTH_SECRET=$(openssl rand -hex 32)" \
     --region=us-central1 --project=enkidu-488723

2. Access admin dashboard via /sign-in?fallback
3. Monitor Clerk status page for resolution
4. After Clerk recovers, remove FALLBACK_AUTH_SECRET from staging env

Note: Do NOT enable fallback auth in production unless the outage exceeds 4+ hours and user impact is critical.

LiteLLM Timeout / AI Service Down

Symptoms: AI features return errors, coach insights fail, check-in prompts not generated.

Diagnosis:

# Check LiteLLM health
curl https://enkidu-litellm-production-HASH.run.app/health/liveliness

# Check /health endpoint for litellm component
curl https://api.objectuve.com/health | jq '.checks.litellm'

Recovery — Kill Switches (instant, no deploy required):

Edit rails_api/config/settings/production.yml:

ai:
  coaching_enabled: false    # Disables user-facing AI (coach, milestones)
  moderation_enabled: false  # Disables content screening
  batch_enabled: false       # Disables insights, check-ins, descriptions

Commit, push, and deploy:

git add rails_api/config/settings/production.yml
git commit -m "fix: disable AI services (LiteLLM outage)"
git push origin master
release  # Deploy to production

Verify: GraphQL mutations return AI_DISABLED error extension.

Recovery — Re-enable: Set flags back to true, commit, deploy.

Redis OOM / Connection Exhaustion

Symptoms: Sidekiq jobs fail, ActionCable disconnects, rate limiting breaks, slow responses.

Diagnosis:

# Check Redis via health endpoint
curl https://api.objectuve.com/health | jq '.checks.redis'

# Check Memorystore metrics in GCP Console
# Production: → Memorystore → enkidu-redis → Monitoring
# Staging:    → Memorystore → enkidu-redis-staging → Monitoring
# Pick the instance matching the affected environment (separate instances per OBJ-468)

Recovery:

1. Check if Sidekiq queues are backed up (large enqueued count in /health)
2. If OOM: Flush non-critical caches via Rails console
3. If connection exhaustion: Check for connection leaks in recent deploys, rollback if needed
4. Scale Memorystore if persistent: GCP Console → Memorystore → Scale

Cloud Run Startup Hang: VPC Egress / Network-Interface Provisioning Contention

Symptoms: A service that was previously healthy starts failing its Cloud Run startup probe repeatedly on new instance starts — health check timeouts, no traffic being served, restart loop.

Diagnosis — read the probe status code before assuming an app/Redis stall:

Cloud Run's own probe-failure status code discriminates between two different failure shapes without needing to touch the container at all:

• DEADLINE_EXCEEDED, zero container output for the full probe budget → the container never got a chance to start. This is a pre-container sandbox/network-interface provisioning hang — Cloud Run is stuck attaching the instance to its configured VPC network interface before your app's boot log ever prints a line. Not a Redis stall, not app code: a genuine app or Redis hang would still show some container output (at minimum the framework's startup banner) before failing.
• CANCELLED → the container started and then died early. This points at the app/dependency layer (app boot crash, Redis/DB connection failure once boot resumed) — investigate there, not the network layer.

# Pull recent revision + probe status
gcloud run revisions list --service=<service-name> --region=us-central1 --limit=5

# Check container logs for the affected revision — zero output = pre-container hang
gcloud logging read 'resource.labels.revision_name="<revision-name>"' --project=enkidu-488723 --limit=50

Confirmed root cause (2026-09-01 enkidu-litellm-staging outage, OBJ-3158, OBJ-3166, PR #2780): Direct VPC egress / network-interface provisioning contention on the shared default subnet under concurrent Cloud Run instance-start load. enkidu-litellm-staging logged 25 consecutive DEADLINE_EXCEEDED hangs (zero container output, full 240s probe budget) before the fix; post-fix cold starts were a clean ~40s with the full startup banner present.

Recovery — for a service with no genuine private-IP dependency:

Drop the Direct VPC egress attachment entirely. This is what unblocked enkidu-litellm-staging (PR #2780) — it removed:

• run.googleapis.com/network-interfaces (network interface annotation)
• run.googleapis.com/vpc-access-egress: private-ranges-only
• its REDIS_URL env var, which staging LiteLLM never actually read (infra/litellm/config.staging.yaml has no cache/redis block)

This fix does not transfer to a service with a real private-IP dependency. enkidu-api-staging, enkidu-worker-staging, and enkidu-crono-staging all stay on the identical VPC egress path (deploy/service.staging.yaml, deploy/worker.staging.yaml, deploy/crono.staging.yaml) because they have a genuine, load-bearing REDIS_URL connection to enkidu-redis-staging's private IP (10.79.34.43) backing Sidekiq and the Rails cache — see Memorystore Redis. Dropping VPC egress on any of those three the way the litellm fix did would cause a real Redis-connectivity outage, not a no-op. If one of them hits this same fingerprint, the fix has to take a different shape: confirm the actual subnet ceiling, then either move it onto a dedicated Serverless VPC Access connector or carve out a dedicated subnet — don't copy the litellm PR as a template. enkidu-api-staging's maxScale is capped (currently 4) as an interim mitigation against reproducing the same thundering-herd pattern, since it was the one remaining unbounded service on this subnet; enkidu-worker-staging and enkidu-crono-staging are pinned to a maxScale of 1 (singleton — minScale on both is schedule-derived as of OBJ-3726 / v4.54 Phase 3a, no longer a constant), which limits but does not eliminate exposure — a single restart during the 2026-09-01 incident still hit the identical hang on both.

Database Connection Exhaustion

Symptoms: 500 errors, health check shows database failed, "PG::ConnectionBad" in Sentry.

Diagnosis:

curl https://api.objectuve.com/health | jq '.checks.database'
# Returns { status, active_connections, max_connections } (rails_api/app/controllers/health_controller.rb)

# Check Cloud SQL metrics — pick the instance matching the affected environment
# (separate instances per OBJ-3416, same isolation pattern as Memorystore's OBJ-468)
# Production: GCP Console → Cloud SQL → enkidu-postgres → Monitoring → Active connections
# Staging:    GCP Console → Cloud SQL → enkidu-postgres-staging → Monitoring → Active connections

Recovery:

1. Check if a recent deploy introduced connection leaks
2. Compare /health's active_connections against max_connections (25 on both instances' db-f1-micro tier) — a brief spike at deploy time (old revision draining while the new one warms its pool) is expected and self-resolves; a sustained high ratio at rest is not
3. Restart Cloud Run service (forces new connections):

   gcloud run services update enkidu-api-production \
     --region=us-central1 --no-traffic
   gcloud run services update-traffic enkidu-api-production \
     --region=us-central1 --to-latest

4. If persistent: increase the connection pool in rails_api/config/database.yml and redeploy — or, for a systemic ceiling problem, a Cloud SQL tier bump. See Cloud SQL for the OBJ-3416 staging-split precedent, which removed cross-environment contention without a tier change.

Agent Runner Failures

Symptoms: AI employee runs stuck in "running" status, no artifacts appearing in review queue.

Diagnosis:

# Check Agent Runner health
curl https://enkidu-agent-runner-production-HASH.run.app/health

# Check for stale runs (stuck > 15 min)
# Rails console:
AiRun.where(status: 'running').where('started_at < ?', 15.minutes.ago).count

Recovery:

1. TimeoutStaleRunsJob automatically marks runs stuck >15 min as failed (runs every 15 min via Crono)
2. Manual cleanup if needed:

   AiRun.where(status: 'running').where('started_at < ?', 15.minutes.ago).update_all(status: 'failed')

3. Check webhook delivery: Look for failed POST /webhooks/ai-workforce in Agent Runner logs
4. If Agent Runner is down: Redeploy from last known-good image

Sidekiq Queue Backup

Symptoms: Emails not sending, notifications delayed, AI jobs queued but not processing.

Diagnosis:

curl https://api.objectuve.com/health | jq '.checks.sidekiq'
# Check: enqueued count, failed count, queue sizes

Recovery:

1. If Redis is healthy but Sidekiq is down: Check Cloud Run instance count (may have scaled to 0)
2. If queue is backed up: Check for failing jobs that are retrying indefinitely
3. Clear dead jobs if needed (Rails console):

   Sidekiq::DeadSet.new.clear  # Nuclear option — clears all dead jobs

Load Test Regression: Connection-Pool Cliff / Sidekiq Burst

Symptoms: The monthly k6 load test tracking-issue comment (enkidu#1754) shows a ⚠️ regression row in its baseline-comparison table — p95 rose more than 20% month-over-month for one or more operations. This is a staging-only, non-blocking signal (it doesn't fail the workflow or gate a deploy), but a real p95 cliff on staging today is a leading indicator for production tomorrow — treat it as an investigation prompt, not noise.

Diagnosis — read the signal before touching anything:

1. Open the flagged comparison row and identify which scenario regressed:
   • l2:* (communityFeed / goals queries) → points at the DB connection pool ceiling or an N+1 query regression. Go to Database Connection Exhaustion above.
   • l1:* (addGoalEvent mutation) → points at Puma thread pool saturation or a Sidekiq enqueue burst (every L1 request enqueues a streak/gamification job). Go to Sidekiq Queue Backup above.
   • l3:* (syncUser) → sign-in throughput; check Clerk JWT verification latency (Sentry → ClerkJwtVerifier) before assuming a DB or Sidekiq cause.
   • l4:* (AI Workforce webhook) → background job creation path; treat like an l1 Sidekiq-burst signal.
2. Open the linked Actions run for the full k6 output and confirm the regression isn't a one-off flake (staging under unrelated smoke-suite load at the same time — see the smoke-staging-demo concurrency group in load-test.yml) before escalating.
3. Cross-reference the run's timestamp against Cloud SQL active connections and Sidekiq queue/latency (GCP Console → Cloud SQL → enkidu-postgres-staging → Monitoring; /health's sidekiq block) for the same window — a load test only reproduces the shape of a spike, the real ceiling is whatever those metrics show at the moment p95 broke. Note: as of OBJ-3416, staging runs on its own dedicated enkidu-postgres-staging instance — it no longer shares enkidu-postgres with production, previews, and jobs, so this signal now reflects staging load in isolation rather than total cross-environment load. See docs/operations/deployment.md for the current two-instance layout.

First mitigations:

• Connection-pool cliff (L2): confirm includes()/preload() is used on any association touched by a recently-merged communityFeed/goals resolver change (see the N+1 Prevention pattern in CLAUDE.md); check rails_api/config/database.yml's pool size against Cloud SQL's max_connections for the affected environment.
• Sidekiq burst (L1/L4): check for a recently-added or newly-slow job on the gamification/streak path (app/jobs/); confirm Cloud Run hasn't scaled the Sidekiq worker down to 0 between runs (cold-start latency shows up as a first-request p95 spike, not a sustained regression).
• Neither path is a production emergency on its own — the signal is staging-only and advisory. Escalate to a real incident (Tier 1/2 rollback above) only if the same symptom is independently confirmed against a production Sentry/Cloud Logging window, not from the load-test comment alone.
• See Baseline storage + regression detection for how the comparison itself is computed, and load_tests/README.md for the scenario scripts and their thresholds.

Budget Alerts

AI spending is monitored via AiBudgetAlertJob (runs hourly via Crono).

Threshold	Action
80% of monthly cap	Slack alert (warning)
100% of monthly cap	Slack alert (critical) + LiteLLM blocks requests

Monthly budget: $500/month production, $50/month staging.

Per-employee caps: $30/month default, enforced at both Agent Runner (per-run) and LiteLLM (monthly) levels.

Rate Limits

Category	Limit	Scope
Coaching	10/min	Per user
Milestones	5/min	Per user
Moderation	Unlimited	System
Batch (insights, check-ins)	Unlimited	System

Rate limit errors return AI_RATE_LIMITED GraphQL error extension.

Post-Incident

After resolving any incident:

1. Check Sentry for lingering errors
2. Verify /health returns all-green
3. Run smoke tests: cd ionic_frontend && npm run smoke:playwright:production
4. Note what happened and what fixed it (for this document)

---

Last updated: 2026-09-14