Disaster Recovery Plan

Product: Objectuve (Enkidu) Owner: Josh Lockhart (solo operator) Status: Active — first issue May 2026 Last reviewed: 2026-07-11

1. Objectives and Scope

This plan defines how to recover Objectuve after a partial or total service failure. It is scoped to the production environment: enkidu-api-production (Rails API on Cloud Run), the enkidu-postgres Cloud SQL instance (PostgreSQL 15, objectuve_production database), the enkidu-redis Memorystore instance, and the enkidu-app Firebase frontend.

Staging recovery is addressed only where it directly affects the production release pipeline.

Recovery Targets

Tier	Service	RTO	RPO	Notes
T0 — Critical	Rails API (`enkidu-api-production`)	15 min	N/A (stateless)	Emergency rollback restores prior revision in seconds; full redeploy < 15 min
T0 — Critical	Auth (Clerk)	Vendor-controlled	N/A	See §6.1; fallback auth available for admin access only
T1 — Important	Database (Cloud SQL)	4 hours	24 hours	GCP automated backups, point-in-time recovery
T1 — Important	Frontend (Firebase)	15 min	N/A (static)	Rebuilt from any tagged commit; fallback URL always available
T2 — Background	Redis (Memorystore)	1 hour	N/A (ephemeral)	Sidekiq retries; ActionCable reconnects; no user data stored
T2 — Background	AI features (LiteLLM)	4 hours	N/A	Kill-switch disables gracefully; non-blocking to core usage
T3 — Low	Agent Runner	8 hours	N/A	AI employee runs queue; no user-blocking path

2. System Map

Users (web/iOS/Android)
  │
  ├── Firebase Hosting (enkidu-app)      → app.objectuve.com
  │     └── Static SPA (Vue 3 / Ionic)
  │
  └── Cloud Run (enkidu-api-production)  → api.objectuve.com
        ├── Cloud SQL Auth Proxy → enkidu-postgres (PostgreSQL 15)
        │     └── objectuve_production database
        ├── Memorystore Redis 7.0 (enkidu-redis, 1GB Basic)
        │     ├── Sidekiq job queues
        │     └── ActionCable pub/sub
        ├── Cloud Run Sidekiq workers
        ├── Cloud Run Cron jobs
        └── External dependencies:
              ├── Clerk (auth — RS256 JWT)
              ├── LiteLLM → Anthropic / OpenAI / Gemini
              ├── GCS bucket (enkidu-storage, media uploads — production only; staging uses its own enkidu-storage-staging bucket, OBJ-1341)
              ├── Mailtrap (transactional email)
              ├── Sentry (error tracking)
              └── FCM / APNs (push notifications)

Agent Runner (enkidu-agent-runner-production) — separate Cloud Run
  └── Communicates with Rails API via HMAC webhook

GCP project: enkidu-488723 · Region: us-central1

3. Failure Scenarios and Response Matrix

Scenario	Blast Radius	Primary Response	Recovery Time
Bad API deploy	All API users	Emergency rollback (§5.1)	< 2 min
Cloud Run OOM / crash loop	All API users	Emergency rollback	< 5 min
Database connection exhaustion	All API users	Restart CR service; pool tuning	< 10 min
Accidental database truncation	All user data	Point-in-time restore (§5.3)	2–4 hours
Cloud SQL instance failure	All API users	GCP-managed failover; then PITR if needed	15–60 min
Redis OOM	Background jobs + WS	Flush non-critical keys; scale if needed (§5.4)	15–30 min
Clerk outage	All sign-ins	Fallback auth for admin only; wait for vendor (§6.1)	Vendor SLA
Firebase outage	Web users	Fallback URL (enkidu-app.web.app); no fix needed	Vendor SLA
GCS failure	Media uploads	Graceful degradation (missing images); wait	Vendor SLA
LiteLLM / AI provider outage	AI features only	Kill-switch in settings/production.yml (§6.3)	< 5 min
Secrets leak / credential rotation	Depends on secret	Rotate in Secret Manager; redeploy all services (§5.6)	30–60 min
CI/CD pipeline broken	Release pipeline	Manual deploy via gcloud CLI (§5.5)	30–60 min
GCP region outage	Everything	No automated failover; user communication; wait or evacuate (§7)	Hours–days

4. Data Backup Strategy

4.1 PostgreSQL (Cloud SQL)

Cloud SQL automated backups and point-in-time recovery are enabled on the enkidu-postgres instance. Settings below were confirmed via gcloud sql instances describe on 2026-05-21 (OBJ-467).

Setting	Confirmed value	How to verify
Automated daily backups	Enabled	GCP Console → Cloud SQL → enkidu-postgres → Backups
Backup window	02:00 UTC (off-peak)	Same path
Backup retention	7 backups	Same path
Point-in-time recovery (PITR)	Enabled	Same path — archives transaction logs to Cloud Storage
PITR transaction-log retention	7 days	Same path

Verified 2026-05-21 (OBJ-467): Automated backups and PITR were found disabled with zero backups on record, and were enabled the same day. An on-demand backup was taken before remediation. A restore was then drill-tested: the backup was cloned to a throwaway instance (gcloud sql instances clone), which reached RUNNABLE in ~28 minutes with the full objectuve_production database intact (71 tables, 120 migrations, core tables populated). The throwaway instance was deleted after verification.

4.2 What Is Not Backed Up Automatically

Data	Location	Recovery
GCS media files (user avatars, goal images)	`enkidu-storage` bucket (production; staging's `enkidu-storage-staging` is out of scope for this plan per §1)	GCS versioning or manual export; no automated DR configured
Redis state	Memorystore Basic tier	Not persisted. All Sidekiq jobs will be lost on failure; ActionCable subscriptions reconnect. Sidekiq retry logic handles job loss for most queues.
Firebase-hosted frontend builds	Firebase Hosting	Rebuilt from any git tag; Artifact Registry retains the corresponding Docker image
Clerk user data	Clerk platform	Managed by Clerk; see their DPA for data retention

4.3 Manual Database Export (off-cluster cold backup)

DB dumps go to gs://enkidu-backups — a private bucket, distinct from enkidu-storage. enkidu-storage is intentionally world-readable and world-listable (allUsers:objectViewer, granted in gcp_setup/02_create_resources.sh) because rails_api/config/storage.yml sets public: true to generate unsigned storage.googleapis.com URLs for user avatars and goal images — that grant is correct and load-bearing, not a misconfiguration to "fix." It is exactly why a database dump must never be exported there: never export a database dump to enkidu-storage.

gs://enkidu-backups is provisioned by gcp_setup/07_create_backups_bucket.sh (run once, idempotent): uniform bucket-level access + --public-access-prevention (so an accidental allUsers grant on this bucket is structurally impossible), a 90-day object-deletion lifecycle rule, and roles/storage.objectAdmin granted to the Cloud SQL instance's own service account — required because both gcloud sql export and gcloud sql import run as that SA.

Bucket name caveat: GCS bucket names are a global namespace. The script defaults to enkidu-backups; if that name is already taken, it falls back to enkidu-enkidu-488723-backups. Confirm which name is actually live before relying on the commands below: gcloud storage buckets describe gs://enkidu-backups --project=enkidu-488723. As of this writing the script has not yet been run against real infra — no credentials were available in the agent session that authored it (OBJ-1345). Run it once and confirm the live bucket name before the export commands below are used for real.

Run before major schema changes or data migrations:

bash

# Identify the Cloud SQL instance connection name
gcloud sql instances describe enkidu-postgres --format="value(connectionName)"
# → enkidu-488723:us-central1:enkidu-postgres

# Export to GCS (adjust BUCKET and TIMESTAMP)
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
gcloud sql export sql enkidu-postgres \
  gs://enkidu-backups/objectuve_production-${TIMESTAMP}.sql.gz \
  --database=objectuve_production \
  --project=enkidu-488723

# Verify the export file exists
gsutil ls gs://enkidu-backups/

Store at least one recent export outside GCS (local download or separate bucket) so a GCS incident does not eliminate both the production database and the backup.

5. Recovery Procedures

5.1 Emergency API Rollback (Tier 0, < 2 min)

Instantly routes 100% of Cloud Run traffic to the previous revision. Does not redeploy; does not run migrations.

bash

# Local CLI shortcut (if configured)
rollback --emergency

# Or directly via gcloud:
gcloud run revisions list \
  --service=enkidu-api-production \
  --region=us-central1 \
  --limit=5

gcloud run services update-traffic enkidu-api-production \
  --region=us-central1 \
  --to-revisions=<PREVIOUS_REVISION>=100

Verify: curl https://api.objectuve.com/health returns all-green.

Limitation: Does not roll back frontend. If the bad deploy included a frontend change, also run a Tier 2 rollback (§5.2).

5.2 Planned Full-Stack Rollback (Tier 1, 5–15 min)

Re-deploys a specific prior version (Docker image already exists in Artifact Registry — no rebuild).

bash

rollback v3.X.Y          # fish CLI shortcut
# Or via GitHub Actions UI: trigger rollback.yml with version input

Note on database migrations: Rollbacks run db:migrate forward (idempotent). They do NOT undo applied migrations. If the broken deploy added a migration that removed a column the old code needs, you must write a new forward migration and ship it as a new release instead of rolling back.

5.3 Database Point-in-Time Recovery (< 4 hours)

Use when data has been accidentally deleted or corrupted and the damage is discovered within 7 days.

DANGER

PITR replaces the instance's data. Confirm the target timestamp carefully. Do this on staging first if the cause is unclear.

Identify the recovery timestamp. Check GCP Cloud Logging for when the bad mutation occurred:

gcloud logging read 'jsonPayload.graphql_operation="DeleteGoal"' \
  --project=enkidu-488723 \
  --freshness=2d --limit=50

Stop all writes to the database. Scale Cloud Run to 0 instances to prevent new writes during restore:

bash

gcloud run services update enkidu-api-production \
  --min-instances=0 --max-instances=0 --region=us-central1
gcloud run services update enkidu-sidekiq-production \
  --min-instances=0 --max-instances=0 --region=us-central1

Initiate PITR from GCP Console:
- Cloud SQL → enkidu-postgres → Backups → Restore
- Select "Restore to a point in time"
- Set timestamp to ~1 minute before the incident
- Create a new Cloud SQL instance (do not overwrite the running instance until verified)
Verify the restored instance. Connect via Cloud SQL Auth Proxy or gcloud sql connect and confirm data integrity.
Swap to the restored instance. Update DATABASE_HOST / DATABASE_NAME in Cloud Run environment (via GCP Console or CI), then scale instances back up.

Restore from GCS export (if PITR is unavailable): Import the most recent .sql.gz export from gs://enkidu-backups (see §4.3 for the bucket-name caveat):

bash

gcloud sql import sql enkidu-postgres \
  gs://enkidu-backups/<EXPORT_FILE>.sql.gz \
  --database=objectuve_production \
  --project=enkidu-488723

5.4 Redis Recovery

Redis (Memorystore Basic tier) is ephemeral — it holds job queues and pub/sub state, not user data. On failure or OOM:

Check health:

bash

curl https://api.objectuve.com/health | jq '.checks.redis'

If OOM — flush non-critical caches (Rails console):

ruby

# Clear ActionCable presence/channel keys (safe to drop)
Redis.new(url: ENV['REDIS_URL']).keys("action_cable:*").each { |k| Redis.new.del(k) }
# Clear Rails cache (regenerates on demand)
Rails.cache.clear

If instance is unreachable — check Memorystore status:

bash

gcloud redis instances describe enkidu-redis \
  --region=us-central1 --project=enkidu-488723

Scale Memorystore if persistent OOM: GCP Console → Memorystore → enkidu-redis → Edit → increase memory.
Sidekiq job recovery: After Redis recovers, Sidekiq reconnects automatically. Jobs that were in-flight when Redis went down will be retried up to Sidekiq's retry limit (25 attempts). Dead jobs can be reviewed in Sidekiq UI or Rails console:
ruby
```
Sidekiq::DeadSet.new.size         # count
Sidekiq::RetrySet.new.each { |j| puts j.item }  # inspect retrying jobs
```

Resolved (OBJ-468, 2026-05-21): Staging now uses a dedicated enkidu-redis-staging Memorystore instance (10.79.34.43, Basic 1GB), isolated at the instance level from production's enkidu-redis (10.234.96.43). A staging job storm or misconfigured Sidekiq concurrency on staging can no longer degrade production Redis.

5.5 Manual Production Deploy (CI/CD Pipeline Down)

If GitHub Actions is unavailable, deploy directly via gcloud:

Backend:

bash

# 1. Identify the image SHA you want to deploy
gcloud artifacts docker images list \
  us-central1-docker.pkg.dev/enkidu-488723/enkidu-registry \
  --filter="tags:v3.X.Y" --format="value(IMAGE,DIGEST)"

IMAGE="us-central1-docker.pkg.dev/enkidu-488723/enkidu-registry/rails-api:v3.X.Y"

# 2. Run migrations
gcloud run jobs execute enkidu-db-migrate \
  --region=us-central1 --wait

# 3. Deploy API
gcloud run deploy enkidu-api-production \
  --image=${IMAGE} \
  --region=us-central1 \
  --project=enkidu-488723
  # Add all --set-secrets flags as per production.yml

Frontend:

bash

cd ionic_frontend
VITE_API_URL=https://api.objectuve.com npm run build
firebase deploy --only hosting:enkidu-app --project=enkidu-488723

5.6 Credential Rotation (Secrets Compromise)

If a secret is compromised or suspected leaked:

Identify the exposed secret from the secret name in GCP Secret Manager:
bash
```
gcloud secrets list --project=enkidu-488723
```

Rotate the secret value (create a new version, disable old):

bash

echo "new-secret-value" | gcloud secrets versions add <SECRET_NAME> --data-file=-
gcloud secrets versions disable <OLD_VERSION> --secret=<SECRET_NAME>

Redeploy all affected Cloud Run services so they pick up the new secret version. Every --set-secrets block must reference <SECRET_NAME>:latest (which automatically resolves to the newest enabled version).
bash
```
# Trigger a new deploy for each affected service
# The easiest path is to trigger staging.yml via workflow_dispatch
# then tag a new production release
```
Rotate externally if the secret is a vendor API key (Clerk, Anthropic, etc.) — change the key in the vendor dashboard first, then update Secret Manager.

Review Cloud Audit Logs for unauthorized use during the exposure window:

bash

gcloud logging read \
  'protoPayload.methodName="google.cloud.secretmanager.v1.SecretManagerService.AccessSecretVersion"' \
  --project=enkidu-488723 --freshness=30d --limit=50

6. Third-Party Vendor Outages

6.1 Clerk Outage (Auth Down)

Impact: All sign-ins and token refreshes fail. Existing sessions may work briefly until JWTs expire.

Detection:

bash

curl https://status.clerk.com/api/v2/status.json

Response:

Post status update on any user-facing channel if outage exceeds 15 minutes.
Do NOT enable fallback auth in production unless the outage exceeds 4 hours and critical administrative access is needed.

If admin access is essential during the outage:

bash

gcloud run services update enkidu-api-staging \
  --set-env-vars="FALLBACK_AUTH_SECRET=$(openssl rand -hex 32)" \
  --region=us-central1 --project=enkidu-488723
# Access admin dashboard via /sign-in?fallback

Remove FALLBACK_AUTH_SECRET after Clerk recovers.

6.2 Mailtrap Outage (Email Down)

Impact: Welcome emails, magic-link sign-in codes, and lifecycle emails fail. Core app functionality is unaffected.

Response: Log errors; monitor Mailtrap status. No user-facing degradation flag needed unless sign-in via email magic code is the user's only auth method.

6.3 LiteLLM / AI Provider Outage

Impact: AI Coach, check-in prompts, milestone generation, and content moderation fail. Core goal/habit tracking is unaffected.

Response — disable AI services instantly (no deploy required, requires git push):

Edit rails_api/config/settings/production.yml:

yaml

ai:
  coaching_enabled: false
  moderation_enabled: false
  batch_enabled: false

Commit, push to master, and tag a release:

bash

git add rails_api/config/settings/production.yml
git commit -m "fix: disable AI services (outage)"
release

Re-enable by setting flags back to true after the provider recovers.

6.4 Firebase Hosting Outage (Frontend CDN Down)

Impact: app.objectuve.com unreachable. Mobile app users (iOS/Android via Capacitor) are unaffected — they call the API directly.

Fallback URL: The direct Firebase URL always works regardless of custom domain issues:

Production: https://enkidu-app.web.app
Staging: https://enkidu-app-staging.web.app

Communicate the fallback URL to users if the outage persists.

7. GCP Regional Outage

A full us-central1 outage would take down Cloud Run, Cloud SQL, Memorystore, and Artifact Registry simultaneously. There is no automatic cross-region failover in the current architecture — by design, to preserve free-tier economics.

Response plan:

Timeline	Action
0–30 min	Confirm it is a GCP incident, not a misconfiguration. Check https://status.cloud.google.com
30 min	Post status update; set maintenance mode if possible
30 min – 4 hours	Wait for GCP recovery (most regional incidents resolve within 4 hours)
4+ hours	Evaluate emergency evacuation to `us-east1` (see below)

Emergency evacuation checklist (4+ hour regional outage):

Export database to a local file from the last available GCS backup (or the most recent Cloud SQL export if accessible).
Provision Cloud SQL in us-east1 and restore from export.
Update Cloud Run deployments to target us-east1.
Update DNS CNAME records (Namecheap) to new Cloud Run / Firebase endpoints.
Deploy Agent Runner to us-east1.

INFO

This evacuation path is not tested and would take significant manual effort. For a solo-operator SaaS at early stage, accepting an extended outage during a regional incident is the appropriate trade-off against the operational cost of multi-region infrastructure.

8. Known Gaps and Open Issues

These are structural risks that exist as of this plan's publication date. They are filed as backlog issues for triage.

Gap	Risk	Severity	Issue
GCS media backup not configured	User avatars and goal images unrecoverable after GCS incident	Medium	File Sage backlog issue
~~Staging and production share `enkidu-redis`~~	Resolved 2026-05-21 (OBJ-468): staging moved to a dedicated `enkidu-redis-staging` Memorystore instance; production Redis is now isolated from staging job storms	Resolved	OBJ-468
~~Backup restore never tested~~	Resolved 2026-05-21 (OBJ-467): backups + PITR enabled and a restore drill-tested via clone to a throwaway instance (~28 min to `RUNNABLE`)	Resolved	OBJ-467 — recurring coverage via §9 Q2 drill
~~`scripts/validate-secrets.sh` not implemented~~	Partially resolved (OBJ-452): cross-block `--set-secrets` consistency + Secret Manager existence checks run pre-deploy on both workflows. A known blind spot remains — `collect_rails_env_refs()` (would enforce "every Rails `ENV` reference is wired into at least one block somewhere") is defined but never called from `main()`, so a var wired into zero `--set-secrets` blocks passes the lint silently. This is exactly how `STRIPE_PRICE_TEAMS_*` shipped unwired to production undetected (OBJ-1339) — closed for that specific case via a `WIRING_ONLY` exemption + a dedicated Cloud Run job rather than by fixing the general gap (closing it fully would require auditing/wiring or allowlisting every Rails `ENV` reference in the app, not just Stripe's). See Deployment § Pre-deploy secrets lint.	Medium	OBJ-452 (resolved), zero-presence gap filed separately by Orion (not GA-blocking)
No DR drill cadence	Recovery procedures untested; actual RTOs unknown	Medium	Schedule quarterly drill
GCS cold backup outside GCS	If GCS bucket is deleted, backups stored in same bucket are gone too	Medium	Export monthly to separate bucket

9. DR Drill Procedure

Run at least once per quarter (target: first Monday of each quarter).

Scope per drill (rotate coverage):

Quarter	Drill Scenario
Q1	Emergency API rollback — deploy a canary release, roll it back, verify health
Q2	Database restore — restore from a recent Cloud SQL backup to a throwaway staging instance; verify row counts
Q3	Secrets rotation — rotate one non-critical secret, verify all services pick it up
Q4	Full manual deploy — disable GitHub Actions, deploy to staging using only `gcloud` + `firebase` CLI

Post-drill: Update RTO column in §1 with observed timings. Update this document with any procedure corrections.

10. Communication Plan

As a solo-operator, the "communication plan" is primarily about user-facing messaging.

Duration	Action
< 15 min	No communication needed for most incidents
15–60 min	Post incident note on status page (if one exists) or via app notification
60+ min	Proactive email or in-app notice; estimate ETR
Data loss	Immediate notice to affected users; describe scope; EU-level incidents may trigger GDPR notification obligation within 72 hours (see Termly Terms of Service)

11. References

Incident Response Runbook — per-failure-mode diagnosis and recovery steps
Deployment — rollback CLI commands, CI/CD pipeline, Cloud Run secrets
Observability — health check, Sentry, GCP Cloud Logging
Alerting — GCP monitoring policies, Sentry alert rules
Vendors — vendor contact and status page links
Architecture System Overview — full system diagram
Related incidents: OBJ-348 (batch tag push silent drops), OBJ-452 (missing Cloud Run secrets), OBJ-73 (staging build race)

Author: Sage (System Architect agent) · Reviewed by: Josh Lockhart · Next review: 2026-08-01

Last updated: 2026-07-11

Coach Personas

Disaster Recovery Plan

1. Objectives and Scope

Recovery Targets

2. System Map

3. Failure Scenarios and Response Matrix

4. Data Backup Strategy

4.1 PostgreSQL (Cloud SQL)

4.2 What Is Not Backed Up Automatically

4.3 Manual Database Export (off-cluster cold backup)

5. Recovery Procedures

5.1 Emergency API Rollback (Tier 0, < 2 min)

5.2 Planned Full-Stack Rollback (Tier 1, 5–15 min)

5.3 Database Point-in-Time Recovery (< 4 hours)

5.4 Redis Recovery

5.5 Manual Production Deploy (CI/CD Pipeline Down)

5.6 Credential Rotation (Secrets Compromise)

6. Third-Party Vendor Outages

6.1 Clerk Outage (Auth Down)

6.2 Mailtrap Outage (Email Down)

6.3 LiteLLM / AI Provider Outage

6.4 Firebase Hosting Outage (Frontend CDN Down)

7. GCP Regional Outage

8. Known Gaps and Open Issues

9. DR Drill Procedure

10. Communication Plan

11. References

Objectuve Docs

Disaster Recovery Plan ​

1. Objectives and Scope ​

Recovery Targets ​

2. System Map ​

3. Failure Scenarios and Response Matrix ​

4. Data Backup Strategy ​

4.1 PostgreSQL (Cloud SQL) ​

4.2 What Is Not Backed Up Automatically ​

4.3 Manual Database Export (off-cluster cold backup) ​

5. Recovery Procedures ​

5.1 Emergency API Rollback (Tier 0, < 2 min) ​

5.2 Planned Full-Stack Rollback (Tier 1, 5–15 min) ​

5.3 Database Point-in-Time Recovery (< 4 hours) ​

5.4 Redis Recovery ​

5.5 Manual Production Deploy (CI/CD Pipeline Down) ​

5.6 Credential Rotation (Secrets Compromise) ​

6. Third-Party Vendor Outages ​

6.1 Clerk Outage (Auth Down) ​

6.2 Mailtrap Outage (Email Down) ​

6.3 LiteLLM / AI Provider Outage ​

6.4 Firebase Hosting Outage (Frontend CDN Down) ​

7. GCP Regional Outage ​

8. Known Gaps and Open Issues ​

9. DR Drill Procedure ​

10. Communication Plan ​

11. References ​

Objectuve Docs

Disaster Recovery Plan

1. Objectives and Scope

Recovery Targets

2. System Map

3. Failure Scenarios and Response Matrix

4. Data Backup Strategy

4.1 PostgreSQL (Cloud SQL)

4.2 What Is Not Backed Up Automatically

4.3 Manual Database Export (off-cluster cold backup)

5. Recovery Procedures

5.1 Emergency API Rollback (Tier 0, < 2 min)

5.2 Planned Full-Stack Rollback (Tier 1, 5–15 min)

5.3 Database Point-in-Time Recovery (< 4 hours)

5.4 Redis Recovery

5.5 Manual Production Deploy (CI/CD Pipeline Down)

5.6 Credential Rotation (Secrets Compromise)

6. Third-Party Vendor Outages

6.1 Clerk Outage (Auth Down)

6.2 Mailtrap Outage (Email Down)

6.3 LiteLLM / AI Provider Outage

6.4 Firebase Hosting Outage (Frontend CDN Down)

7. GCP Regional Outage

8. Known Gaps and Open Issues

9. DR Drill Procedure

10. Communication Plan

11. References