Alerting

Objectuve uses two alerting systems: Sentry for application-level alerts and GCP Cloud Monitoring for infrastructure alerts.

Alert Channels

Setup

Configure a notification channel before creating alerts:

Slack:

gcloud monitoring channels create \
  --type=slack \
  --display-name="Objectuve Alerts" \
  --channel-labels=channel_name=#alerts \
  --project=YOUR_PROJECT

Email:

gcloud monitoring channels create \
  --type=email \
  --display-name="Objectuve Alerts Email" \
  --channel-labels=email_address=team@objectuve.com \
  --project=YOUR_PROJECT

Get the channel ID for use in alert policies:

gcloud beta monitoring channels list --project=YOUR_PROJECT

Sentry Alerts

Sentry uses a Monitors & Alerts model: a monitor (detector) defines the condition against project data, then one or more connected alerts attach notification actions (Slack, email, etc.) to it via Create a New Alert. Configure new rules directly in the Sentry dashboard under Monitors & Alerts.

1. Error Rate Spike

• Trigger: Error count > 10 in a 5-minute window
• Action: Notify Slack channel
• Rationale: Catches sudden regressions or downstream failures

2. New Issue Detection

• Trigger: First occurrence of any new issue
• Action: Notify Slack channel
• Rationale: Catches new error types immediately (this is Sentry's default behavior — verify it's enabled)

3. Performance Regression

• Trigger: p95 transaction duration > 2s for any transaction
• Action: Notify Slack channel
• Rationale: Catches slow queries, N+1s, or infrastructure degradation

4. Critical Path — Fallback Pool Triggered (Pipes, retired)

• Trigger: Message contains pipes:fallback-triggered more than 5 times in a 24-hour window
• Action: Notify Slack channel (#alerts)
• Status: Retired 2026-07-23 — the Pipes puzzle engine that emitted this message was deleted in the v4.12 Zip cutover (OBJ-1671, PR #1735); pipes:fallback-triggered can no longer fire. The Sentry alert rule itself is external config and hasn't been touched as part of this doc pass — safe to delete on the next Sentry alert-rule cleanup, or leave as an inert rule. See rule 5 for its replacement.
• Environment: Production only

5. Critical Path — Zip Fallback Pool Triggered

• Trigger: Message contains zip:fallback-triggered more than 5 times in a 24-hour window
• Action: Notify Slack channel (#alerts)
• Rationale: The Zip puzzle engine (ionic_frontend/src/play/zip/) is now the live Critical Path mechanic (v4.12 cutover, OBJ-1671) — a curated fallback puzzle is served when generation exceeds the 500ms budget, mirroring the retired Pipes fallback pattern above. More than 5 fallbacks in 24h suggests generator latency regression or low-end device struggles.
• Environment: Production only
• Verification after merge: In Sentry, navigate to Monitors & Alerts and verify the monitor exists with condition message == 'zip:fallback-triggered' and threshold 5 events in 24h

6. Native OAuth Sign-In Failures

• Trigger: oauth.strategy:[oauth_google, oauth_apple] errors on the native OAuth transaction — static threshold, High: above 0 (i.e. ≥1 error in a 1-hour window), Resolved: ≤ 0
• Action: Notify Slack channel (#alerts) — connected alert "Native OAuth sign-in failures → #alerts", tags shown in the message: oauth.strategy, clerk.native, environment, release, throttle: every trigger
• Rationale: OBJECTUVE-CLIENT-C2 ("Google Sign-In has no client ID configured") sat unresolved in production 2026-08-16–2026-08-20 across 6 occurrences with full diagnostic tags, and was only found because a human tried to sign in and filed OBJ-2772 — client-side capture worked, nobody was paged. This monitor closes that gap for the oauth_google/oauth_apple native strategies emitted by the current useNativeOAuth.ts/useClerkNative.ts capture paths (OBJ-2775, OBJ-2784).
• Environment: Production only
• Monitor: Native OAuth sign-in failures — project objectuve-client, dataset Errors, count()
• Verification after merge: This needs a live-fire check, not just "monitor exists" — the 28-day backtest under the oauth.strategy filter is flat 0, so this monitor would not have caught the OBJ-2772 incident retroactively (those events didn't carry a matching oauth.strategy value). Trigger one native OAuth failure on a production build and confirm a #alerts post within the hour.

7. High-User-Count, Long-Unresolved Client Error

• Trigger: A client error issue affects more than 50 users AND has been unresolved for more than 14 days (starting thresholds — tune after the first month of data against this rule)
• Action: Notify Slack channel (#objectuve-alerts)
• Rationale: This closes the gap that let OBJECTUVE-CLIENT-7V — the cold-navigation teamId race, Sentry's default grouping bucket for every unfingerprinted GraphQL error in the frontend — sit at 287 affected users / 36,396 events for roughly 4 months with no alert ever firing (OBJ-2348). No rule existed for "an error a lot of people are hitting that nobody has looked at."
• Depends on the apollo-client.ts GraphQL error fingerprinting fix (OBJ-2348, PR #2540) having landed first, and it has — scope.setFingerprint([operation.operationName, discriminator]) (ionic_frontend/src/apollo-client.ts:125-131, wired into both the FORBIDDEN and general error-capture branches) now keys Sentry's grouping on the GraphQL operation name plus a stable error discriminator (an extensions.code, falling back to the error path, falling back to a constant), instead of the fixed new Error(...) construction site every GraphQL error previously shared. Why the dependency matters: before fingerprinting, every GraphQL error in the app — successful requests' edge cases included — collapsed into that one perpetually-open bucket; a user-count/age rule pointed at it would only ever match the aggregate, which is noise on no single real defect. With fingerprinting live, each distinct operation/discriminator pair accumulates its own Sentry issue, so a threshold on "one issue, many affected users, still open" is now a signal on one real, specific failure mode rather than an app-wide catch-all.
• Environment: Production only
• Verification after merge: In Sentry, navigate to Alerts > Alert Rules and verify the rule exists with conditions "an issue affects more than 50 users" and "an issue is older than 14 days and still unresolved" (or the closest native equivalents Sentry's alert-rule builder offers — confirm exact condition names at creation time)

HUMAN ACTION — Josh

This rule is Sentry console configuration (Alerts > Create Alert Rule), not repo config — no agent can create it (see the note above ## Sentry Alerts). Josh needs to create this rule directly in the Sentry dashboard using the trigger and conditions above.

Application-Level Alerts (Slack Webhook)

These are hourly Crono-scheduled jobs that page directly via SlackNotifier (rails_api/app/services/slack_notifier.rb), a stateless wrapper around the SLACK_BUDGET_ALERT_WEBHOOK_URL incoming webhook. SlackNotifier never raises — delivery is best-effort — so each job also calls Sentry.capture_message/Sentry.capture_exception for a durable record and re-raises on internal failure so Sidekiq retries. All four currently share this one webhook (additive, not separate integrations per job).

AI Budget Alert

• Job: AiMetrics::AiBudgetAlertJob
• Trigger: month-to-date AI spend crosses a configured threshold (default 80%/100%)
• Dedup: 45 days per threshold
• See AI Operations Runbook for the full budget-alert response procedure.

Stale Payment Record Reconciliation

• Job: Billing::ReconcileStalePaymentRecordsJob
• Trigger: a PaymentRecord stuck pending 2+ hours where Stripe's own Checkout session confirms paid and the session's livemode is true — i.e. a dropped checkout.session.completed webhook (OBJ-2479 gate 2). A paid test-mode session is logged at warn and does not page (OBJ-2852) — see the runbook below for why.
• Dedup: 1 day per record
• See Stale Payment Record Reconciliation Runbook for the full response procedure.

Undelivered AI Artifact Reconciliation

• Job: AiWorkforce::ReconcileUndeliveredArtifactsJob
• Trigger: an approved AiArtifact still delivered_at: nil 1+ hour after review — i.e. a dropped ai_workforce.artifact_approved dispatch or a stuck DeliverArtifactJob (OBJ-2479 gate 2)
• Dedup: 1 day per artifact

GDPR Overdue Request Alert

• Job: Gdpr::OverdueRequestAlertJob
• Trigger: a GdprRequest past its 72-hour EU GDPR response deadline (GdprRequest.overdue) — i.e. a request stuck received, processing, or failed past due_by, previously visible only via a client-side admin-dashboard badge that the sidebar's PendingCountsService#gdpr_count doesn't even cover for processing/failed requests (OBJ-3674 should-fix)
• Dedup: 1 day per request

Email Delivery Canary

• Jobs: Email::DeliveryCanarySendJob (Crono, daily 07:00, inline in the Crono container) sends a synthetic canary email and alerts if the send itself fails; Email::DeliveryCanaryVerifyJob (Sidekiq, worker container, enqueued 10 minutes after send, self-re-enqueues on non-terminal Mailtrap states until a 30-minute deadline) resolves the delivery verdict and alerts. Full pipeline: docs/architecture/email.md § Delivery Canary.

• Three alert kinds — deliberately not collapsed into one:

  • Delivery failure — the canary genuinely did not reach the inbox (Mailtrap reports bounced, or the message never reached a terminal status before the 30-minute deadline). This is the incident.
  • Cannot verify — Objectuve could not determine what happened (couldn't resolve Mailtrap's sending_message_id for the token, Mailtrap returned an unrecognized status, or Mailtrap was unreachable through the deadline). This is an infra blip, not proof email delivery is broken.
  • Send failure (OBJ-2909) — DeliveryCanarySendJob#perform itself raised (e.g. .deliver_now fails) before the job could even enqueue DeliveryCanaryVerifyJob, so the Verify job's own alerting is never reached — a Phase 2 (Send) failure, not a Phase 3 (Verify) one. Before this alert existed, a Send-stage failure had no dedicated paging path at all — only generic Sentry capture, plus /health's fail-open, weekday-only-checked email_canary staleness field (see below).

  Collapsing these into one alert type is what trains a team to ignore its own alerts: a run of "cannot verify" noise from a transient Mailtrap API hiccup would read identically to a real delivery outage, and after enough false pages the team stops trusting any of them. build_alert_body (delivery_canary_verify_job.rb) keeps the first two visually distinct in Slack — :rotating_light: EMAIL CANARY - DELIVERY FAILURE vs. :warning: EMAIL CANARY - CANNOT VERIFY — and DeliveryCanarySendJob#alert_send_failure (delivery_canary_send_job.rb) posts its own :x: EMAIL CANARY - SEND FAILURE prefix, so delivery-failure and send-failure alerts should be treated as paging an incident; cannot-verify should not.

• Dedup: none on a time window — a fresh token each day means at most one delivery-failure or cannot-verify alert per calendar day per branch, bounded by the Verify job's own re-enqueue-until-30-minutes logic rather than a dedup cache key like the three jobs above. Send-failure has no re-enqueue logic of its own — it's naturally bounded to once per calendar day by DeliveryCanarySendJob's daily 07:00 Crono cadence.

• Re-proving the alert path: bundle exec rake email_canary:live_fire (rails_api/lib/tasks/email_canary.rake) runs DeliveryCanaryVerifyJob.perform_now against a token with no cache record, forcing the missing-record → cannot-verify branch end to end and prefixing the Slack message [LIVE-FIRE TEST]. This is a documented, repeatable way to re-prove the alerting path itself still works — not a one-time ceremony.

• /health surfaces the canary's own liveness — HealthController#check_email_canary (informational, fail-open: a stale canary never 503s /health) reports three states: ok (a confirmed :delivered within EMAIL_CANARY_STALE_THRESHOLD_HOURS, default 26), not_configured (EMAIL_CANARY_RECIPIENT unset), and stale (no confirmed delivery within the threshold). stale is the failure mode OBJ-2820 exists to surface: previously, "no alert fired" meant both "everything is fine" and "the canary silently stopped running," indistinguishably. A stale reading alongside daily cannot-verify alerts specifically points at the canary's Mailtrap message-id lookup resolving incorrectly, not at email delivery itself being broken.

• Sentry fingerprinting (OBJ-2971). send_alert (delivery_canary_verify_job.rb) and alert_send_failure (delivery_canary_send_job.rb) pass an explicit fingerprint: ['email-canary', kind.to_s, environment.to_s] to Sentry.capture_message, plus level: :error for delivery_failure/send_failure and :warning for cannot_verify. Before this, Sentry's default message-based grouping folded all four kind×environment combinations into one issue (OBJECTUVE-API-2W), so a production delivery failure and a staging cannot-verify were indistinguishable in Sentry. token/sent_at stay out of the fingerprint (they'd otherwise turn one issue per kind into one per day) and live in extra: instead, where they stay queryable.

• Known gap, recorded honestly: these alerts currently post through SLACK_BUDGET_ALERT_WEBHOOK_URL via SlackNotifier — the same webhook the AI Budget Alert above uses — so they appear in Slack under that job's identity, not a dedicated "Email Canary" one. (The Sentry side of the alert is no longer a gap — objectuve/objectuve-api's Sentry project has been ingesting real events since OBJ-2821's blank-SENTRY_DSN guard shipped 2026-08-22, confirmed live on OBJ-2971 by the very OBJECTUVE-API-2W fold this fingerprint fix splits apart.)

• Fail-closed watchdog (OBJ-2970). email_canary sits in HealthController::INFORMATIONAL_CHECKS alongside crono and is fail-open at the HTTP layer by design — a stale canary never 503s /health. Dave — Email canary liveness check is its dedicated fail-closed layer (mirrors Dave — Crono liveness check): a daily autopilot that reads .checks.email_canary.status on both environments — ok → pass, stale/error → fail, not_configured or an absent key → blocked, not fail (an unconfigured recipient or an undeployed check is not a delivery outage) — re-checks once after 60s before acting on a fail, and dedupes against any open "Email canary liveness" issue rather than filing a duplicate. Diagnostic only; it never attempts a fix. Full runbook: docs/guides/multica-autopilots.md.

GCP Cloud Monitoring Alerts

1. Uptime Check (API Health)

First, create the uptime check:

gcloud monitoring uptime create \
  --display-name="API Health Check" \
  --resource-type=uptime-url \
  --hostname=api.objectuve.com \
  --path=/health \
  --protocol=HTTPS \
  --check-frequency=60s \
  --project=YOUR_PROJECT

Then create an alert policy for uptime check failure:

gcloud monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="API Health Check Failed" \
  --condition-display-name="Health check failing" \
  --condition-filter='resource.type="uptime_url" AND metric.type="monitoring.googleapis.com/uptime_check/check_passed"' \
  --if="< 1" \
  --duration=300s \
  --combiner=OR \
  --project=YOUR_PROJECT

2. Cloud Run Latency

gcloud monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="API Latency p95 > 3s" \
  --condition-display-name="High request latency" \
  --condition-filter='resource.type="cloud_run_revision" AND metric.type="run.googleapis.com/request_latencies"' \
  --if="> 3000" \
  --duration=300s \
  --aggregation='{"alignmentPeriod":"60s","perSeriesAligner":"ALIGN_PERCENTILE_95"}' \
  --combiner=OR \
  --project=YOUR_PROJECT

3. Cloud Run Error Rate

gcloud monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="API Error Rate > 5%" \
  --condition-display-name="High error rate" \
  --condition-filter='resource.type="cloud_run_revision" AND metric.type="run.googleapis.com/request_count" AND metric.labels.response_code_class!="2xx"' \
  --if="> 0.05" \
  --duration=300s \
  --combiner=OR \
  --project=YOUR_PROJECT

4. Cloud SQL Connection Count

gcloud monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="Cloud SQL Connections > 80%" \
  --condition-display-name="High connection count" \
  --condition-filter='resource.type="cloudsql_database" AND metric.type="cloudsql.googleapis.com/database/postgresql/num_backends"' \
  --if="> 80" \
  --duration=300s \
  --combiner=OR \
  --project=YOUR_PROJECT

5. Cloud SQL CPU

gcloud monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="Cloud SQL CPU > 80%" \
  --condition-display-name="High CPU utilization" \
  --condition-filter='resource.type="cloudsql_database" AND metric.type="cloudsql.googleapis.com/database/cpu/utilization"' \
  --if="> 0.8" \
  --duration=300s \
  --combiner=OR \
  --project=YOUR_PROJECT

Why --combiner=OR is here

gcloud monitoring policies create's flag-based builder never sets combiner on its own — omit it and the API rejects the policy with INVALID_ARGUMENT: Field alert_policy.combiner had an invalid value, since a single-condition policy still needs an explicit combiner. OR is the correct, conventional choice for a policy with one condition. See item 6 below for where this exact gap surfaced live.

6. Cloud Run Startup-Probe Failure

• Trigger: rate of Cloud Run startup-probe / revision-readiness failures > 0, sustained 5 minutes (--duration=300s)

• What it detects: Cloud Run system-log entries matching STARTUP TCP probe failed or STARTUP HTTP probe failed for any of the four core services (litellm/api/worker/crono) in the given environment — a revision failing to reach Ready, not a single transient probe blip

• Environment: Staging and production, provisioned separately (see below)

• Rationale: the 2026-09-01 17:10–18:27 UTC enkidu-litellm-staging outage (OBJ-3158) surfaced by accident at 17:36 instead of paging at 17:12 — no alert existed on Cloud Run startup-probe failure rate for any staging service (OBJ-3166 SS7, OBJ-3167). Production is the larger exposure of the two and had no equivalent alert until OBJ-3365.

• Why it reads Cloud Run system logs instead of /health: the hourly staging-health check is a poll against /health, and depends on the service already being reachable to answer it — a hung startup blinds the health check itself. Cloud Run only starts routing traffic (so /health can only be reached at all) once a revision passes its own startupProbe, which targets /health/live — process liveness only, checks nothing (HealthController#live, rails_api/app/controllers/health_controller.rb) — precisely so a degraded dependency doesn't block a new revision from reaching Ready. This alert reads Cloud Run's own system logs directly instead, so it still fires when the service the health check would poll is the one that's hung.

• Threshold rationale: 300s comfortably clears each service's own probe retry budget (failureThreshold: 3 × periodSeconds: 10s ≈ 30s — deploy/service.production.yaml, deploy/worker.production.yaml, deploy/crono.production.yaml), so a single retry cycle won't page. Confirmed against real log data on both environments: staging's 76-minute outage (max gap between failures 283s) and a shorter ~7-minute ordinary staging failure burst both sit inside the window; production's measured ordinary deploy-time bursts (131s, 176s) also sit inside it and won't page, while the 2026-09-01 incident (74 api + 90 worker events in one day) would have.

• Provisioning:

  • Script: gcp_setup/16_cloud_run_startup_probe_alerts.sh, parameterized by ENVIRONMENT=staging|production (OBJ-3365 — previously staging-only as 11_staging_startup_probe_alerts.sh; renumbered 11_ → 16_ by OBJ-3419's script-numbering collision fix)
  • Staging: an ENVIRONMENT=staging step inside the provision-cloudsql-alert-policy job in .github/workflows/staging.yml, run on every push to master
  • Production: a dedicated provision-startup-probe-alert-policy job in .github/workflows/production.yml, run on every production release-tag push, independent of the Rails/frontend deploy jobs (no needs:)
  • Both jobs authenticate via WIF as github-actions-deployer, which holds logging.logMetrics.create/.list and monitoring.alertPolicies.create/.list (gcp_setup/github-actions-alert-policy-role.yaml, applied by hand via gcp_setup/13_github_actions_alert_policy_role.sh)
  • Log-based metric / policy display name: cloud-run-startup-probe-failures-staging / "Staging Cloud Run startup-probe failure"; cloud-run-startup-probe-failures-production / "Production Cloud Run startup-probe failure"
  • Notification channel is read from the MONITORING_NOTIFICATION_CHANNEL_ID_STAGING/_PRODUCTION secrets — never hardcoded

• OBJ-3638 (2026-09-09 – 2026-09-11): NOTIFICATION_CHANNEL_ID was unset on both jobs from day one. Neither MONITORING_NOTIFICATION_CHANNEL_ID_PRODUCTION nor MONITORING_NOTIFICATION_CHANNEL_ID_STAGING was provisioned as a repo secret when these jobs merged, so NOTIFICATION_CHANNEL_ID resolved to the empty string and the script's own guard (gcp_setup/16_cloud_run_startup_probe_alerts.sh:80-85) hard-failed both jobs — exactly as designed, just six days too late to be caught before it read as release-train noise. Concretely: production's provision-startup-probe-alert-policy job failed on all three v4.8.3 attempts (2026-09-09, runs 34356807906/34357011985/34361161288), and staging's equivalent step inside provision-cloudsql-alert-policy (.github/workflows/staging.yml) failed identically on every push in the same window — that staging failure is what OBJ-3232's ticket was tracking; it was never a separate root cause, just the same unprovisioned-secret gap hitting the staging job first. Both secrets were provisioned 2026-09-09 16:57 UTC (confirmed via gh secret list --repo objectuve-softworks/enkidu).

• OBJ-3695 (2026-09-12): the first production run past the OBJ-3638 secret fix hit a second, different failure — the log-based metric hadn't propagated to the Monitoring API yet. gcloud logging metrics create succeeded, but the immediately-following gcloud monitoring policies create call failed with Cannot find metric(s): ... — confirmed on the "Provision production Cloud Run startup-probe alert policy" step of production run 34715862184 (tag v4.8.4, .github/workflows/production.yml:551). Log-based metrics can take up to ~10 minutes to become resolvable by the Monitoring API's metric-descriptor index — a separate propagation lag from the Logging API, which serves the metric back successfully the instant create returns. gcloud logging metrics describe $METRIC_NAME is not a valid readiness probe for this gap — it answers from the Logging API and returns success immediately, before the Monitoring API has indexed the metric, so polling it would never observe the lag it needs to wait out; this is the single most re-derivable wrong turn on this failure, worth remembering here rather than only in the PR that fixed it. Fix: gcp_setup/16_cloud_run_startup_probe_alerts.sh now retries the gcloud monitoring policies create call itself, bounded to ~10 minutes, matching only the propagation error signature and failing fast on anything else — the same house pattern as retry_binding()/is_propagation_error() in gcp_setup/08_runtime_service_accounts.sh:149-206. Both environments invoke the same script (.github/workflows/staging.yml:915, .github/workflows/production.yml:551), so one fix covers both, though staging's steady-state runs won't exercise the new retry path.

• Status as of 2026-09-12: staging remains CI-confirmed live, unchanged since 2026-09-11 (see run 34631027584 above). Production's notification-channel gap (OBJ-3638) is closed, but production has still never completed a clean run of this job: the first v* push after that fix (v4.8.4, run 34715862184) got past the notification-channel guard and immediately hit the propagation gap documented above (OBJ-3695) instead. That gap is now fixed (bounded retry, above), but production remains unconfirmed — no production release-tag push has run since the propagation fix landed. The next v* tag push is what closes this out; don't read "fix applied" as "provisioned" — check for a clean job run (or gcloud monitoring policies list, permissions allowing) first.

• Guardrail added (OBJ-3638): .github/workflows/required-github-secrets-drift.yml + scripts/check-required-github-secrets.mjs now diff every secrets.NAME reference across .github/workflows/*.yml against the live provisioned-secret list (daily + workflow_dispatch, via RELEASE_TOKEN — reading secret names needs the repo's Secrets permission, which isn't grantable to the default GITHUB_TOKEN) and file/reopen a tracking issue on any gap. This would have caught the OBJ-3638/OBJ-3232 gap the day the job merged instead of six days and three releases later. It also surfaced eight pre-existing, unrelated gaps (ACHIEVEMENTS_EMPTY_EMAIL, ACHIEVEMENTS_POPULATED_EMAIL, CLERK_SECRET_KEY_PROD, CODECOV_TOKEN, ITC_TEAM_ID, RAILS_WEBHOOK_SECRET, SENTRY_ISSUES_API_TOKEN, SLACK_BUDGET_ALERT_WEBHOOK_URL) as a side effect of being built — flagged for separate triage, not fixed here.

  Verify staging live (copy-pasteable, run against enkidu-488723):

  gcloud logging metrics describe cloud-run-startup-probe-failures-staging --project=enkidu-488723
  gcloud monitoring policies list --project=enkidu-488723 --filter='displayName="Staging Cloud Run startup-probe failure"'

  The logging metrics describe call needs only roles/logging.viewer-equivalent read access and is safe for any crew agent to run. The monitoring policies list call needs roles/monitoring.viewer (or broader) — the crew's dave-autopilot service account does not hold it (Permission denied, confirmed 2026-09-08 and reconfirmed 2026-09-11), so that half of verification needs a human or a differently-scoped credential; CI's own provisioning-step logs (see above) are the crew-accessible substitute for the staging half.

  Once production deploys, the equivalent pair is cloud-run-startup-probe-failures-production / displayName="Production Cloud Run startup-probe failure".

  Fixed (OBJ-3167 follow-up, superseded by OBJ-3539): the first live application of this policy hit exactly the INVALID_ARGUMENT: ... combiner had an invalid value failure described in the tip under items 1–5 above, back when the script's gcloud monitoring policies create call was still flag-based with no --combiner=OR — staging's policy existed only because that flag was added ad hoc for the manual run. That flag-based fix (#3011) is no longer what's in the script: OBJ-3539 (PR #3015) replaced the entire flag-based create call with --policy-from-file, so the tip above (which describes the still-flag-based examples in items 1–5) no longer applies to item 6's provisioning path. Current call, unchanged since PR #3015 (gcp_setup/16_cloud_run_startup_probe_alerts.sh:167-170):

  gcloud monitoring policies create \
    --project="$PROJECT_ID" \
    --notification-channels="$NOTIFICATION_CHANNEL_ID" \
    --policy-from-file="$POLICY_FILE"

  There is no --combiner flag anywhere in the script — combiner: OR is instead set directly in the committed JSON (infra/monitoring/alert-policy-startup-probe-staging.json / -production.json). The script's own comment above the create call confirms this: "The condition (threshold, duration, aggregation grouped by service, combiner) lives in $POLICY_FILE, not in flags" (gcp_setup/16_cloud_run_startup_probe_alerts.sh:152-154).

7. Cloud SQL Connection Saturation

• Trigger: num_backends (active Postgres connections) > 80% of max_connections (20 of 25 on the shared db-f1-micro tier), sustained 5 minutes
• What it detects: a deploy-time revision-overlap connection spike (old + new revision both holding connections) that can starve a new revision's db:prepare and fail the deploy — root cause of OBJ-3154
• Environment: production and staging, provisioned as two separate named policies (staging never had its own coverage until the OBJ-3416 Cloud SQL split gave it a dedicated instance)

Environment	Display name	Instance
Production	Cloud SQL Connection Saturation - enkidu-postgres	enkidu-postgres (25 confirmed live via /health)
Staging	Cloud SQL Connection Saturation - enkidu-postgres-staging	enkidu-postgres-staging (25 confirmed live via /health, 2026-09-18, OBJ-3956)

• Provisioning: policy-as-data — infra/monitoring/alert-policy-cloudsql-connections-enkidu-postgres.json applied via gcp_setup/14_create_cloudsql_connection_alert.sh (OBJ-3156), and its staging sibling infra/monitoring/alert-policy-cloudsql-connections-staging.json applied via gcp_setup/15_create_staging_cloudsql_connection_alert.sh (OBJ-3419). Both use --policy-from-file, so the --if/--combiner flag pitfalls above don't apply — the JSON already sets combiner: OR explicitly. num_backends emits one time series per database, so both JSON files set crossSeriesReducer: REDUCE_SUM grouped by resource.label.database_id to sum instance-wide rather than comparing per-database series (each of which can sit well under threshold while the instance itself is saturated).
• Status: unconfirmed. Per docs/operations/deployment.md § dave-autopilot diagnostics role — IAM posture decisions (OBJ-3232) — the canonical record — the production policy is "reviewed, ready" but stays unprovisioned until monitoring.alertPolicies.create is granted to github-actions-deployer, tracked separately as OBJ-3232 Task 6. Decision 3 (OBJ-3956) closed the verification gap on the read side — dave-autopilot's custom role gained monitoring.alertPolicies.list/.get, which would let Dave confirm either policy's live/attached state directly — but that grant is an IaC record only as of this writing (gcp_setup/21_dave_autopilot_monitoring_read_role.sh, merged, not yet applied live): a human with Project IAM Admin on enkidu-488723 still needs to hand-run it. Neither the staging nor the production policy's live/attached status has been re-checked with the new permission, because the new permission isn't live yet. Don't treat either script, or Decision 3 landing, as proof either policy is provisioned — check deployment.md's Decision 3 entry (or GCP Console > Monitoring > Alerting directly) for current status.
• Verification (once provisioned):

  gcloud monitoring policies list --project=enkidu-488723 --filter='displayName="Cloud SQL Connection Saturation - enkidu-postgres"'
  gcloud monitoring policies list --project=enkidu-488723 --filter='displayName="Cloud SQL Connection Saturation - enkidu-postgres-staging"'

  Same permission caveat as item 6: this needs roles/monitoring.viewer, which dave-autopilot does not hold — confirmed Permission denied running the commands above against enkidu-488723 on 2026-09-08 (flag syntax itself confirmed valid; the denial is IAM, not a bad command). Run with a credentialed human account, or check GCP Console > Monitoring > Alerting directly.

Alert Thresholds Summary

Alert	Threshold	Duration	Severity
Uptime check failed	Check fails	5 minutes	Critical
Cloud Run startup-probe failure	rate > 0	5 minutes	Critical
API latency p95	> 3 seconds	5 minutes	Warning
API error rate	> 5%	5 minutes	Critical
Cloud SQL connections (generic example, item 4)	> 80% of max	5 minutes	Warning
Cloud SQL CPU (generic example, item 5)	> 80%	5 minutes	Warning
Cloud SQL Connection Saturation (prod + staging, item 7)	num_backends > 20 (80% of 25)	5 minutes	Warning — not yet provisioned, see item 7
Sentry error spike	> 10 errors	5 minutes	Warning
Sentry new issue	First occurrence	Immediate	Info
Sentry perf regression	p95 > 2s	N/A	Warning
Critical Path fallback pool (Pipes, retired)	> 5 events	24 hours	Inert — engine deleted
Critical Path fallback pool (Zip)	> 5 events	24 hours	Warning
Native OAuth sign-in failures	≥ 1 error	1 hour	Warning
High-user-count unresolved client error	> 50 users, > 14 days unresolved	N/A	Warning — HUMAN ACTION, not yet created

Escalation Guidance

1. Info alerts (new Sentry issue): Triage during business hours. Assign to relevant team member.
2. Warning alerts (latency, CPU, connections): Investigate within 1 hour. Check dashboards and recent deployments.
3. Critical alerts (uptime failure, error rate spike): Investigate immediately. Check /health endpoint, Cloud Run logs, and recent deploys. Consider rollback if correlated with a deployment.

Last updated: 2026-09-18 (OBJ-3956: item 7's staging row now cites /health's live-confirmed max_connections=25 instead of an unconfirmed assumption; item 7's status corrected from "not yet confirmed live" (stale as of the 2026-09-08 OBJ-3232 writing) to "unconfirmed" — Decision 3 (OBJ-3958) granted dave-autopilot alert-policy read but that grant is not yet applied live, so neither policy's live/attached state has actually been re-checked)

Previously (2026-09-12, OBJ-3695/OBJ-3697): documented the metric-propagation failure mode that surfaced on production run 34715862184 (tag v4.8.4) — the first production push past the OBJ-3638 secret fix, immediately hitting a second failure at policy-create because the log-based metric hadn't yet propagated to the Monitoring API's metric-descriptor index (a separate lag from the Logging API, which is why gcloud logging metrics describe doesn't work as a readiness probe here). gcp_setup/16_cloud_run_startup_probe_alerts.sh now retries monitoring policies create across that gap; item 6's known-issue list and "Status as of" bullet updated accordingly

Previously (2026-09-12, OBJ-3674): added the GDPR Overdue Request Alert catalog entry for the new Gdpr::OverdueRequestAlertJob

Previously (2026-09-08, OBJ-3548): corrected item 6's "Fixed (OBJ-3167 follow-up)" paragraph, which had gone stale — it still credited the flag-based --combiner=OR fix (#3011) after OBJ-3539/PR #3015 replaced that gcloud monitoring policies create call with --policy-from-file; combiner: OR now comes from the committed JSON policy files, not a script flag, confirmed against current gcp_setup/16_cloud_run_startup_probe_alerts.sh

Previously (2026-09-08, OBJ-3232/OBJ-3365 follow-up): added the missing --combiner=OR to gcp_setup/16_cloud_run_startup_probe_alerts.sh's gcloud monitoring policies create call and updated item 6's known-issue note accordingly; fixed gcp_setup/13_github_actions_alert_policy_role.sh's create || update fallback, which masked a validation failure and hung on gcloud's interactive "Replace existing role (Y/n)?" prompt; shortened gcp_setup/github-actions-alert-policy-role.yaml's description under GCP's 300-char cap, relocating the full rationale into YAML comments — permissions unchanged

Previously (2026-09-08, OBJ-3167 Task 3): confirmed the Cloud Run startup-probe alert is live on staging and updated its status accordingly; fixed 5 pre-existing broken gcloud monitoring policies create examples under items 1–5 that used five nonexistent flags — three condition-threshold-* variants plus two separate aggregation-* alignment/aligner flags — replacing them with the real --if/--duration/--aggregation/--combiner interface, flag-parse-verified against a nonexistent project; documented the previously-undocumented Cloud SQL Connection Saturation alert (item 7, OBJ-3156/OBJ-3419) and added its Alert Thresholds Summary row; flagged a known missing---combiner bug in gcp_setup/16_cloud_run_startup_probe_alerts.sh

Previously (2026-09-08, OBJ-3365): documented the Cloud Run Startup-Probe Failure alert for both staging and production, added its Alert Thresholds Summary row, and fixed the stale gcloud monitoring channels list command under Alert Channels to gcloud beta monitoring channels list