Observability

Objectuve uses a GCP-native + Sentry observability stack. All telemetry flows through Google Cloud Logging/Monitoring and Sentry — no additional vendors.

For operational procedures specific to the AI stack (kill-switch flips, budget alert response, rate-limit tuning, API key rotation), see ai-runbook.md.

Stack Overview

Layer	Tool	Purpose
Structured Logging	Lograge → GCP Cloud Logging	JSON request logs, searchable and parseable
Error Tracking	Sentry (backend + frontend)	Exception capture with user context
Health Checks	GET /health endpoint	Component-level status (database, Redis, Sidekiq, LiteLLM, agent_runner)
Performance Tracing	Sentry APM	Request traces, GraphQL spans, web vitals
Uptime Monitoring	GCP Uptime Checks	Endpoint availability alerting
Alerting	GCP Cloud Monitoring + Sentry	Automated incident detection
Dashboards	GCP Cloud Monitoring + Sentry	Production health visibility

Structured Logging

Rails uses lograge to emit single-line JSON logs per request. Cloud Logging auto-parses these.

Fields Logged

Field	Description
method	HTTP method (GET, POST)
path	Request path
status	HTTP status code
duration	Total request time (ms)
db	Database query time (ms)
view	View rendering time (ms)
request_id	Unique request ID for correlation
user_id	User's public_id (if authenticated)
graphql_operation	GraphQL operation name (since all requests hit POST /graphql)
params	Filtered request params (excluding noise like controller, action, query, variables)
time	ISO 8601 timestamp

Searching Logs in Cloud Logging

# All requests for a specific user
gcloud logging read 'jsonPayload.user_id="abc123"' --project=enkidu-488723

# All requests for a specific GraphQL operation
gcloud logging read 'jsonPayload.graphql_operation="CreateGoal"' --project=enkidu-488723

# Slow requests (> 1s)
gcloud logging read 'jsonPayload.duration > 1000' --project=enkidu-488723

# Errors only
gcloud logging read 'jsonPayload.status >= 500' --project=enkidu-488723

Creating Log-Based Metrics

Log-based metrics let you alert on patterns in your logs. For example, to create a metric for slow GraphQL operations:

1. Go to Cloud Logging > Log-based Metrics
2. Create a counter metric with filter: jsonPayload.duration > 2000
3. Use this metric in Cloud Monitoring alerting policies

Error Tracking (Sentry)

Backend (Rails)

• DSN: Set via SENTRY_DSN environment variable
• Environments: staging and production only
• Performance tracing: 10% of transactions in production, 50% in staging
• User context: Automatically set on each GraphQL request (public_id + email)
• Breadcrumbs: Active Support logger + HTTP logger for richer error context
• Excluded exceptions: RoutingError, RecordNotFound (noisy, not actionable)
• GraphQL tracing: Built-in GraphQL::Tracing::SentryTrace creates spans for all GraphQL operations

Frontend (Vue)

• Package: @sentry/vue
• DSN: Set via VITE_SENTRY_DSN environment variable
• Environment: Set via VITE_SENTRY_ENVIRONMENT; falls back to import.meta.env.MODE if unset. Injected per-deploy by CI (staging or production)
• Performance tracing: browserTracingIntegration with Vue Router integration
• User context: Set when Clerk user syncs (via useClerkSync)
• GraphQL errors: Captured in Apollo error handler with operation context tags
• Web vitals: LCP, INP, CLS, TTFB, FCP captured automatically
• Release tag: Sentry.init({ release: __APP_VERSION__ }) in ionic_frontend/src/sentry.ts:15. __APP_VERSION__ is set from the deployed git tag at build time — see Frontend version injection.

Agent Runner (Node.js)

• Package: @sentry/node
• Entry point: agent_runner/src/instrument.ts — imported as the first line of index.ts so @sentry/node auto-instruments Express before any other module loads
• DSN: Set via SENTRY_DSN environment variable
• No-op behaviour: When SENTRY_DSN is unset (local dev, NODE_ENV=test), Sentry.init is never called — the service runs silently without crash reporting
• Environment: Set via SENTRY_ENVIRONMENT; falls back to NODE_ENV if unset. Set explicitly in CI so staging events aren't mislabelled as production
• Release tag: Set via SENTRY_RELEASE (optional); passed through to Sentry.init({ release }) for per-release attribution in Sentry
• Express error handler: Sentry.setupExpressErrorHandler(app) is placed after all routes in index.ts
• Unhandled rejections: captureException(reason) is called in the unhandledRejection process handler

Sentry frontend release: stale tags v3.4.74 → v3.8.5 (post-mortem)

Frontend Sentry events between v3.4.74 (last manual package.json bump on Apr 5, 2026) and v3.8.5 (Apr 27, 2026 ship) were tagged with release: 3.4.74 regardless of which version actually deployed. Errors and performance traces shipped, but the release attribution was wrong for every frontend deploy in that window.

Cause. __APP_VERSION__ was sourced from process.env.npm_package_version, which reads ionic_frontend/package.json#version. The release command and the production deploy workflow both bumped the git tag but never the package.json field, so the bundled value was frozen at the value last manually written to package.json (3.4.74). Sentry's release: __APP_VERSION__ inherited the same stale value.

Fix. OBJ-74 bumped package.json to 3.8.6 as a one-time patch. OBJ-75 made the source CI-driven so it can't drift again — see Frontend version injection. From the first build that ships under that mechanism forward, frontend Sentry releases match the deployed tag.

Implications for historical analysis. Any Sentry filter, dashboard, or "errors per release" comparison covering frontend events in this window points at the wrong release name:

• All releases v3.4.75 → v3.8.5 are absent from the Sentry frontend release list.
• The release:3.4.74 bucket in Sentry contains events from every deploy across that window mixed together; it's effectively unusable for per-release attribution.
• Backend Sentry releases were unaffected — SENTRY_RELEASE=${{ needs.validate.outputs.version }} in production.yml has always tracked the tag (with a v prefix; see the format-mismatch note below).
• Frontend-only regressions surfaced in this period may have been misattributed during triage.

Decision needed (observability owner). Two options for the historical bucket:

1. Reset the frontend release list — drop release:3.4.74 events older than the fix and let the new mechanism start cleanly.
2. Accept the discontinuity — leave the historical bucket in place as a known-bad blob, document it here, move on.

No work-in-progress on this; flagging here so whoever picks it up has the context.

Sentry frontend/backend release format mismatch

Backend SENTRY_RELEASE (set in .github/workflows/production.yml) keeps the v prefix (e.g. v3.8.5). Frontend release is bare (e.g. 3.8.5) because vite.config.ts reads VITE_APP_VERSION, which the production workflow exports as ${GITHUB_REF_NAME#v} so existing templates that render v{{ appVersion }} continue to display correctly.

The two streams therefore can't be joined on release name in Sentry's UI without manual normalization. Aligning to bare on both sides (drop the v from backend SENTRY_RELEASE) is the simpler resolution since the frontend is constrained by template rendering. This is out of scope of OBJ-75 — flagged here as a separate observability decision for whoever owns release-tag conventions.

Finding User-Specific Errors

In Sentry, search by user.id:PUBLIC_ID to find all errors for a specific user. The user's email is also attached for quick identification.

AI Settings Observability

Two v4.59 Phase 5 additions (CAP-22, CAP-23, OBJ-3753) close observability gaps around Ai::SettingsGuard (rails_api/app/services/ai/settings_guard.rb), the shared fail-closed helper every AI feature-flag check routes through.

Rejected AI mutations now log a warn

Before this phase, a rejected AI mutation was completely silent: all six Ai::SettingsGuard guard clauses raise GraphQL::ExecutionError, and the mutation's own rescue GraphQL::ExecutionError; raise re-raised with zero Rails.logger call and zero Sentry breadcrumb — unlike the rescue StandardError branch beside it, which does both. Ai::SettingsGuard.enabled? now emits a warn-level log (carrying flag_name, category, and context) on the falsy-return/disabled path, alongside its existing fail-closed error log for the unknown-flag case — a single change point covering all six call sites, including Ai::ServiceClient and GenerateGoalQuickUpdatesJob. The boolean contract enabled? returns is unchanged; nothing that depended on it — including the one Sidekiq-job call site — needed to change. This is a log line, not a new Sentry event: no exception is captured, only the breadcrumb trail gets richer for whoever's triaging a "why didn't this AI feature fire" report.

AI settings parity guard

A scheduled drift workflow (in the shape of this repo's other *-drift.yml guards — see clerk-auth-config-drift.yml/deploy-secret-drift.yml for the pattern) asserts that the four ai.*_enabled flags agree across settings.yml and every settings/{development,test,staging,production}.yml file, failing with the flag name and the disagreeing files when they don't. It runs on schedule and workflow_dispatch, and — like the capacity guard in Rails Concurrency & Capacity — is not in master CI gate's required contexts; promoting a context to required is a repo-admin decision, stated in the workflow's own header rather than implied.

This is narrower than Phase 3's retracted /health ai_settings check: that check's premise (a live-vs-committed flag divergence) was proven not to exist, and this guard doesn't rebuild it. It also never surfaces spend or model-selection data — monthly_budget_cents, budget_alert_thresholds, *_monthly_budget_cents, and default_coach_model are excluded from its output by design; only the flag booleans and which files disagree are reported.

Health Check Endpoint

GET /health

Returns component-level health status.

Response (200 OK):

{
  "status": "ok",
  "timestamp": "2026-03-24T10:30:00Z",
  "checks": {
    "database": { "status": "ok", "active_connections": 4, "max_connections": 100 },
    "redis": { "status": "ok" },
    "sidekiq": {
      "status": "ok",
      "processed": 15420,
      "failed": 3,
      "enqueued": 0,
      "dead": 0,
      "queues": [
        { "name": "default", "size": 0, "latency": 0.0 }
      ]
    },
    "litellm": { "status": "ok" },
    "agent_runner": { "status": "not_applicable", "message": "agent_runner sidecar runs in enkidu-worker-staging" }
  }
}

Response (503 Service Unavailable):

{
  "status": "degraded",
  "timestamp": "2026-03-24T10:30:00Z",
  "checks": {
    "database": { "status": "ok", "active_connections": 4, "max_connections": 100 },
    "redis": { "status": "error", "message": "Connection refused" },
    "sidekiq": { "status": "error", "message": "Connection refused" },
    "litellm": { "status": "error", "message": "execution expired" },
    "agent_runner": { "status": "not_applicable", "message": "agent_runner sidecar runs in enkidu-worker-staging" }
  }
}

What Each Check Verifies

Check	What It Does	Failure Means
database	SELECT 1 via ActiveRecord, plus one pg_stat_activity/current_setting('max_connections') round trip that reports active_connections/max_connections on the ok path (added in OBJ-3031 to surface Cloud SQL connection-slot exhaustion before it trips — see Database Connection Exhaustion)	PostgreSQL is unreachable
redis	PING via Redis client	Redis is down (affects Sidekiq + ActionCable)
sidekiq	Sidekiq::Stats + queue inspection, plus dead-queue size vs. SIDEKIQ_DEAD_THRESHOLD (default 25)	Sidekiq API unreachable (depends on Redis); or degraded (not error) when the dead-queue size exceeds the threshold — jobs are stuck permanently even though Sidekiq itself is reachable. See Sidekiq dead set runbook to clear it.
litellm	GET /health/liveliness on the LiteLLM Cloud Run service (OIDC-authenticated)	LiteLLM is unreachable — blocks all AI Coach / AI Workforce completions
agent_runner	Informational only — always returns not_applicable, since the agent_runner sidecar runs on enkidu-worker-{env}, not the web service that serves /health. Does not affect the overall status or HTTP code. See Health check informational sub-checks gotcha for why this check must return a neutral status rather than let a doomed TCP probe emit "down".	N/A (never gates the overall verdict)

Sidekiq's degraded path: unlike the other checks, sidekiq can report status: "degraded" on an otherwise-healthy connection — this happens when Sidekiq::DeadSet#size exceeds SIDEKIQ_DEAD_THRESHOLD (default 25). A degraded sidekiq check flips the top-level status to "degraded" and the HTTP status to 503, same as an error. Never gate on stats.failed (a lifetime cumulative counter) — it grows monotonically on a busy service and will eventually false-positive. This shared-infra degraded state hard-fails the Playwright smoke pre-flight for every open PR, not just the one that caused it — see Sidekiq dead set runbook to diagnose and self-serve a fix from the Actions tab, no gcloud required.

Usage

• Load balancer: Use GET /up (Rails default) for simple up/down checks
• Detailed monitoring: Use GET /health for component-level status
• Uptime monitoring: Point GCP Uptime Checks at /health

Performance Monitoring

Backend: Sentry APM

Sentry captures transaction traces for all requests. GraphQL operations get individual spans via GraphQL::Tracing::SentryTrace, showing:

• Total query execution time
• Individual field resolution times
• Database query spans (via ActiveRecord instrumentation)

View traces in Sentry > Performance > Transactions.

Frontend: Web Vitals

Sentry automatically captures Core Web Vitals:

Metric	What It Measures
LCP	Largest Contentful Paint — loading performance
INP	Interaction to Next Paint — responsiveness
CLS	Cumulative Layout Shift — visual stability
TTFB	Time to First Byte — server response time
FCP	First Contentful Paint — initial render time

View in Sentry > Performance > Web Vitals.

PostHog Acquisition Funnel Events

Objectuve tracks the user acquisition funnel via PostHog events, surfacing key conversion moments across the store-to-first-goal journey.

Event	Trigger	Location	Frequency
store_listing_visited	Page load on marketing landing site	marketing_landing/index.html, main.js	Every page load
first_open	First successful app boot post-install	ionic_frontend/src/App.vue onMounted	Once per device (localStorage guard: objectuve_first_open_fired)
first_goal_created	After createGoal mutation completes successfully	ionic_frontend/src/composables/useGoalForm.ts	Once per device (localStorage guard: objectuve_first_goal_created_fired)
app_installed	Collapsed into first_open (see note below)	—	—

Note on app_installed: The JavaScript runtime cannot observe the OS-level app install event. Both app_installed and first_open would fire at the same moment (first time the app JS executes post-install). PostHog convention is to use first_open for this concept, so the two events were consolidated.

PostHog Project Key: The production PostHog project key (phc_CcuNbgW2uNpBrK2yMs8mcezGbC3aJ9CDwCinM76idWEd) is hardcoded in marketing_landing/index.html (public key — no secret). It is sourced from .github/workflows/staging.yml and is public-facing by design (PostHog client keys are intentionally published on every web page).

UTM attribution on first_open

first_open carries source attribution properties when the user arrived via a tagged link. Attribution is captured from the landing URL by captureSourceAttribution() (ionic_frontend/src/lib/sourceAttribution.ts) and persisted to localStorage before Vue Router performs its first navigation (which rewrites window.location). trackFirstOpen() reads the persisted value via readPersistedSourceAttribution().

Property	Source	Notes
utm_source	?utm_source= query param	e.g. producthunt, hackernews, twitter
utm_medium	?utm_medium= query param	e.g. referral, social
utm_campaign	?utm_campaign= query param	e.g. launch
invite_code	?invite_code= query param	Referral invite code if present
ref	?ref= query param	Short referral tag
referrer	document.referrer	External referrer URL; internal referrers filtered out

localStorage persistence key: objectuve_source_attribution (defined in sourceAttribution.ts:93)

Initial-touch semantics: Written only once — on the first landing that carries attribution. Subsequent untagged page loads do not overwrite the original source. This matches standard funnel-attribution modeling.

Super-property propagation: After first_open fires, attribution is registered via posthog.register() (acquisitionAnalytics.ts:64) so all downstream events — including first_goal_created — carry the same UTM values as PostHog super-properties without additional instrumentation.

First-touch person properties: posthog.setPersonProperties() (acquisitionAnalytics.ts:65–70) writes initial_utm_source, initial_utm_medium, initial_utm_campaign, etc. ($set_once semantics) so the person profile stays attributed across sessions.

Platform scope: Web/PWA and Android Universal Links (via the appUrlOpen Capacitor handler at App.vue:177). Native iOS App Store cold-installs cannot carry UTM params without a deferred-deeplink SDK — see the launch runbook for the iOS attribution gap callout.

Critical Path Analytics (R5+)

Critical Path daily puzzles emit PostHog events tracking player engagement, progression, and abandonment. See critical-path-analytics.md for the full event taxonomy, PostHog funnel setup (tile tapped → puzzle started → completed), and 7-day-player cohort definition for retention analysis.

Key events:

• critical_path_tile_tapped — user opened the puzzle tile
• critical_path_puzzle_started — user began solving
• critical_path_puzzle_completed — user won
• critical_path_puzzle_abandoned — user navigated away mid-game
• critical_path_tutorial_viewed — user viewed how-to overlay
• critical_path_undo_tapped — user used undo

PWA Events (v1.15+)

The Progressive Web Application layer (web/desktop users) emits PostHog events for installation and updates:

Event	Fired When	Source	Segment
pwa_installed	User completes PWA installation (not App Store)	window.addEventListener('appinstalled', ...) in src/lib/swInit.ts	Web only (!Capacitor.isNativePlatform())
pwa_update_applied	User accepts SW update and refresh completes	usePwaUpdateToast.ts after applyUpdate() resolves	Web only (!Capacitor.isNativePlatform())

No extra properties — both events fire with name only.

Suppression conditions:

• import.meta.env.MODE === 'development' — events are suppressed during local dev
• Capacitor.isNativePlatform() — events are suppressed on native iOS/Android

Monitoring queries:

-- PWA installs this week
SELECT count() FROM events WHERE event='pwa_installed' AND toDate(timestamp) >= today() - 7

-- Update adoption (installed users who updated)
SELECT count(distinct person_id) FROM events 
  WHERE event='pwa_update_applied' AND toDate(timestamp) >= today() - 7

For PWA architecture details, see docs/architecture/pwa-augment.md.

Periodic Load Testing (k6)

Monthly scheduled k6 load tests run against staging only — never production — to catch performance regressions before they reach a real deploy. This is a signal source, not a deploy gate: the workflow never runs on pull_request and blocks nothing.

Full script details (thresholds, auth setup, hand-running instructions, troubleshooting) live in load_tests/README.md — this section covers the scheduled CI wiring only.

What runs

Five scenarios run by default, each with a p95/error-rate threshold, plus a sixth that's opt-in only (see below):

Scenario	Flow	Risk area
L1	addGoalEvent mutation	Puma thread pool + Sidekiq enqueue
L2	communityFeed + goals queries	N+1 query risk, DB connection pool ceiling
L3	syncUser mutation	Sign-in throughput (Clerk JWT path)
L4	POST /webhooks/ai-workforce	HMAC-signed machine-to-machine webhook, background job creation path
L5	GET /health	Dependency check (database, Redis, Sidekiq)
L6	Concurrent AI-coaching mutations vs. a cheap authenticated read	Puma-thread contention (parked — see below)

L6 — AI-coaching contention (parked)

load_tests/l6_ai_contention.js is structurally different from L1–L5: instead of measuring one flow against itself, it drives concurrent AI-coaching mutations (refineDescription, getInsight — each occupying a Puma thread synchronously for the duration of the LiteLLM call, at the time this script was written) alongside a cheap MeQuery read, and asserts the read's p50 latency. It's the load-test-level proof of the containerConcurrency/thread-capacity fix documented in Rails Concurrency & Capacity: before the fix, an AI mutation holding a thread starved unrelated request latency (read p50 1.81s); after, 57ms.

Status: parked as of v4.59 Phase 3 (CAP-21, OBJ-3751), and it can never reproduce the original contention again. 72ba08433 (v4.60 Phase 5, OBJ-3762) moved refineDescription/getInsight — the last two synchronous AI mutations this script relied on — onto Sidekiq via Ai::DispatchAiRequestJob. Every AI mutation in the script now returns immediately after enqueueing, so there's no Puma thread held for the length of an LLM call anymore, and the script's p(50)>800 floor thresholds (its proof that a call actually reached the LLM, not a fast no-op) would fail on every future run for that reason — not a regression. Un-parking precondition: a synchronous AI mutation exists again (check rails_api/config/initializers/ai_request_handlers.rb for an Ai::AiRequestHandlerRegistry kind with zero production registration before reusing this script or its pattern).

It's parked behind an opt-in run_l6 workflow_dispatch input on .github/workflows/load-test.yml — the monthly cron and an unopted manual dispatch both skip it; only an explicit run_l6: true manual dispatch runs it. Hand-runnable directly at any time regardless of parking: k6 run load_tests/l6_ai_contention.js (see load_tests/README.md for required env vars).

Trigger

.github/workflows/load-test.yml:

• Monthly cron — 0 10 1 * * (1st of the month, 10:00 UTC)
• Manual dispatch — workflow_dispatch, no inputs. Trigger from the Actions tab (Load Tests (staging) → Run workflow, on master) or via gh:

  gh workflow run load-test.yml

The workflow shares the smoke-staging-demo concurrency group (cancel-in-progress: false, queue: max) with staging.yml and smoke-goal-cleanup.yml, so a scheduled or manually dispatched run queues behind other staging smoke traffic instead of racing it against the shared demo account.

k6 install

CI installs k6 via grafana/setup-k6-action@v1, pinned to k6-version: '1.8.1' — not latest, so a run's log always names the exact binary that produced its results. Each of the five scenarios then runs as a native k6 run <script> --summary-export=<path> step. See the "upstream action transferred to another org" gotcha in CLAUDE.md for why this is pinned rather than left on latest.

Required secrets

Secret	Used by	Notes
FALLBACK_AUTH_SECRET	L1–L5	Same fallback-auth secret the Playwright smoke suite uses for staging
SMOKE_FALLBACK_EMAIL	L1–L5	Optional — defaults to demo@objectuve.com
RAILS_WEBHOOK_SECRET	L4 only	HMAC-SHA256 signing secret for Webhooks::AiWorkforceController, matching the value agent_runner signs its real webhook calls with

A "Verify required secrets" step fails fast with a fix hint if FALLBACK_AUTH_SECRET or RAILS_WEBHOOK_SECRET is missing. Because this workflow never runs on pull_request, a missing secret only fails a scheduled/dispatched run — it never blocks PR CI.

Where results land

Every run — pass or fail — posts a summary comment to the tracking issue (enkidu#1754) with a per-scenario table of status, p95, and error rate. The GitHub Actions run itself is linked at the top of each comment.

Responding to a failure or threshold breach

1. Open the summary comment on issue #1754 and identify which scenario(s) failed (❌) or missed threshold.
2. Open the linked Actions run for the full k6 output (per-operation http_req_duration/http_req_failed breakdown).
3. Cross-reference with Sentry APM and Cloud Logging (see Performance Monitoring and Structured Logging above) for the same time window to correlate with a deploy, migration, or traffic spike.
4. For L4 specifically, a 422 on every request is expected, not a failure — the script's run_id never resolves to a real AiRun. Only a 401 (bad signature) or a 5xx indicates a real problem. See load_tests/README.md's Troubleshooting section.

Staging row cleanup

L1 and L4 write request volume to the shared staging demo account on every run (L1 creates real GoalEvent rows; L4's webhook lookup never creates an AiRun row since the synthetic run_id never resolves, but still hits the endpoint). Both scripts keep modest default VUs/duration specifically to bound this. Staging's existing smoke-goal-cleanup.yml/smoke-activity-cleanup.yml crons purge the demo account's Goal/UserAction/UserNotification/UserFeedItem/IntegrationConnection/IntegrationActivity tables daily — see the "unbounded list field" gotcha in CLAUDE.md — but do not currently target GoalEvent specifically; if monthly load-test volume ever needs its own cleanup, extend one of those crons rather than adding a new delete path.

Baseline storage + regression detection

Every run — scheduled or manually dispatched — persists its combined k6 summary to GCS and diffs it against last month's, purely as a signal. This gates nothing: a regression is a non-blocking annotation on the tracking-issue comment, never a failed check.

Where results land. Each run uploads one date-prefixed JSON object to gs://enkidu-load-baselines/summaries/<YYYY-MM-DD>-run<GITHUB_RUN_ID>.json, pruned to the last KEEP_BASELINES (24, ~2 years of monthly runs) via the same gcloud storage ls | sort -r | tail -n +N pattern mobile-android-release.yml uses for build artifacts. Auth is WIF (google-github-actions/auth@v3, id-token: write), same GCP_PROJECT_ID/GCP_SERVICE_ACCOUNT/GCP_WORKLOAD_IDENTITY_PROVIDER as the rest of CI — no static key. The bucket is provisioned by gcp_setup/10_create_load_baselines_bucket.sh (idempotent; grants the deployer SA roles/storage.objectAdmin, no signed-URL role needed since nothing downloads directly from a browser).

How the comparison works. load_tests/compare_baseline.mjs (.github/workflows/load-test.yml's "Compare current run against baseline" step) downloads the most recent object whose date prefix falls in a calendar month before the current run's — ascending lexicographic sort on the YYYY-MM-DD-prefixed object name doubles as chronological sort — then compares each scenario's per-operation p95 (http_req_duration{operation:X}, the same tags the "Build k6 summary" step already extracts) against that baseline. Any operation whose p95 rose more than 20% (REGRESSION_THRESHOLD_PCT in compare_baseline.mjs) is flagged ⚠️ regression in a comparison table appended to the tracking-issue comment (enkidu#1754).

Non-blocking by construction, not just by convention. The step runs with continue-on-error: true, and compare_baseline.mjs's main() never calls process.exit with a non-zero code — a regression only ever emits a ::warning:: annotation. The first run of all time, or the first run of a new month with nothing stored yet, reports "No prior month baseline found — this run establishes the first baseline" instead of erroring; the current run's own baseline still uploads.

Reading a regression signal. The comparison table's Δ column shows the percent change per operation; cross-reference the flagged operation against What runs above to identify the risk area (e.g. an L2 regression points at the DB connection pool, an L1 regression points at Sidekiq enqueue throughput) and see the connection-pool-cliff / Sidekiq-burst playbook in the incident-response runbook for first mitigations.

Verifying the diff math. load_tests/compare_baseline.test.mjs covers extractOperationMetrics, computeDelta (including the exactly-at-threshold boundary), compareBaselines, and markdown rendering — run via node --test --experimental-test-coverage --test-coverage-lines=80 load_tests/compare_baseline.test.mjs, wired into load-test.yml itself (not the main PR CI pipeline — see the Open follow-ups below).

Open follow-up (non-blocking, flagged in Roy's Phase 3 review, not yet actioned): compare_baseline.test.mjs only runs inside load-test.yml, which is gated behind schedule/workflow_dispatch — it never runs on a normal PR, so a change to the diff math wouldn't be caught until the next monthly cron. A future PR could wire node --test load_tests/compare_baseline.test.mjs into the main CI pipeline as an unconditional job.

Runbook: Common Investigations

"The API is slow"

1. Check Sentry Performance for slow transactions
2. Check Cloud Run metrics for elevated latency (Cloud Console > Cloud Run > Metrics)
3. Check Cloud SQL metrics for high CPU or connection count
4. Check /health endpoint for Sidekiq queue backlog
5. Check Cloud Logging for slow queries: jsonPayload.duration > 2000

"Users are getting errors"

1. Check Sentry Issues for new/spiking errors
2. Filter by user: user.id:PUBLIC_ID
3. Check error breadcrumbs for context
4. Check Cloud Logging for corresponding request: jsonPayload.request_id="REQUEST_ID"

"Is the API up?"

1. Check GCP Uptime Check status in Cloud Monitoring
2. curl https://api.objectuve.com/health — inspect component status
3. Check Cloud Run service status in Cloud Console

"Background jobs aren't processing"

1. Check /health — look at sidekiq.enqueued and queue latency
2. Check Cloud Logging for Sidekiq errors
3. Check Redis connectivity via /health redis check
4. Check Sidekiq process status on Cloud Run

Last updated: 2026-09-17 (v4.59 Phase 5, OBJ-3753/OBJ-3916 — L6 parked status, CAP-22 rejected-mutation logging, CAP-23 AI settings parity guard)