Objectuve Docs

Appearance

v4.15 — Periodic Load Testing ​

Enkidu now knows where its Cloud SQL connection pool and Sidekiq queue actually break, before a real traffic spike finds out for us.

Summary ​

Before this milestone, Enkidu had zero load-testing infrastructure. Performance regressions in the database connection pool, Sidekiq burst capacity, or GraphQL N+1 hotspots would only surface once real production traffic hit them — reactively, in an incident.

This milestone adopted k6 OSS for staging-only periodic load testing, built in three phases: hand-runnable scripts first, then monthly scheduled CI, then baseline storage with regression detection. It never touches production — the workflow has no pull_request trigger and gates no deploy, by design. It's a signal source: a monthly snapshot of how staging holds up under synthetic load, posted where a human will see it, with enough history to tell a real regression from normal noise.

Five scripts now exercise the platform's highest-risk paths — goal-event writes (Sidekiq enqueue), community/goals reads (N+1 and connection-pool pressure), sign-in throughput (Clerk JWT), the AI Workforce webhook (HMAC-signed machine-to-machine path), and the /health dependency check — on a monthly cron, with results landing as a tracking-issue comment and a GCS-backed month-over-month baseline comparison.

Goal ​

Adopt k6 OSS for staging-only periodic load testing so the Cloud SQL connection-pool saturation point and Sidekiq burst behavior are known before a real traffic spike hits production.

Scope — What Shipped ​

  • load_tests/ directory: five k6 scripts —
    • l1_add_goal_event.js — addGoalEvent mutation (Puma thread pool + Sidekiq enqueue)
    • l2_read_queries.js — communityFeed + goals query fields (N+1 risk, DB connection pool ceiling)
    • l3_sync_user.js — syncUser mutation (sign-in throughput, Clerk JWT path)
    • l4_ai_workforce_webhook.js — POST /webhooks/ai-workforce (HMAC-signed webhook, background job creation path)
    • l5_health.js — GET /health (database, Redis, Sidekiq dependency check)
    • Shared load_tests/lib/config.js + lib/graphql.js auth and GraphQL helpers; load_tests/README.md
  • .github/workflows/load-test.yml — monthly cron (0 10 1 * *) + manual workflow_dispatch, mirroring playwright-smoke.yml's staging-targeting and fallback-auth secret wiring. Shares the smoke-staging-demo concurrency group so it queues behind other staging smoke traffic rather than racing it.
  • Per-run summary comment posted to tracking issue enkidu#1754, with a per-scenario status/p95/error-rate table.
  • GCS baseline storage: every run's combined k6 summary uploads to gs://enkidu-load-baselines/summaries/<YYYY-MM-DD>-run<GITHUB_RUN_ID>.json, pruned to the last 24 (~2 years of monthly runs).
  • load_tests/compare_baseline.mjs — diffs the current run's per-operation p95 against the prior month's stored baseline, flagging any operation whose p95 rose more than 20% as a non-blocking ⚠️ regression annotation on the tracking-issue comment. Never fails the workflow, never gates a deploy.
  • Operational documentation: docs/operations/observability.md § Periodic Load Testing (k6) (CI wiring, secrets, response runbook, baseline mechanics) and docs/operations/incident-response.md § Load Test Regression: Connection-Pool Cliff / Sidekiq Burst (first-mitigations playbook for a flagged regression).

Phases ​

PhaseNameStatusPlansHighlights
1k6 scripts, hand-runnable against stagingShipped1L1–L3 scripts + shared auth/GraphQL helpers; no CI wiring yet, staging-only base URL default
2Monthly scheduled CI against stagingShipped1.github/workflows/load-test.yml, adds L4/L5, posts results to a tracking issue
3Baseline storage + regression detectionShipped1GCS-backed monthly baselines, non-blocking >20% p95-regression flag

Key Decisions ​

  • Staging-only, by construction, not just by convention — every script defaults to a staging base URL and refuses to hit production unless explicitly overridden; the CI workflow never sets a production URL and never runs on pull_request. This was the milestone's top-listed risk ("accidental production targeting") and was mitigated architecturally rather than by policy alone.
  • Advisory, never a deploy gate — the regression flag is deliberately non-blocking: the comparison step runs with continue-on-error: true, and compare_baseline.mjs's main() never exits non-zero. A p95 regression produces a ::warning:: annotation and a flagged row in the tracking-issue comment, nothing more.
  • Phase 4 (a post-deploy blocking gate) deferred, not cut — Sage's original proposal included a fourth phase that would turn this into a real deploy gate. The milestone explicitly deferred it until Phase 3 has produced at least 3 monthly baselines, so there's enough signal history to gauge whether a >20% threshold is the right bar before anything becomes blocking.
  • Reuse existing patterns rather than inventing new ones — auth reuses the Playwright smoke suite's FALLBACK_AUTH_SECRET/SMOKE_FALLBACK_EMAIL; the CI workflow mirrors playwright-smoke.yml's structure; the GCS upload/prune pattern mirrors mobile-android-release.yml's artifact storage. No new auth mechanism, no new CI shape, no new storage pattern.

Requirements Coverage ​

6 / 6 deliverables satisfied (quoted from the MILESTONE-AUDIT). This is an infra/CI-scripting milestone with no application-facing requirement IDs, so deliverables are tracked by the ROADMAP's phase checklist items rather than a numbered REQUIREMENTS.md.

CategoryCountStatus
Phase 1 (scripts + proposal doc)2All satisfied
Phase 2 (CI workflow + ops runbook)2All satisfied
Phase 3 (baseline storage + regression flag)2All satisfied

Full detail: v4.15-periodic-load-testing-MILESTONE-AUDIT.md on GitHub.

Outcomes ​

Enkidu now has a monthly, staging-only signal for how the platform holds up under synthetic load across its highest-risk paths: goal-event writes, community/goals reads, sign-in throughput, the AI Workforce webhook, and /health. Anyone can also hand-run any script on demand (load_tests/README.md) without waiting for the monthly cron. A flagged regression in the tracking-issue comment points straight at the affected risk area — a connection-pool cliff or a Sidekiq burst — and the incident-response runbook gives first mitigations. None of this blocks a PR or a deploy: it's a leading indicator a human reads, not a gate.

Tech Debt ​

  • (Phase 3, standing) load_tests/compare_baseline.test.mjs only runs inside load-test.yml itself (gated behind schedule/workflow_dispatch), so a change to the regression-diff math wouldn't be caught by CI until the next monthly cron actually exercises it. Flagged in Roy's Phase 3 review; not yet wired into the main PR CI pipeline.
  • Phase 4 (post-deploy blocking gate) remains deferred, pending at least 3 monthly baselines from Phase 3 — an intentional scope decision from the outset, not a gap.
  • 7802d4880 — [Codi] feat(load-tests): k6 hand-runnable scripts for L1-L3 + proposal doc reconciliation (OBJ-1703) (#1750)
  • 4fdaf2799 — [Codi] feat(load-tests): add monthly scheduled CI load-test workflow (OBJ-1704) (#1755)
  • 2fc7b0cf1 — [Codi] feat(load-tests): add GCS baseline storage + p95 regression detection (OBJ-1705) (#1756)

Retroactive milestone-close note: this page was authored 2026-08-04, ~11 days after the milestone actually shipped (2026-07-24) — the standard milestone-close artifacts were skipped at the time and the gap was found during OBJ-2150's milestone-ledger reconciliation (Orion), then compiled here per OBJ-2159 (Dori). Dates above reflect the original ship date, not the authoring date.

Last updated: 2026-08-04

Loading…