Architecture Proposal: Periodic Load Testing Strategy — May 2026

Status: Approved — in progress as milestone v4.15 (Phase 1 shipped) Author: Sage (monthly review, 2026-05-22) Scope: Load testing infrastructure + ops runbooks + GitHub Actions automation

Problem

The codebase has no load testing infrastructure. Two structural failure modes are unquantified:

1. Connection pool cliff

Cloud Run scales horizontally, multiplying open Puma connections per instance. If total connections exceed the Cloud SQL tier max, requests fail immediately — not a slow degradation. The saturation threshold is unknown.

2. Sidekiq burst behavior

Every AddGoalEvent mutation dispatches a background job. At 10 production workers and a 25-second timeout, a traffic spike produces an unconstrained burst enqueue rate with no queue-depth alerting.

Impact

Without load testing:

Connection pool exhaustion surfaces only in production under concurrent load
Sidekiq queue depth can grow unbounded during spikes, blocking new work
No baseline for healthy p95 latency — performance regressions hide in noise
No validation before scaling to 1,000+ concurrent users

Proposed Solution

Tool: k6 OSS

Zero-cost, JavaScript scripting, native GraphQL support, GitHub Actions integration.

Why k6:

Cost: Free (k6 OSS). Cloud ($199/mo) optional for distributed load.
GraphQL: First-class support — queries/mutations in pure JS, no translation.
GitHub Actions: Native CI integration with result reporting.
Threshold assertions: Built-in (e.g., p95 < 500ms).
Results persistence: k6 outputs JSON; we store in Cloud Storage for comparison.

Test Coverage

Five flows mapped to highest-risk concurrency surfaces. L1–L3 ship in Phase 1 (hand-runnable); L4–L5 are added in Phase 2 alongside CI wiring.

Flow	Purpose	VUs (default)	Duration (default)	Threshold
L1: `addGoalEvent` mutation	Puma thread pool + Sidekiq enqueue (streak/gamification jobs)	5	1m	p95 < 800ms, error rate < 1%
L2: `communityFeed` + `goals` queries	N+1 query risk, DB connection pool ceiling	10	1m	p95 < 300ms, error rate < 1%
L3: `syncUser` mutation	Sign-in throughput (Clerk JWT path — fallback auth stands in, see Auth below)	5	1m	p95 < 800ms, error rate < 1%
L4: AI Workforce webhook	Background job creation	—	—	Defined in Phase 2
L5: Health probe	Dependency check (DB, Redis)	—	—	Defined in Phase 2

VUs/duration are per-script defaults, overridable per run via LOAD_TEST_VUS/LOAD_TEST_DURATION env vars — see load_tests/README.md. Thresholds are mutation-vs-query split (mutations <800ms, queries <300ms), not the original per-flow spread — confirmed in the milestone ROADMAP and encoded directly in each script's options.thresholds.

Phased Rollout

Phase 1: k6 scripts in repo — ✅ shipped

Location: load_tests/

load_tests/
├── lib/
│   ├── config.js              # Base URL + staging-only guard, auth headers
│   └── graphql.js             # Shared GraphQL request helper (per-operation tagging)
├── l1_add_goal_event.js       # L1: addGoalEvent mutation
├── l2_read_queries.js         # L2: communityFeed + goals queries
├── l3_sync_user.js            # L3: syncUser mutation
└── README.md                  # Install, auth, running scripts, troubleshooting

Deliverable: Runnable locally, e.g. k6 run load_tests/l1_add_goal_event.js. Defaults to https://staging.api.objectuve.com/graphql; lib/config.js actively refuses to run against a URL that looks like production (api.objectuve.com) unless LOAD_TEST_ALLOW_PRODUCTION=true is set. No package.json/npm dependency — k6 is a single binary, installed separately (see Auth below).

Auth: Scripts authenticate via the SessionToken header (not Authorization: Bearer), using the same fallback-auth path as the Playwright smoke suite — SessionToken: Fallback <FALLBACK_AUTH_SECRET>:<SMOKE_FALLBACK_EMAIL>. FALLBACK_AUTH_SECRET is required (same secret staging's fallback auth checks against, rails_api/app/controllers/graphql_controller.rb); SMOKE_FALLBACK_EMAIL defaults to the seeded staging demo user. See load_tests/README.md#auth for the full env var contract.

Phase 2: GitHub Actions cron (not yet started)

Trigger: monthly cron + manual dispatch, mirroring .github/workflows/playwright-smoke.yml's structure — that workflow, not a smoke-tests.yml (no such file exists in this repo), is the real CI analog for scheduling and results reporting.

yaml

# .github/workflows/load-test.yml (planned name — matches the ROADMAP, not yet created)
name: Monthly Load Tests
on:
  schedule:
    - cron: '0 10 1 * *'
  workflow_dispatch:

jobs:
  load-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: grafana/k6-action@v0.4
        with:
          filename: load_tests/l1_add_goal_event.js
      # Post results as a summary comment to a tracking issue; add L4/L5 scripts

k6 OSS (not the paid Cloud tier) runs this — no K6_CLOUD_TOKEN needed. cloud: true shown in earlier drafts of this proposal assumed the $199/mo Cloud tier, which this milestone does not adopt.

Superseded. This snippet is historical — it predates the workflow's actual implementation and was never updated to match it. The live workflow uses grafana/setup-k6-action@v1 (pinned k6-version: '1.8.1') plus native k6 run steps, not grafana/k6-action@v0.4; the original action was transferred to another org and archived, becoming unresolvable (see the "upstream action transferred to another org" gotcha in CLAUDE.md). See .github/workflows/load-test.yml for the current implementation and docs/operations/observability.md#periodic-load-testing-k6 for the operator-facing runbook.

Deliverable: Automated monthly runs against staging. Results posted to a tracking issue. Dori adds an ops runbook covering how to read a scheduled run's results and respond to a threshold breach.

Phase 3: Baseline storage + regression detection (not yet started)

Write each run's k6 JSON summary to Google Cloud Storage, mirroring the artifact-storage pattern in .github/workflows/mobile-android-release.yml (enkidu-mobile-builds bucket, provisioned via gcp_setup/04_create_mobile_builds_bucket.sh). No load-test-results bucket exists yet — Phase 3 provisions one from scratch; there is no enkidu-load-test-results bucket or K6_CLOUD_TOKEN secret in this repo today. On each run, compare current p95 to the prior baseline; flag (non-blocking) if p95 regresses >20%.

bash

# After each run (illustrative — bucket name TBD at Phase 3 implementation time)
gsutil cp /tmp/results.json gs://<load-test-results-bucket>/$(date +%Y-%m-%d).json

Deliverable: Historical trend tracking. Non-blocking regression flag if p95 rises >20% against the stored baseline. Dori updates the incident-response playbook and adds a CHANGELOG entry once this phase ships.

Phase 4: Lightweight post-deploy gate — deferred

Smoke test in staging after each deploy, blocking or near-blocking production promotion. Deferred beyond this milestone's scope (v4.15 covers Phases 1–3 only): the ROADMAP defers this until Phase 3 has produced at least 3 monthly baselines, so there's enough signal to judge whether a deploy gate is warranted before committing to one. Requires coordination with Riley's CI pipeline when picked back up.

Deliverable (future): Blocks merges if load test fails — gating decision revisited after Phase 3 has 3 months of baseline data.

Implementation Checklist

[x] Phase 1: Add k6 scripts to load_tests/ directory (l1_add_goal_event.js, l2_read_queries.js, l3_sync_user.js, shared lib/)
[x] Phase 1: Add load_tests/README.md with instructions
[ ] Phase 1: Verify scripts run locally against staging (requires a human with FALLBACK_AUTH_SECRET — not run in CI)
[ ] Phase 2: Create .github/workflows/load-test.yml mirroring playwright-smoke.yml
[ ] Phase 2: Add L4/L5 scripts (AI Workforce webhook, health probe)
[ ] Phase 2: Test workflow on manual trigger (workflow_dispatch)
[ ] Phase 2: Schedule cron for first-of-month
[ ] Phase 2: Dori — ops runbook for reading scheduled-run results and responding to a breach
[ ] Phase 3: Provision a GCS bucket for k6 JSON summaries (none exists yet)
[ ] Phase 3: Store baseline from first run; flag (non-blocking) if p95 regresses >20%
[ ] Phase 3: Dori — incident-response playbook update + CHANGELOG entry
[ ] Phase 4 (deferred): Integrate a post-deploy gate into production.yml, after 3+ monthly baselines from Phase 3

Success Criteria

[ ] Scripts execute without error against staging (Phase 1 scripts shipped; a human still needs to run them with the fallback secret — not yet run in this milestone)
[ ] Monthly CI runs are automated (Phase 2, not yet started) + historical storage (Phase 3)
[ ] p95 latency tracked month-over-month (Phase 3)
[ ] Regression flag fires if p95 rises >20% (Phase 3)
[ ] Zero production incidents traced to connection pool or Sidekiq burst (ongoing, measured once Phase 3 has baseline data)

Risk if deferred

Another traffic spike will surface connection pool exhaustion without warning
Sidekiq queue backlog can grow unbounded without visibility
p95 latency drifts silently with each deployment
Scaling to 10,000+ DAU will be unsafe without pre-validated load capacity

PBC Check

Free-tier infra capacity: k6 OSS is free. Cloud tier ($199/mo) optional and deferred.
Free-tier feature integrity: Load tests do not modify user data. Staging-only. Unaffected.
Cost: $0–$199/mo depending on phase 4 opt-in. Currently $0 if phases 1–3 only.

Rollback / Unwind Path

If k6 proves unmaintainable:

Delete load_tests/ directory
Delete .github/workflows/load-test.yml (once Phase 2 creates it)
No dependencies on load tests elsewhere — zero cleanup cost

References

Sage proposal issue: OBJ-502
Child epic: OBJ-503 — Periodic load testing, k6 OSS, phased rollout, staging-only
Milestone: .planning/milestones/v4.15-periodic-load-testing-ROADMAP.md — confirmed 3-phase scope, thresholds, phase ownership
Connection pool architecture: docs/architecture/system.md (Cloud Run + Cloud SQL tier section)
Sidekiq configuration: rails_api/config/sidekiq.yml (default 10 workers)
Auth: rails_api/app/controllers/graphql_controller.rb (SessionToken: Fallback path)
Scripts: load_tests/README.md, load_tests/l1_add_goal_event.js, load_tests/l2_read_queries.js, load_tests/l3_sync_user.js
Related runbooks (to be extended in Phase 2/3): docs/operations/observability.md, docs/operations/incident-response.md

Last updated: 2026-07-24

Next Steps

Phase 1 — shipped: load_tests/ scripts (L1–L3), shared lib/, README.
Phase 1 verification: a human with FALLBACK_AUTH_SECRET runs the scripts locally against staging to confirm they work end-to-end — not yet done.
Phase 2: Codi + Dori add .github/workflows/load-test.yml (mirroring playwright-smoke.yml) with L4/L5 scripts; Dori writes the results-runbook.
Phase 2 soft-launch: Run manual trigger once before scheduling the monthly cron.
Phase 3: Codi + Dori provision a GCS bucket, wire baseline storage + regression flagging; Dori updates the incident-response playbook and adds a CHANGELOG entry.
Phase 4 decision: deferred until Phase 3 has produced 3+ monthly baselines.

Blocking: none — Phase 1 scripts and docs are in review on gsd/v4.15-periodic-load-testing.

Coach Personas

Architecture Proposal: Periodic Load Testing Strategy — May 2026

Problem

1. Connection pool cliff

2. Sidekiq burst behavior

Impact

Proposed Solution

Tool: k6 OSS

Test Coverage

Phased Rollout

Phase 1: k6 scripts in repo — ✅ shipped

Phase 2: GitHub Actions cron (not yet started)

Phase 3: Baseline storage + regression detection (not yet started)

Phase 4: Lightweight post-deploy gate — deferred

Implementation Checklist

Success Criteria

Risk if deferred

PBC Check

Rollback / Unwind Path

References

Next Steps

Objectuve Docs

Architecture Proposal: Periodic Load Testing Strategy — May 2026 ​

Problem ​

1. Connection pool cliff ​

2. Sidekiq burst behavior ​

Impact ​

Proposed Solution ​

Tool: k6 OSS ​

Test Coverage ​

Phased Rollout ​

Phase 1: k6 scripts in repo — ✅ shipped ​

Phase 2: GitHub Actions cron (not yet started) ​

Phase 3: Baseline storage + regression detection (not yet started) ​

Phase 4: Lightweight post-deploy gate — deferred ​

Implementation Checklist ​

Success Criteria ​

Risk if deferred ​

PBC Check ​

Rollback / Unwind Path ​

References ​

Next Steps ​

Objectuve Docs

Architecture Proposal: Periodic Load Testing Strategy — May 2026

Problem

1. Connection pool cliff

2. Sidekiq burst behavior

Impact

Proposed Solution

Tool: k6 OSS

Test Coverage

Phased Rollout

Phase 1: k6 scripts in repo — ✅ shipped

Phase 2: GitHub Actions cron (not yet started)

Phase 3: Baseline storage + regression detection (not yet started)

Phase 4: Lightweight post-deploy gate — deferred

Implementation Checklist

Success Criteria

Risk if deferred

PBC Check

Rollback / Unwind Path

References

Next Steps