Objectuve Docs

Appearance

v4.59 — Backend Concurrency Remediation ​

A burst of concurrent AI-coaching requests no longer makes every other request on the same API instance slow.

Summary ​

A user report that "the backend seems to be taking very long and similarly threw Sentry errors" led to a diagnosis, from Dave, that the Rails API's concurrency model had drifted apart from its own deploy configuration. RAILS_MAX_THREADS was set nowhere in the repo, so Puma ran its hardcoded default of 3 threads in a single process while every deploy/service.*.yaml advertised containerConcurrency: 80 — 27x the instance's real execution capacity. Seven GraphQL AI mutations called LiteLLM synchronously in-request with a 30-second timeout, so a small burst of coaching requests occupied every available Puma thread, and every other request on that instance — a cheap indexed lookup, a health check, anything — inherited the wait. Sentry's OBJECTUVE-API-2Q had been unresolved since 2026-08-22, with GraphqlController#execute p95 at 19.5 seconds.

Planning corrected one half of the original diagnosis (the AR pool, at 5, was never the binding constraint — only the 3 Puma threads were) and surfaced three findings the diagnosis had missed: Cloud Run only triggers autoscaling at roughly 60% of containerConcurrency, so an oversized value silently defeats the platform's own scale-out signal; deploy/service.production.yaml declared no maxScale at all, inheriting a 100-instance ceiling against a database with roughly 25 available connections; and production's Sidekiq worker was already running concurrency 10 against an AR pool of 5 — pool-starved today, independent of this incident.

Five phases shipped directly to master, one PR per phase, over three days. The first two phases made the model explicit — decoupling the two env vars that used to share one knob, and committing a per-database connection budget as a real artifact rather than prose in a manifest comment. The next two phases applied that model, staging first (proving the fix with a new load test that measured a cheap read's latency while AI mutations saturated the instance) and production second, where a re-measurement found the real peak had grown past what the committed budget could safely fund. The final phase made the model self-enforcing: a CI guard that fails the build the moment any future manifest change lets the arithmetic drift apart again, plus the documentation that explains why the model exists and what its current honest limit is.

Goal ​

A burst of concurrent AI-coaching requests stops making every other request on the same instance slow. The Rails API's concurrency capacity is explicitly declared, bounded, sized against a written per-database connection budget, and mechanically guarded so the drift that caused this cannot silently reappear.

Scope — What Shipped ​

  • Decoupled concurrency config (rails_api/config/puma.rb, rails_api/config/database.yml) — PUMA_MAX_THREADS and DB_POOL_SIZE each resolve independently through their own fallback chain (still falling back to the shared RAILS_MAX_THREADS for back-compat, then to their own historical default), instead of one variable silently driving both.
  • deploy/capacity-budget.json (new) — the committed, machine-readable per-Cloud-SQL-instance connection budget: ceiling, provenance, reserves, and every consuming service's declared draw. The single source every later phase sizes against and the CI guard reads.
  • Worker pool-starvation fix — enkidu-worker-production/-staging now declare PUMA_MAX_THREADS explicitly (the worker runs Puma and Sidekiq sharing one AR pool, a Phase 2 finding outside the original diagnosis) and their DB_POOL_SIZE covers Sidekiq's own concurrency.
  • Staging capacity retune (deploy/service.staging.yaml) — containerConcurrency dropped to match resolved Puma threads; PUMA_MAX_THREADS/DB_POOL_SIZE set explicitly and sized against the budget file.
  • load_tests/l6_ai_contention.js (new) — a k6 scenario that drives concurrent AI-coaching mutations while measuring a separate cheap read's latency, the specific poisoning pattern this milestone exists to prevent. Wired into .github/workflows/load-test.yml. Parked behind an opt-in run_l6 dispatch input after the contention it measures stopped being reproducible (see Tech Debt).
  • Production capacity retune (deploy/service.production.yaml) — explicit maxScale: "3", PUMA_MAX_THREADS: "1", DB_POOL_SIZE: "1", containerConcurrency: 1, sized to the committed budget (see Outcomes for the honest limit this carries).
  • scripts/check-deploy-capacity.mjs (new) + CI workflow (deploy-capacity-guard.yml) — four mechanical rules across all six deploy/*.yaml manifests: every service declares maxScale; containerConcurrency never exceeds resolved Puma threads; every worker's pool covers its own environment's Sidekiq concurrency; and Σ(pool × maxScale) + reserves stays within each Cloud SQL instance's declared ceiling. 20 passing test cases, including two live regression cases.
  • scripts/check-ai-settings-parity.mjs (new) + a warn-level log on AI's AI_DISABLED rejection path (rails_api/app/services/ai/settings_guard.rb) — a scheduled drift guard asserting the four ai.*_enabled flags agree across every settings file, plus closing a gap where a rejected AI mutation previously logged nothing at all.
  • docs/operations/rails-concurrency-capacity.md (new) — the operations reference for this whole model: how the two env vars resolve, how the budget file works, what the CI guard checks, and a "Known limitation" section stating plainly what is still under-provisioned and how to clear it.
  • New docs/development/gotchas.md entry, docs/operations/observability.md L6 coverage, and a CHANGELOG.md entry.

Phases ​

PhaseNameStatusPlansHighlights
1Decouple Puma threads from the ActiveRecord poolShipped1Pure enabler, zero behavior change — confirmed live via no manifest setting either var through this phase.
2Declare the connection budget and fix worker pool starvationShipped1Committed capacity-budget.json; found and fixed a live worker pool-starvation bug outside the original diagnosis.
3Retune staging API capacity and prove the fixShipped2New l6_ai_contention.js load test proved the fix — read p50 1.81s poisoned → 57ms fixed — before its measurement window closed permanently mid-phase.
4Retune production API capacityShipped1Sized production to the committed budget; honestly disclosed that the budget itself is now below the re-measured real peak.
5CI capacity guard and documentationShipped3Mechanical guard against drift reappearing, plus the full operations write-up including the known limitation.

Key Decisions ​

  • Decouple the env var, keep the shared fallback rung. PUMA_MAX_THREADS/DB_POOL_SIZE each resolve independently but still fall back to RAILS_MAX_THREADS before their own hardcoded default — dropping that rung would make a future RAILS_MAX_THREADS=8 silently do nothing, a worse failure than the coupling being fixed.
  • deploy/capacity-budget.json as a committed artifact, not manifest-comment prose. Every later phase sizes against it and the CI guard reads it programmatically — the single mechanism that makes "the numbers add up" checkable rather than asserted.
  • Lower containerConcurrency first, size pools second. Cloud Run only autoscales at ~60% of containerConcurrency, so an oversized value defeats the platform's own scale-out trigger independent of any connection-budget question — this is the acute, zero-connection-cost fix, and it shipped in Phase 3/4 ahead of any pool-sizing debate.
  • CAP-16 was explicitly waived, not silently met. Re-measuring production's real connection draw (Cloud Monitoring num_backends, 7-day window) found 19-20 peak connections, not the stale 15 the original diagnosis assumed — and against a 25-connection ceiling with 22 already committed to reserves, worker, and crono, no arrangement of the remaining 3 connections can cover that peak. The manifest ships correctly sized to the budget; the budget itself needs a human-run tier bump to actually clear observed peak. This is recorded as a deliberate, disclosed availability regression, not an oversight.
  • Reclaiming worker connections (option c) was evaluated and rejected. The worker absorbed all seven AI-coaching mutations during the concurrent v4.60 Async AI Delivery Path cutover, so cutting its pool to fund API capacity would trade away the exact throughput this milestone exists to protect.
  • L6 is parked, not deleted, once its measurement window closed. v4.60's async cutover finished moving every synchronous AI mutation off the request thread 52 seconds after L6's own after-run — the contention it measures can never be reproduced again on this codebase. The script stays as permanent evidence and a template for a future contention test, gated behind an opt-in run_l6 dispatch input rather than deleted or left permanently red.
  • SC-7 stays open at milestone close, deliberately. The 48-hour Sentry watch on OBJECTUVE-API-2Q opened at the production deploy and runs to ~2026-09-18T12:30Z — after this milestone's close. Holding an entire milestone open for a monitoring window was judged worse than closing with one criterion honestly marked open and handed to Vicki's standing monitoring.

Requirements Coverage ​

35 / 37 requirements satisfied (quoted from MILESTONE-AUDIT.md) — audit status passed-with-caveat. The two outstanding: CAP-16 (production sized above observed peak) is partially met and explicitly waived, not a gap discovered at audit; CAP-24 (the Sentry watch, tracking success criterion SC-7) is open by design, a 48-hour monitoring window that had not finished running as of milestone close.

CategoryCountStatus
CAP-* (capacity/config)2624 satisfied, 1 waived (CAP-16), 1 open by design (CAP-24)
GUARD-* (CI guard)4All satisfied
DOC-* (documentation)5All satisfied
SC-* (milestone success criteria)76 satisfied, 1 open by design (SC-7, tracks CAP-24)

Full requirements list: v4.59-backend-concurrency-remediation-ROADMAP.md on GitHub.

Outcomes ​

Capacity is now declared, bounded, sized against a committed budget file, and mechanically guarded — a manifest change that lets containerConcurrency exceed real Puma capacity, or lets a Cloud SQL instance's committed connections exceed its ceiling, now fails CI rather than shipping silently, the same way this incident's root cause did. Staging is proven fixed by a real before/after load test. Production is correctly sized to its committed budget, but that budget is itself below the re-measured real peak — production's API is capped at 3 concurrent DB-backed requests against a measured 19-20 peak, disclosed in docs/operations/rails-concurrency-capacity.md's "Known limitation" section, and clears only when a human runs gcloud sql instances patch enkidu-postgres --tier=db-g1-small (~+$15-20/month). That command has been with Josh since 2026-09-16 and is not yet applied. The 48-hour Sentry watch confirming OBJECTUVE-API-2Q stops recurring is still running as of this milestone's close and rides Vicki's standing monitoring, not a milestone gate.

Tech Debt ​

  • (Phase 3) l6_ai_contention.js can no longer reproduce the contention it was built to measure. The concurrent v4.60 milestone moved every synchronous AI mutation off the request thread 52 seconds after L6's own after-run captured its proof. Parked behind an opt-in run_l6 dispatch input rather than deleted, since the before/after evidence and the script itself remain useful precedent for a future contention test.
  • (Phase 4) Production's capacity ceiling is below its measured peak. See Outcomes above — a disclosed, deliberate availability regression pending a human-run Cloud SQL tier bump.
  • (Phase 3, carried into Phase 4) The worker's DB_POOL_SIZE: 12 / maxScale: 1 may now be under-sized for the AI-mutation workload it absorbed post-v4.60 cutover — flagged as a finding to track, explicitly not a scope expansion for this milestone.
  • d765a85f9 — Phase 1: decouple Puma thread count from AR pool size (#3165)
  • d6ee6ce50 — Phase 2: declare per-database connection budget, fix worker pool starvation (#3184)
  • a63c31694 — Phase 3: retune staging API capacity + AI-contention load test (#3215)
  • ff9ed1387 — Phase 3: park L6, fix its read threshold to p(50) (#3282)
  • a76cc2694 — Phase 4: size production API Cloud Run capacity (#3290)
  • 74903660 — Phase 5: add deploy-capacity-guard CI check (#3293)
  • 70472795 — Phase 5: rails concurrency & capacity model, gotcha, and observability updates (#3295)
  • 2aef3696f — Phase 5: log warn on AI_DISABLED rejection path + settings parity guard (#3296)

Last updated: 2026-09-17

Loading…