Rails concurrency & capacity

How Objectuve sizes Puma threads, the ActiveRecord connection pool, and Cloud Run's own scaling knobs against the two Cloud SQL instances backing them — and the CI guard that keeps the four sized quantities from drifting apart again (v4.59 Phase 5, OBJ-3753).

For the incident that made this a documented model instead of tribal knowledge, see the shared env-var gotcha in docs/development/gotchas.md.

The knobs

Four values, spread across two files per service, decide how many concurrent DB-backed requests a Cloud Run service can actually serve without queuing or exhausting Postgres:

Knob	Where it's set	What it controls
PUMA_MAX_THREADS	Container env var, read by rails_api/config/puma.rb:30	How many request threads one Puma worker process runs
DB_POOL_SIZE	Container env var, read by rails_api/config/database.yml:22	How many ActiveRecord connections one process may hold open
containerConcurrency	deploy/*.yaml, spec.template.spec.containerConcurrency	How many concurrent requests Cloud Run routes to one container instance before starting another
autoscaling.knative.dev/maxScale	deploy/*.yaml annotation	The hard ceiling on how many instances of a service Cloud Run will ever run at once

The first two are per-process; the last two are per-Cloud-Run-service. All four have to agree with each other, and the per-service pair has to agree with the shared Cloud SQL instance's connection ceiling — that's what deploy/capacity-budget.json and the CI guard both check.

How the fallback chain resolves

Both Puma and ActiveRecord resolve their sizing through the same two-step fallback, but they don't share a final default:

# rails_api/config/puma.rb:30
threads_count = ENV.fetch('PUMA_MAX_THREADS') { ENV.fetch('RAILS_MAX_THREADS', 3) }

# rails_api/config/database.yml:22
pool: <%= ENV.fetch("DB_POOL_SIZE") { ENV.fetch("RAILS_MAX_THREADS") { 5 } } %>

Both check their own specific var first (PUMA_MAX_THREADS / DB_POOL_SIZE), then fall back to the shared RAILS_MAX_THREADS, then land on a hardcoded default that differs between the two files — 3 for Puma, 5 for ActiveRecord. If nothing is ever set, threads and pool size don't just fail to match by accident of resolution order — the two files were never designed to converge on the same number even in the unset case. See the gotchas entry for the historical failure this produced.

In practice, this doesn't fire today: every deploy/*.yaml manifest declares DB_POOL_SIZE and PUMA_MAX_THREADS explicitly (v4.59 Phases 3–4), so the fallback chain never actually resolves through RAILS_MAX_THREADS or the hardcoded defaults in a deployed environment. It's live wherever a manifest doesn't declare both — local dev, a new environment, or a future service manifest added without following this doc — which is exactly why the guard (below) resolves the fallback chain itself rather than skipping an unset var.

How to read deploy/capacity-budget.json

One entry per Cloud SQL instance (enkidu-postgres for production, enkidu-postgres-staging for staging), each with:

• ceiling.max_connections — the instance's real max_connections, confirmed live via gcloud sql instances describe (not assumed from the tier default).
• reserves — connections nothing here can claim: superuser, cloudsqladmin, migrate_job.
• consumers — one entry per Cloud Run service that connects to this instance, each carrying its manifest path, db_pool_size, max_scale, and the resulting worst-case connections (db_pool_size × max_scale).
• arithmetic — sized_connections (sum of every consumer's worst case) + reserved_connections, against the ceiling, giving available_for_unsized_consumers.

This file is the single source of truth the guard checks against — not the deploy manifests' comments, which restate the same arithmetic in prose for human readers but can drift from the JSON if only one of the two is updated.

How to change capacity safely

1. Update deploy/capacity-budget.json first. Recompute the target instance's arithmetic block with the new numbers before touching any manifest — this is what the guard reads, and it's the step most likely to get skipped under time pressure.
2. Edit the target service's deploy/*.yaml. Keep DB_POOL_SIZE and PUMA_MAX_THREADS sized to the container's real concurrency model, not just copied from a sibling service:
   • A web-only container (service.*.yaml) runs Puma only, so pool and threads are sized 1:1 — e.g. deploy/service.production.yaml:130-133 sets both to 1.
   • A container running Sidekiq and Puma sharing one ActiveRecord pool (worker.*.yaml) needs the pool sized to cover both — deploy/worker.production.yaml:109-112 sets DB_POOL_SIZE: 12 for a Sidekiq concurrency of 10 (rails_api/config/sidekiq.yml:30, the production: override, not the top-level default of 5 at sidekiq.yml:7) plus 2 for Puma.
3. Set containerConcurrency to that same resolved PUMA_MAX_THREADS. Leaving it at Cloud Run's default (80) against a real thread count in the low single digits is the containerConcurrency-vs-threads mismatch the gotchas entry documents — Cloud Run's autoscaler won't add an instance until requests are already queued behind the one thread actually available.
4. Re-check the instance ceiling. Σ(db_pool_size × maxScale) across every consumer of that instance, plus reserves, must stay at or under ceiling.max_connections — update the arithmetic block in deploy/capacity-budget.json in the same commit as the manifest change (step 1, restated as a checkpoint).
5. Run the guard locally before pushing: node scripts/check-deploy-capacity.mjs — fails on any of the four rules below, naming the file, both numbers, and which rule broke.
6. CI re-checks it on push. .github/workflows/deploy-capacity-guard.yml runs the same script on PR and on push to master. It is not in master CI gate's required contexts — promoting a context to required is a repo-admin decision, same precedent as milestone-close-ordering-guard.yml (see Planning Canon) — so a red run today is a strong signal to act on, not a merge block.

What the guard enforces

scripts/check-deploy-capacity.mjs (Phase 5a, OBJ-3914) reads deploy/capacity-budget.json and every deploy/*.yaml, and fails on:

1. Any Cloud Run service manifest with no declared maxScale. An unbounded ceiling defeats the whole budget — every consumer in capacity-budget.json has to have a real worst-case connection count.
2. Any service whose containerConcurrency exceeds its resolved PUMA_MAX_THREADS — resolving the same fallback chain documented above, so a manifest that omits the var is checked against its real default, not skipped.
3. Any worker service whose DB_POOL_SIZE is below its sidekiq.yml concurrency — read from the per-environment override (production:/staging:), not just the top-level default, since a check against only the top-level 5 would miss sidekiq.yml's production: 10 override.
4. Any Cloud SQL instance where Σ(pool × maxScale) + reserves exceeds its ceiling — compared with <=, not <: both instances sit at exactly zero margin today (see Known limitation below), so a strict < would fail cleanly-provisioned trees.

Each failure names the manifest file, the two numbers that disagree, and which of the four rules broke — a bare "capacity check failed" wasn't considered actionable enough to ship.

This guard is not in master CI gate's required contexts on merge. Promoting a context to required is a repo-admin decision, not something this guard's own workflow file can grant itself — its header comment says so explicitly rather than implying it blocks merges today.

Known limitation

A green capacity guard means every declared number is internally consistent — it does not mean the service is adequately provisioned.

• Production's API (enkidu-api-production) is capped at 3 concurrent DB-backed requests (DB_POOL_SIZE: 1 × maxScale: 3, deploy/service.production.yaml:58,130-133; deploy/capacity-budget.json's enkidu-postgres.consumers entry for this service).
• The measured peak over the same window is 19–20 connections (Cloud Monitoring num_backends, 7-day trailing window, measured 2026-09-16 — see deploy/capacity-budget.json's enkidu-postgres.consumers[enkidu-api-production].note) — roughly 6–7× the capped concurrency above.
• This is deliberate and disclosed, not an oversight: enkidu-postgres's db-f1-micro tier caps max_connections at 25, and reserves (8) plus the worker (12) plus crono (2) already consume 22 of those 25 — 3 is everything left for the API regardless of how it's arranged between the three consumers. The only way to raise it is to raise the ceiling: gcloud sql instances patch enkidu-postgres --tier=db-g1-small, ≈+$15–20/month — already disclosed as the recommended, not-yet-applied tier bump in GCP Cost Analysis § Cloud SQL.
• A green check-deploy-capacity.mjs run does not mean this is fixed. The guard checks that the declared numbers are internally consistent with each other and with the instance ceiling — it has no notion of "adequately provisioned against observed peak," and passes clean today with the API capped 6–7× below what was actually measured. Raising the ceiling (the tier bump above) is the only lever that closes this gap; sizing the manifests differently cannot, since reserves + worker + crono alone already model to 22 of the 25-connection budget.

Known limitation: deploy-time revision overlap

A second, independent gap on the same two instances: a deploy-time Cloud Run revision overlap (the old revision still holding its connection pool while the new revision boots and opens its own) can spike well past either instance's ceiling, and neither check-deploy-capacity.mjs's rules 1–4 nor the reserves block in deploy/capacity-budget.json accounts for it — see the reserve-conflation gotcha for why the reserve doesn't absorb this.

OBJ-3957 added an advisory rule 5 to the guard (overlap_draw(S) = reserves + Σ_{other consumers}(pool×maxScale) + 2×(redeploying consumer's pool×maxScale), maxed over every consumer S) to print, not fail on, this exposure — every instance already fails it today by design, so a blocking rule would red the tree on day one. Reproduce with node scripts/check-deploy-capacity.mjs. These are worst-case upper bounds — old and new revisions rarely both sit at full maxScale simultaneously, since Cloud Run drains the old revision as traffic shifts — not a measured, sustained draw:

Instance	Worst-case redeploying consumer	Overlap draw	Ceiling
enkidu-postgres (production)	enkidu-worker-production	37	25
enkidu-postgres-staging	enkidu-api-staging	33	25

Production's 37/25 is the more serious of the two instances — it exceeds its ceiling by 12 connections against staging's 8, and it's the instance already running 6–7× over its steady-state cap per the API-sizing limitation above. As with that limitation, closing this gap needs a capacity decision (raising maxScale, pool sizes, or the Cloud SQL tier) — tracked as open for Josh on OBJ-3956, not something either the guard or this doc's sizing guidance resolves on its own.

---

Last updated: 2026-09-18 (v4.9.0, OBJ-3956/OBJ-3957 — documented the deploy-time revision-overlap limitation)