GCP Cost Analysis — August 2026
Cost analysis of billing account 01E02D-214CFC-4CC425 / project enkidu-488723 (the only project attached to that billing account), covering the period 2026-08-01 → 2026-08-31, plus ranked optimization recommendations. Triggered by monthly GCP spend hovering around ~$620/mo. Region is us-central1 (Iowa) throughout. This doc also tracks v4.54 (GCP Cost Remediation)'s disposition on each recommendation — see Status and Ranked Optimizations.
Status
This is a measured August 2026 snapshot, not a live read — treat every figure below as dated unless a table row below marks it as separately re-verified. An earlier draft of this analysis asserted an orphaned e2-small VM worth $15–20/mo that had, in fact, already been deleted five and a half weeks before the measurement window opened (ac7963eca) — that misread is exactly the failure mode this section exists to prevent from recurring.
| Figure | Measured / re-verified | Status |
|---|---|---|
| Cloud Run config (8 services), duty cycle, egress totals, Memorystore, Compute Engine | 2026-08-01 → 2026-08-31; re-verified live 2026-09-07 (Phase 1a, OBJ-3465) | Confirmed against live GCP — no drift beyond expected growth |
| Artifact Registry versions/size, repo resource name | Original Aug figures were falsified (an entire package, litellm-proxy, was silently omitted); corrected 2026-09-07 (Phase 1a) | Corrected — see Artifact Registry |
| Crono egress destination mechanism | Original Aug read was wrong (claimed internal/regional traffic); corrected 2026-09-08 (Phase 4, OBJ-3468) | Corrected — see Egress |
| Artifact Registry cleanup-policy dry-run baseline | 2026-09-09 (Phase 2, OBJ-3466) — a newer, separate apply-time snapshot, not a re-read of the Phase 1a figures above | New — see optimization #1 |
| Modeled total spend (≈$484.64/mo) | Derived from the Aug 2026 window | Still modeled for the Aug 2026 window itself; a separate measured run rate (≈$548.53/mo, from 7 complete UTC days of billing export, 2026-09-11 → 2026-09-17) now exists alongside it — Phase 6b.2b-i (OBJ-3936, 2026-09-18), see Reconciliation Gap |
As of this writing (2026-09-09, Phase 6a): items 1 (Artifact Registry cleanup), 3 (production minScale), 4 (Crono throttling mode), and 5 (Crono egress) have a recorded disposition below. Item 2 (staging scheduled idling) and item 6 (the reconciliation gap's billing-export closure) are held on Phases 3 and 6b.2b respectively and are not yet resolved — see their own sections for what's pending and why.
Updated 2026-09-18 (Phase 6b.2b-i, OBJ-3936): item 6 now has a partial disposition — the billing export cleared 7 complete UTC days and the Reconciliation Gap section below moved from plausible candidates to measured results for everything the export could reach. Item 2 (staging scheduled idling, Phase 3) is still pending, and the slice of item 6 that depends on it is held on Phase 6b.2b-ii.
Method & Limitations
- Quantities are measured, not estimated. Cloud Run figures come from Cloud Monitoring v3
timeSeriesreads ofrun.googleapis.com/container/billable_instance_time,run.googleapis.com/container/cpu/allocation_time,run.googleapis.com/container/memory/allocation_time,run.googleapis.com/container/network/sent_bytes_count, andrun.googleapis.com/job/completed_task_attempt_count, over 2026-08-01 → 2026-08-31.cpu/allocation_timeandmemory/allocation_timeare the billed quantities (vCPU-seconds and GiB-seconds respectively) — no container-count modeling was needed to get from "what ran" to "what's billed." Artifact Registry figures (sizeBytes,cleanupPolicies) come from the Artifact Registry v1 REST API directly — also authoritative, not inferred. - Dollar figures are derived from Cloud Billing Catalog API list prices (us-central1, USD) — they are not read off an invoice. Every dollar figure below is
measured quantity × catalog rate, with both factors shown, so it's reproducible. Subtotals and totals are computed from the unrounded values and rounded once at the end, so a column of displayed cells can sum to a cent or two away from the subtotal printed beside it. - A BigQuery billing export was enabled by hand on 2026-09-11 (see Billing Export below for dataset/table/coverage detail). Enabling it was a Cloud Console action, and only a Cloud Console action —
gcloud billingexposes onlyaccounts,budgets, andprojects, and the Cloud Billing API has no export-configuration resource, so nothing in this repo could script that step. It no longer blocks per-SKU historical cost from being queried programmatically. Updated 2026-09-18 (Phase 6b.2b-i, OBJ-3936): the export now holds 7 complete UTC days and has partially closed the reconciliation gap below — see Reconciliation Gap. - The modeled total below is ≈$484.64/mo against the stated ~$620/mo, leaving a ~$135 residual. That gap is stated honestly in the Reconciliation Gap section — the numbers here are not reverse-fit to land on $620.
- Methodology trap for anyone re-running these queries: a single 30-day-wide
alignmentPeriod(2592000s) on a DELTA metric silently over-reports — reproduced on the crono egress figures below as 20.55 / 25.68 GiB against the true 18.32 / 21.71 GiB.ALIGN_SUMdoesn't safely span periods that wide for these metrics. Use hourly alignment (3600s) and sum in code instead; that's what every figure in this doc actually uses, and it's what reconciles exactly.
Billing Export
A BigQuery billing export was enabled by hand on 2026-09-11 — dataset billing_export, project enkidu-488723, location us-central1 (must match the Cloud Run region: a mismatched dataset location can't be joined against regional data later without a copy). Both the standard and detailed usage cost exports were enabled, landing in:
gcp_billing_export_v1_01E02D_214CFC_4CC425(standard usage cost)gcp_billing_export_resource_v1_01E02D_214CFC_4CC425(detailed usage cost)
This step could not be scripted — gcloud billing exposes only accounts, budgets, and projects, and the Cloud Billing API has no export-configuration resource. The reproducible remainder (dataset/IAM assert-or-apply, a verification query) lives in gcp_setup/19_billing_export.sh; its header is the source of truth for every figure in this section.
For the operational how-to — re-provisioning, the regional-vs-multi-region decision and why it stands, sourced backfill thresholds, and the row-settling querying hazard — see the Billing Export Runbook. This section stays scoped to the cost analysis those mechanics produce.
Coverage
First data: 2026-09-11 (enablement). The export is not backfilled — it only carries data from enablement forward, never retroactive history.
As of 2026-09-18, 7 complete UTC days exist: 2026-09-11 → 2026-09-17. Verified 2026-09-18 (Phase 6b.2b-i, OBJ-3936) directly against both export tables — they agree exactly:
UTC day hours covered cost 2026-09-11 24 $12.16 2026-09-12 24 $19.40 2026-09-13 24 $17.24 2026-09-14 24 $18.50 2026-09-15 24 $24.03 2026-09-16 24 $18.17 2026-09-17 24 $16.79 2026-09-18 (today, at query time) is truncated at 11 hours — the latest available day is always truncated at its end by export lag; that part of the original claim still holds.
Correction to an earlier claim in this section: a version written 2026-09-13 said no complete UTC day existed yet, and that 2026-09-11 was truncated at its start (first usage row 05:00Z, enablement 16:09Z). That was true on 2026-09-13 — billing-export rows settle over 1–3 days, and 2026-09-11 has since backfilled to a full 24-hour day, as the table above shows. The "earliest 2026-09-20" estimate for reaching 7 complete days (in Querying the export below) was likewise superseded — the gate was actually met on 2026-09-18, two days early.
The 2026-09-09 and 2026-09-10 rows (3 and 10 rows respectively, $0.00) are zero-cost SKU placeholders, not a backfill — don't misread them as retroactive data landing before enablement.
Querying the export
The export is not backfilled, so a query scoped to "last month" returns zero rows against a correctly configured export — and would trip 19_billing_export.sh's own zero-rows guard if run that way. Scope any query to enablement forward instead. Worked per-SKU example (from gcp_setup/19_billing_export.sh's header):
SELECT
service.description AS service,
sku.description AS sku,
ROUND(SUM(cost), 2) AS total_cost
FROM `enkidu-488723.billing_export.gcp_billing_export_v1_01E02D_214CFC_4CC425`
WHERE DATE(usage_start_time) >= '2026-09-11'
GROUP BY service, sku
ORDER BY total_cost DESC
LIMIT 20Measured 2026-09-18 (Phase 6b.2b-i, OBJ-3936), 7 complete UTC days (2026-09-11 → 2026-09-17, see Coverage above) — re-run against gcp_billing_export_resource_v1_01E02D_214CFC_4CC425 and cross-checked against gcp_billing_export_v1_01E02D_214CFC_4CC425 (both tables agree on every daily total):
| Service | 7-day cost | Monthly run rate |
|---|---|---|
| Cloud Run | $82.86 | $359.87 |
| Firebase Hosting | $18.95 | $82.29 |
| Cloud Memorystore for Redis | $15.95 | $69.25 |
| Cloud SQL | $4.44 | $19.29 |
| Artifact Registry | $3.43 | $14.90 |
| Secret Manager | $0.66 | $2.88 |
| Compute Engine | $0.01 | $0.05 |
| Total | $126.31 | $548.53 |
Monthly run rate = 7-day cost ÷ 7 × 30.4 — a projection, not an invoice figure; the export doesn't yet hold a full calendar month. What this changes about the Reconciliation Gap: see Reconciliation Gap. How it compares to the modeled total: see Modeled Total.
Inventory
Cloud Run Services (8)
Live configuration, read from the running services (gcloud run services list), not from the repo's manifests (which don't cover LiteLLM — see note below). Re-verified live 2026-09-07 (Phase 1a, OBJ-3465) — all 8 rows confirmed, no drift.
| Service | CPU | Memory | minScale | maxScale | cpu-throttling |
|---|---|---|---|---|---|
| enkidu-api-production | 1000m | 512Mi | 1 | (none) | default true (request-based) |
| enkidu-api-staging | 1000m | 512Mi | 1 | 4 | default true (request-based) |
| enkidu-crono-production | 1 | 768Mi | 1 | 1 | false (instance-based) |
| enkidu-crono-staging | 1 | 768Mi | 1 | 1 | false (instance-based) |
| enkidu-litellm-production | 1 | 1Gi | 1 | 20 | default true (request-based) |
| enkidu-litellm-staging | 1 | 1Gi | (none) | 5 | default true (request-based) |
| enkidu-worker-production | 1000m + 1000m (2 containers) | 1024Mi + 768Mi | 1 | 1 | false (instance-based) |
| enkidu-worker-staging | 1000m + 1000m (2 containers) | 1024Mi + 768Mi | 1 | 1 | false (instance-based) |
The LiteLLM resource shape (CPU/memory/scale) is not declared anywhere in the repo — infra/litellm/ holds only Dockerfile and config YAMLs; the shape is set imperatively by the deploy-litellm workflow. The table above is the live config, not a manifest read.
Cloud Run Jobs (21)
~5,800 total task attempts in August. See Job Attempts below for the breakdown; dominated by preview-environment lifecycle jobs (bootstrap/teardown/ensure smoke accounts).
Cloud SQL (1 instance)
enkidu-postgres: db-f1-micro, 10 GB PD_SSD, zonal (no HA/replica), backups enabled, PITR enabled.
v4.59 Phase 4 (OBJ-3752, 2026-09-16) tier-bump disclosure. db-f1-micro's max_connections=25 (confirmed live via gcloud sql instances describe, an explicit flag, not the tier default falling through unset) leaves production's API service only 3 connections of headroom against deploy/capacity-budget.json's modeled worst case — below the 19-20 connection peak Cloud Monitoring actually measured over the trailing 7 days. Raising the ceiling is the only lever that clears this without cutting the worker's AI-coaching throughput; the recommended path is a tier bump, db-f1-micro → db-g1-small, ≈+$15–20/month (list price delta on top of the ~$10.51/mo Cloud SQL line above). This is a recommendation handed back to a human via OBJ-3752's PR — it has not been applied. This is disclosed here per this doc's own standard (see Status) precisely because v4.54 (this doc) was the cost-remediation effort a tier bump would otherwise look like it's quietly reversing.
Memorystore for Redis (2 instances)
Both Basic tier, 1 GB, REDIS_7_0, READY — re-verified live 2026-09-07, no drift:
| Instance | IP | Environment |
|---|---|---|
enkidu-redis | 10.234.96.43 | production |
enkidu-redis-staging | 10.79.34.43 | staging |
Both are legitimately wired — grep -n -A1 REDIS_URL deploy/*.yaml shows all three staging manifests point at 10.79.34.43 and all three production manifests point at 10.234.96.43, with no cross-environment leak. Neither instance is a dead cost. The theory that staging Redis is unused has been checked and ruled out, not assumed — see Reconciliation Gap.
Artifact Registry (1 repo)
us-central1-docker.pkg.dev/enkidu-488723/enkidu-registry — format DOCKER, mode STANDARD_REPOSITORY, created 2026-02-28.
Corrected 2026-09-07 (Phase 1a, OBJ-3465) — the original figures were falsified, not just stale. Live: sizeBytes: 480381593904 (447.39 GiB) across 7,757 image versions in three packages — rails-api 2,713 + agent-runner 2,373 + litellm-proxy 2,671. The original doc's total (≈4,989 versions / 439.27 GiB) counted only rails-api (2,665) and agent-runner (2,324); it silently omitted litellm-proxy entirely, a package that on its own (2,671 versions) carries more versions than either package that was counted. cleanupPolicies: null / cleanupPolicyDryRun: null was confirmed still unset at re-verification time.
Also corrected: the repository's live resource name is enkidu-registry, not enkidu as the original doc stated — a script or doc reference using the old name would 404 against the real API.
As of 2026-09-09, a cleanup policy is live on this repository, pruning it today — see Artifact Registry cleanup policy — APPLIED below for what it keeps, what it prunes, and how to override it before a risky release.
Compute Engine (0 instances)
Nothing is running, and nothing was running during the window. gcp_setup/04_preview_db_vm.sh provisions an e2-small VM named enkidu-preview-db to run Postgres + Redis containers for per-PR ephemeral preview backends. That VM did exist — but it was deleted well before August opened, so its contribution to the costs analyzed here is $0. Verified live (re-confirmed 2026-09-07, Phase 1a — 0 instances, 0 disks, no change):
$ gcloud compute instances list --project=enkidu-488723
Listed 0 items.
$ gcloud compute disks list --project=enkidu-488723
Listed 0 items.Cloud Audit Logs carry the full lifecycle — created 2026-02-28T00:35:07Z, deleted 2026-07-17T19:16:25Z, both by josh@objectuve.com:
$ gcloud logging read \
'protoPayload.methodName=("v1.compute.instances.insert" OR "v1.compute.instances.delete")
AND protoPayload.resourceName:"enkidu-preview-db"' \
--project=enkidu-488723 --freshness=400d
2026-07-17T19:16:48Z v1.compute.instances.delete josh@objectuve.com
2026-07-17T19:16:25Z v1.compute.instances.delete josh@objectuve.com
2026-02-28T00:35:26Z v1.compute.instances.insert josh@objectuve.com
2026-02-28T00:35:07Z v1.compute.instances.insert josh@objectuve.comThat delete date also resolves the one confusing data point in the repo. The architecture the VM was built for was retired on 2026-04-05 (39415e33d, "Replace branch-based deploys with tag-based release flow") — the current deploy-preview job (.github/workflows/preview.yml:81-297) only pushes a Firebase Hosting channel pointed at the shared staging.api.objectuve.com, with no gcloud run deploy or gcloud compute call anywhere in the file. Yet .planning/milestones/v4.5-runtime-least-privilege-ROADMAP.md records a 2026-07-16 finding that the VM was "RUNNING and actively backs 10 live preview PRs." That sighting lines up with .github/workflows/ci.yml's lint-workflows guard comment, which documents "~20 orphaned enkidu-api-pr-*/enkidu-db-setup-pr-* Cloud Run resources that nothing ever cleaned up, discovered and cleaned up manually on 2026-07-17" — the same day as the audit log's delete. The "10 live preview PRs" were that leaked cohort, and the VM was decommissioned along with it.
Consequence for this analysis: Compute Engine is neither an unmeasured cost line nor an optimization candidate, and is explicitly out of scope for v4.54's remediation work — see Ruled out: Compute Engine under Ranked Optimizations.
Cloud Storage
| Bucket | Mean August size |
|---|---|
enkidu-mobile-builds | 0.184 GiB |
enkidu-storage-staging | 0.001 GiB |
enkidu-storage | 0.00003 GiB |
Four other known buckets round out the inventory. enkidu-gdpr-exports-production, enkidu-gdpr-exports-staging, and enkidu-storage-enkidu-488723 returned no time series at all in August (empty/unused). enkidu-load-baselines (created by gcp_setup/10_create_load_baselines_bucket.sh to store k6 load-test JSON summaries under a 24-baseline capped retention) was missed by the original inventory and wasn't independently re-measured here, but small capped-count JSON objects make it immaterial to the total regardless. GCS is a non-factor at effectively $0/mo, and is explicitly out of scope for v4.54's remediation work.
Cloud Logging
23.816 GiB ingested in August (logging.googleapis.com/billing/bytes_ingested), against a free allowance of the first 50 GiB/mo. $0/mo — see the negative finding below. Explicitly out of scope for v4.54's remediation work.
Measured Usage
Instance-Hours
Metric: run.googleapis.com/container/billable_instance_time.
| Service | Billable seconds | Instance-hours |
|---|---|---|
| enkidu-api-production | 2,677,348 | 743.7 |
| enkidu-api-staging | 2,691,381 | 747.6 |
| enkidu-crono-production | 2,676,345 | 743.4 |
| enkidu-crono-staging | 2,677,842 | 743.8 |
| enkidu-litellm-production | 2,677,216 | 743.7 |
| enkidu-litellm-staging | 65,313 | 18.1 |
| enkidu-worker-production | 2,676,266 | 743.4 |
| enkidu-worker-staging | 2,678,004 | 743.9 |
| TOTAL | 18,819,715 | 5,227.7 |
744 h = 31 days × 24 h exactly. 7 of 8 services ran one instance 24/7 for the entire month.enkidu-litellm-staging is the only service that scales to zero, at 18.1 h / 744 h = 2.4% utilization — the in-estate proof that scale-to-zero is viable on this stack (see optimization #2).
Billed Quantities & Duty Cycle
Metrics: run.googleapis.com/container/cpu/allocation_time (vCPU-seconds) and run.googleapis.com/container/memory/allocation_time (GiB-seconds). These are the billed quantities, not a proxy for them.
| Service | vCPU-seconds | GiB-seconds | CPU duty cycle |
|---|---|---|---|
| enkidu-api-production | 28,166 | 1,337,467 | 1.05% |
| enkidu-api-staging | 217,148 | 1,342,550 | 8.1% |
| enkidu-crono-production | 2,676,459 | 2,005,189 | 100% (instance-based) |
| enkidu-crono-staging | 2,685,868 | 2,004,249 | 100% (instance-based) |
| enkidu-litellm-production | 9,787 | 2,674,530 | 0.37% |
| enkidu-litellm-staging | 79,833 | 61,518 | n/a (scale-to-zero) |
| enkidu-worker-production | 5,351,876 | 4,678,929 | 100% (instance-based, 2 vCPU) |
| enkidu-worker-staging | 5,371,036 | 4,676,807 | 100% (instance-based, 2 vCPU) |
Duty cycle = cpu_allocation_seconds / billable_instance_seconds / vCPU-per-instance. These numbers cross-validate the live manifests exactly: worker CPU ≈ 2× instance-seconds (2 vCPU across its 2 containers); worker memory ≈ 1.75 GiB × instance-seconds (1024Mi + 768Mi); api memory ≈ 0.5 GiB × instance-seconds; litellm-production memory ≈ 1 GiB × instance-seconds.
Re-verified live 2026-09-07 (Phase 1a): enkidu-api-production and enkidu-litellm-production duty cycles matched the Aug-2026 figures above exactly (1.052% / 0.366%). A second Aug 1 → now (2026-09-07) window showed both duty cycles had grown since August (1.349% / 0.634%) with traffic increase — still solidly "the standing instance is the cost, not the traffic" territory; this doesn't change item 3's verdict below.
Egress
Metric: run.googleapis.com/container/network/sent_bytes_count.
| Service | GiB |
|---|---|
| enkidu-api-production | 1.45 |
| enkidu-api-staging | 11.95 |
| enkidu-crono-production | 18.32 |
| enkidu-crono-staging | 21.71 |
| enkidu-litellm-production | 0.00 |
| enkidu-litellm-staging | 0.02 |
| enkidu-worker-production | 2.73 |
| enkidu-worker-staging | 2.42 |
| TOTAL | 58.62 |
Re-verified live 2026-09-07 (Phase 1a) — every row and the total confirmed exactly against the Aug 2026 window.
enkidu-crono-production + enkidu-crono-staging alone account for 40.0 GiB — 68% of all Cloud Run egress — from two ingress: internal schedulers that serve no public traffic at all (see optimization #5).
Corrected 2026-09-08 (Phase 4, OBJ-3468) — the original "mostly internal/regional" read was wrong, not just imprecise. Cloud Monitoring's own metric.labels.kind destination classification (google / private / internet) shows >99.8% of both crono services' egress is genuine internet-bound traffic: enkidu-crono-production 18.2828 of 18.3218 GiB (99.78%), enkidu-crono-staging 21.6943 of 21.7149 GiB (99.91%). True private (VPC-internal) traffic is under 20 MiB/service for the whole month. The dollar conclusion survives, on different grounds: it's a few dollars total not because most of the traffic stays inside GCP and is free, but because ~40 GiB/month of genuine internet egress is still cheap at Cloud Run's Premium Tier internet-egress rate (~$0.12/GiB, ~$4.80/mo combined: $2.19 production + $2.60 staging). That's a meaningfully different claim — this line scales with traffic and stays a live cost signal as usage grows, rather than implying there's nothing to watch. Root cause traced to a code-level defect, not scheduler traffic shape — see optimization #5.
Job Attempts
Metric: run.googleapis.com/job/completed_task_attempt_count. ~5,800 task attempts across 21 jobs in August. Top jobs by attempt count:
| Job | Attempts |
|---|---|
| enkidu-teardown-preview-smoke-account | 1,703 |
| enkidu-bootstrap-preview-smoke-account | 1,032 |
| enkidu-db-migrate-staging | 788 |
| enkidu-demo-reset-staging | 679 |
| enkidu-seed-templates-staging | 678 |
| enkidu-ensure-preview-smoke-account-staging | 675 |
| enkidu-activity-cleanup-production | 48 |
| enkidu-activity-cleanup-staging | 42 |
Plus a handful of one-off probe/debug jobs (obj-pr-probe, obj1711-probe, obj2480-dupe-check, obj2826-live-fire, obj2852-clear-stale-pr, enkidu-restore5-count, enkidu-flag-check-staging). Job attempts are dominated by preview-environment lifecycle, not production workload. Modeled cost: ≈$1.40/mo (Jobs CPU $0.000018/vCPU-s + Jobs Memory $0.000002/GiB-s against these short-lived task attempts) — negligible against the totals below, and called out separately rather than folded into the modeled total (see Modeled Total).
Per-Service Cost Model
Unit Prices (Cloud Billing Catalog API, us-central1, USD, list price)
| SKU | Rate |
|---|---|
| Cloud Run Services CPU (Instance-based) | $0.000018 / vCPU-second |
| Cloud Run Services Memory (Instance-based) | $0.000002 / GiB-second |
| Cloud Run Services CPU (Request-based) | $0.000024 / vCPU-second |
| Cloud Run Services Memory (Request-based) | $0.0000025 / GiB-second |
| Cloud Run Services Min Instance CPU | $0.0000025 / vCPU-second |
| Cloud Run Services Min Instance Memory | $0.0000025 / GiB-second |
| Cloud Run Jobs CPU | $0.000018 / vCPU-second |
| Cloud Run Jobs Memory | $0.000002 / GiB-second |
| Memorystore Redis Basic M1 (Iowa) | $0.049 / GiB-hour |
| Artifact Registry storage | $0 for first 0.5 GiB, then $0.10 / GiB-month |
| Cloud SQL PostgreSQL Zonal Micro (Iowa) | $0.0105 / hour |
| Cloud SQL PostgreSQL Zonal Standard storage | $0.17 / GiB-month |
| Cloud SQL PD Snapshot (backups) | $0.08 / GiB-month |
| Cloud Logging | $0 for first 50 GiB/mo, then $0.50 / GiB |
| Cloud Scheduler | 3 free jobs/project/month, then $0.10/job-month (published rate — not cross-checked against the live Billing Catalog API the way every other rate in this table was; see optimization #4) |
Billing service IDs resolved directly (not guessed): 152E-C115-5142 Cloud Run — this is the only service the user's console cost-breakdown URL was filtered to; 149C-F9EC-3994 Artifact Registry; 5AF5-2C11-D467 Cloud Memorystore for Redis; 9662-B51E-5089 Cloud SQL; 5490-F7B7-8DF6 Cloud Logging; 95FF-2EF5-5EA1 Cloud Storage.
Instance-Based Services
Services running with cpu-throttling: false bill their full allocation for every billable second, regardless of request volume — CPU at the instance-based rate, memory at the instance-based rate:
| Service | CPU $ (vCPU-s × $0.000018) | Memory $ (GiB-s × $0.000002) | Total $ |
|---|---|---|---|
| enkidu-worker-production | 96.33 | 9.36 | 105.69 |
| enkidu-worker-staging | 96.68 | 9.35 | 106.03 |
| enkidu-crono-production | 48.18 | 4.01 | 52.19 |
| enkidu-crono-staging | 48.35 | 4.01 | 52.35 |
| Subtotal | 316.26 |
Request-Based Services
Services with default (request-based) throttling split their billable instance-seconds into active seconds (CPU actually consumed while handling a request, billed at the higher request-based CPU/memory rates) and idle seconds (the standing minScale instance idling between requests, billed at the lower Min Instance CPU/memory rates). Active seconds = cpu_allocation_seconds / vCPU-per-instance; idle seconds = billable_instance_seconds − active seconds:
| Service | Active s | Idle s | Total $ |
|---|---|---|---|
| enkidu-api-production | 28,166 | 2,649,182 | 10.65 |
| enkidu-api-staging | 217,148 | 2,474,233 | 14.76 |
| enkidu-litellm-production | 9,787 | 2,667,429 | 13.60 |
| enkidu-litellm-staging | all active (no minScale) | — | 2.07 |
| Subtotal | 41.07 |
Cloud Run services total: $316.26 + $41.07 ≈ $357.34/mo.
Non-Cloud-Run Lines
| Line | Calculation | $/mo |
|---|---|---|
| Memorystore for Redis (2 instances) | 2 × (1 GiB × 744 h × $0.049/GiB-hr) | 72.91 |
| Artifact Registry storage | (439.27 − 0.5 GiB) × $0.10/GiB-mo | 43.88 |
| Cloud SQL compute | 744 h × $0.0105/hr | 7.81 |
| Cloud SQL storage | 10 GiB × $0.17/GiB-mo | 1.70 |
| Cloud SQL backups/PITR | ~estimate | ~1.00 |
| Cloud Storage | — | ~0 |
| Cloud Logging | 23.816 GiB < 50 GiB free allowance | 0 |
Cloud SQL subtotal: ≈$10.51/mo (the least precise line in the model — backups/PITR is an estimate, not a directly-measured quantity). Artifact Registry storage above intentionally still uses the original August sizeBytes (439.27 GiB), not the Phase-1a-corrected 447.39 GiB — this $43.88/mo line is carried forward unchanged as a stated ceiling, not recomputed, because Artifact Registry's real dollar exposure is dominated by the layer-deduplication effect described in optimization #1, not by which sizeBytes snapshot feeds this formula. Read that section, not this row, for the current accounting.
Modeled Total
| Component | $/mo |
|---|---|
| Cloud Run services (instance-based + request-based) | 357.34 |
| Memorystore for Redis | 72.91 |
| Artifact Registry | 43.88 |
| Cloud SQL | ~10.51 |
| Cloud Storage | ~0 |
| Cloud Logging | 0 |
| Modeled total | ≈$484.64 |
Cloud Run Jobs (≈$1.40/mo, see Job Attempts) are real but are called out separately above rather than folded into this headline figure — they're two orders of magnitude smaller than the residual gap below and don't change the conclusion.
This is a modeled figure, not a measured one — derived from Cloud Billing Catalog list-price rates applied to live-queried Cloud Monitoring/Artifact Registry quantities, not read off an invoice.
Measured Total (billing-export run rate)
Measured 2026-09-18 (Phase 6b.2b-i, OBJ-3936), 7 complete UTC days (2026-09-11 → 2026-09-17) — see Querying the export for the query and per-day breakdown:
| Component | $/mo (measured run rate) |
|---|---|
| Cloud Run | 359.87 |
| Firebase Hosting | 82.29 |
| Memorystore for Redis | 69.25 |
| Cloud SQL | 19.29 |
| Artifact Registry | 14.90 |
| Secret Manager | 2.88 |
| Compute Engine | 0.05 |
| Measured total | ≈$548.53 |
This is a projection (7-day cost ÷ 7 × 30.4), not an invoice figure — the export doesn't yet hold a full calendar month, so read it alongside the ≈$484.64 modeled total above, not as a replacement for it. The two disagree mainly because Firebase Hosting, Secret Manager, and the full Cloud SQL cost weren't in the original model's scope — not because the modeled Cloud Run/Memorystore figures were wrong (both land within ~5% of measured, model-validated — see Reconciliation Gap below).
Reconciliation Gap
Modeled ≈$484.64/mo vs. stated ~$620/mo → residual ≈$135/mo.
Status: partially resolved with measured billing-export data (2026-09-18, Phase 6b.2b-i, OBJ-3936). The export now holds 7 complete UTC days (2026-09-11 → 2026-09-17, see Coverage), so what follows moves from plausible candidates to measured results everywhere the export can reach. The remainder that the export can't explain is stated plainly below, not forced closed.
Model validated — Cloud Run and Memorystore. Measured run rates land within ~5% of the modeled figures — this confirms the underlying Cloud Monitoring/catalog-rate model for these two services, not a coincidence to explain away:
| Service | Measured $/mo | Modeled $/mo | Delta |
|---|---|---|---|
| Cloud Run | 359.87 | 357.34 | +0.7% |
| Memorystore for Redis | 69.25 | 72.91 | −5.0% |
Verified — Firebase Hosting. This was the Reconciliation Gap's first-listed unverified candidate ("billed under a separate Firebase product line this analysis didn't measure") — it's now measured, 100% under the Outgoing Bandwidth SKU (Storage cost is $0 every day). Read it as a range, ≈$60–82/mo, not a single point figure: the full 7-day window gives $82.29/mo, but 2026-09-15 was a $7.12 outlier against a $1.21–3.05/day baseline on the other six days — excluding it, the six-day run rate is $59.94/mo. Root-causing the 2026-09-15 spike is open and out of scope for this task.
Newly measured — Cloud SQL understated, Secret Manager unmodeled. Cloud SQL measures at $19.29/mo against the model's ~$10.51/mo — an $8.78/mo understatement (the model's backups/PITR line was an estimate, not a measured quantity; see Non-Cloud-Run Lines). This measured actual is a useful input to the pre-existing v4.59 Cloud SQL tier-bump disclosure (see Cloud SQL above) without rewriting or relocating that section. Secret Manager, not modeled at all, measures at $2.88/mo.
Accounting so far: unmodeled Firebase Hosting ($82.29) + Secret Manager ($2.88) + the Cloud SQL understatement ($8.78) ≈ $93.95 of the ≈$135/mo residual, ~70%. The remaining ≈$41/mo stays unexplained — say so plainly, don't force it closed. Artifact Registry measures at $14.90/mo against the model's $43.88/mo (a −$28.98/mo delta) — that's Phase 2's cleanup-policy work landing (see Artifact Registry cleanup policy — APPLIED), not an unmodeled cost, so it isn't counted toward the residual above.
What could still explain the remaining ≈$41/mo (unverified — do not read either of these as confirmed):
- Egress beyond intra-region. Total measured Cloud Run egress is 58.62 GiB, almost entirely billable internet egress (see Egress above, corrected 2026-09-08) — at Cloud Run's ~$0.12/GiB Premium Tier rate this is a few dollars, not $41, even accounting for the correction.
- Support plan and/or other unmodeled SKUs not covered by the seven services the export measured (Cloud Run, Firebase Hosting, Memorystore, Cloud SQL, Artifact Registry, Secret Manager, Compute Engine) or by Cloud Storage / Cloud Logging.
What it is not (verified, ruled out — not assumed):
- Not Cloud Logging. 23.816 GiB ingested in August is well under the 50 GiB/mo free allowance — this line is $0, eliminated as an explanation.
- Not Compute Engine. The
enkidu-preview-dbVM was deleted on 2026-07-17 — five and a half weeks before this window opened — andgcloud compute instances listreturnsListed 0 items.today (re-confirmed 2026-09-07), with no disks left behind. See Compute Engine in the inventory. It billed $0 in August, and the billing export's own measured $0.05/mo run rate for it (see Querying the export) confirms it explains none of the gap. - Not the staging Redis "unused" theory. Both Memorystore instances are legitimately wired to their respective environments (verified against
deploy/*.yamlREDIS_URLvalues) — neither is dead spend, and the measured run rate above is within 5% of the model, not an anomaly.
Ranked Optimizations
Ordered biggest/cleanest-and-lowest-risk win first, not raw dollar size: #1 is a zero-runtime-risk cleanup ranked ahead of #2 despite #2's larger dollar figure, because #2 carries the outage-adjacent caveat described in its own section below. Every lever below is grounded in a specific repo file/line. Each numbered item below carries its v4.54 disposition — applied, declined, recommended but not implemented, investigated, or pending — recorded as a decision with its evidence, not just a proposal, so it isn't re-derived from scratch next quarter.
1. Artifact Registry cleanup policy — APPLIED
Disposition: APPLIED 2026-09-09 (PR #3017 @ 939c14c89, OBJ-3466). A cleanup policy is live on enkidu-registry today, pruning it on GCP's own asynchronous schedule — see Operating the Artifact Registry cleanup policy below for what it keeps, what it prunes, and how to override it before a risky release.
The repo confirmed this was a known, explicit gap rather than an oversight nobody noticed: .github/workflows/ci.yml:354 documents tag deletion as "Deliberately out of scope," and .github/workflows/production.yml:131 only deleted a single version tag as part of a re-tag idempotency step — not a retention/cleanup policy. Together with the live API's (pre-fix) cleanupPolicies: null, this confirmed no cleanup policy existed anywhere in this pipeline before Phase 2.
Read this before quoting a dollar figure for this item — both numbers below are upper bounds, not realized savings. Artifact Registry bills deduplicated Docker layers, not the naive sum of each image's own imageSizeBytes. The dry-run baseline captured at apply-time (2026-09-09, OBJ-3466 — a separate, newer snapshot from the Phase 1a inventory figures above, taken two days later): 7,893 versions across the same 3 packages (rails-api 2,756 + agent-runner 2,416 + litellm-proxy 2,721), summing to 2,670.92 GiB of nominal per-image size — but the repository's real on-disk size (gcloud artifacts repositories describe) was ~477 GiB, a ~5.6× gap almost entirely explained by shared base-image/gem/node_modules layers reused build-to-build. The 3,499-digest delete-candidate set (empty intersection with the live-referenced-digest set, verified programmatically before the policy was written) summed to 1,191.52 GiB nominal — ≤$119.15/mo at $0.10/GiB-month, a ceiling, not a forecast, for the same dedup reason. Both the original doc's ≈$43.88/mo estimate and this $119.15/mo ceiling are, as the measurement below confirms, overstatements of real recoverable spend.
Measured recovery, 2026-09-11 (OBJ-3549, closed): ≈$32.04/mo — the warning above, confirmed. GCP's cleanup pipeline runs asynchronously (~24h), so nothing had been deleted when the policy was enabled. Re-measuring gcloud artifacts repositories describe enkidu-registry --location=us-central1 --project=enkidu-488723 at 2026-09-11T19:46Z, ~62h post-enable, gave:
| Value | Kind | |
|---|---|---|
| Original doc estimate | ≈$43.88/mo | estimate — naive per-image size sum |
| Dry-run delete-candidate set | $119.15/mo | ceiling — nominal sizing, not a forecast |
| Post-prune re-measurement | ≈$32.04/mo | measured — real, billed recovery |
Raw readings: pre-enable baseline 488,157.679 MiB (~476.72 GiB, captured 2026-09-09 at apply-time) vs. post-prune 160,041.406 MiB (~156.29 GiB, measured 2026-09-11T19:46Z) — a recovered delta of 328,116.273 MiB (320.43 GiB → ≈$32.04/mo at $0.10/GiB-month). The 156.29 GiB left standing is what the repository costs today: ~$15.63/mo.
Measured recovery is 26.9% of the $119.15/mo ceiling and 73.0% of the ≈$43.88/mo estimate. This is not a shortfall — it's the layer-dedup warning above landing exactly as predicted. Artifact Registry bills deduplicated on-disk layers, not the naive sum of each image's own size, so the delete-candidate set's nominal size overstated real reclaimed space by the same ~5.6× shared-layer factor the apply-time baseline itself documented. The policy avoided ever publishing $119.15/mo as a forecast; $32.04/mo is real, billed money.
The delta is net, not gross — read ≥320.43 GiB as a floor, not an exact figure. The pre-enable baseline was captured at policy-enable time and the re-measurement is ~62h later; CI kept pushing new versions to all three packages throughout that window (the v4.8.3-tagged images were built 2026-09-09T14:08–14:09Z, after the baseline). The 320.43 GiB delta is prune-recovery net of ordinary new-push growth in the same window — true prune-only recovery is ≥320.43 GiB. The caveat works in this conclusion's favour, not against it.
Nothing load-bearing was removed — proven by digest resolution, not by the green deploy. The v4.8.3 production release ran ~9h40m after the policy was enabled — inside GCP's ~24h async pruning window — so it may predate the actual deletions and proves nothing about the post-prune state on its own. What proves it: at ~62h post-enable, all four production Cloud Run services' (enkidu-api-production, enkidu-crono-production, enkidu-worker-production, enkidu-litellm-production) active image digests were resolved directly against the registry — every one still resolved, and both underlying digests still carried the policy-protected tags (v4.8.3, production-latest). The green v4.8.3 deploy is corroborating evidence only.
Operating the Artifact Registry cleanup policy
What it keeps:
- Every
v*-tagged release, forever —.github/workflows/rollback.yml's planned-rollback path can target any past version tag with no documented depth limit, so this is not time-boxed.production-latestandstaging-latest— the moving pointer tagsproduction.yml/staging.ymlpush on every deploy — are also kept forever. - The 10 most recent versions per package (
keepCount=10, theKEEP_COUNTdefault) — margin above the live-referenced set, and doublerollback.yml's own last-5-revisions visibility window.
What it prunes: untagged/unreferenced images older than 30 days (OLDER_THAN_DAYS default), and only what the keep rules above don't already protect — Keep always wins on overlap under GCP's own cleanup-policy semantics.
No packageNamePrefixes filter, by design. The policy applies uniformly to whatever gcloud artifacts packages list returns — today's three packages, or any future fourth. This is deliberate: the failure this phase inherited from the original doc was a hardcoded two-package list that silently missed litellm-proxy, and a package-name-scoped policy could repeat that exact mistake against a future package.
How to override before a risky release: KEEP_COUNT and OLDER_THAN_DAYS are environment-variable overrides on gcp_setup/18_artifact_registry_cleanup_policy.sh — they are not hardcoded requirements. The default invocation (./18_artifact_registry_cleanup_policy.sh) is dry-run and mutates nothing; --enable applies for real. Raise KEEP_COUNT or raise OLDER_THAN_DAYS before a release you might need to roll back further than the default window covers, then re-run in dry-run mode first to confirm what would (and wouldn't) be kept.
Applying identity: artifact-registry-cleanup@enkidu-488723.iam.gserviceaccount.com, bound to a custom role projects/enkidu-488723/roles/artifactRegistryCleanup (7 permissions: repositories.get/.update, packages.list, versions.list/.delete, dockerimages.list, tags.list) at the repository resource (enkidu-registry / us-central1) — not project-wide. Contrast with the Cloud Monitoring scripts (gcp_setup/12_*, 13_*), which bind project-wide because Cloud Monitoring offers no resource-level IAM; Artifact Registry does, so this identity's write access never appears in the project-wide IAM policy.
Already-pruned digests are not recoverable. This is why the dry-run gate and the proven-empty referenced-digest intersection (verified programmatically per package before any policy file is written — gcp_setup/18_artifact_registry_cleanup_policy.sh aborts rather than writing a policy if a referenced digest ever falls into a delete-candidate set) are acceptance criteria, not suggestions.
Gotcha: enabling for real requires --no-dry-run explicitly, not just the absence of --dry-run — see gotchas.md.
2. Staging is a near-complete 24/7 mirror of production — pending (Phase 6b)
Disposition: not yet applied. Phase 3 (staging scheduled idling) has not shipped as of this writing — this section is a deliberate placeholder for Phase 6b, not an omission.
For reference, the original analysis and its caveat still hold as background: ≈$173/mo (enkidu-worker-staging $106.03 + enkidu-crono-staging $52.35 + enkidu-api-staging $14.76) of staging Cloud Run cost runs on the same always-on schedule as production, despite staging having no user-facing uptime requirement outside working hours. enkidu-litellm-staging is the in-estate counter-example that this is viable here (no minScale, 2.4% utilization in August, $2.07/mo) — but it's also the service that took the 2026-09-01 staging outage (OBJ-3166, a scale-from-zero provisioning-rate thundering herd, root cause confirmed not address-space exhaustion — subnet capacity was measured and ruled out, OBJ-3171). deploy/service.staging.yaml:30-54 documents the standing mitigation and states that raising or removing enkidu-api-staging's maxScale: "4" needs new evidence, not just traffic growth (independently, OBJ-3154's connection-pool ceiling caps at the same value).
Do not drop minScale on enkidu-api-staging without clearing that evidence bar. Scheduled idling (nights/weekends, with a warm window before the working day) remains the recommended, lower-risk form of this lever — that's what Phase 3 will apply and this section will document once it ships.
3. Production minScale — DECLINED (both services)
Disposition: DECLINED, both services, on separate measured evidence (2026-09-08, Phase 5a, OBJ-3469). No production config was changed.
enkidu-api-production runs at a 1.05% CPU duty cycle and enkidu-litellm-production at 0.37% (see Billed Quantities & Duty Cycle) — most of what's paid on these two is minScale: 1 standing-instance rent (Min Instance CPU/Memory rates), not active request handling. That's real, but it doesn't settle the tradeoff on its own. A latency budget was stated before either verdict, anchored to something measured rather than a round number:
| Service | Latency budget (stated first) | Measured cold start (Aug 2026) | Verdict |
|---|---|---|---|
enkidu-api-production | ≤2s — 5–9× its own measured warm p95 (230–424ms) | p50 21.4s, typical range 15.3–51.1s, worst-case 216.2s | DECLINED — misses budget 8–100× |
enkidu-litellm-production | ≤15s — half of the 30s synchronous-mutation timeout it sits inside | p50–p95 33.5–40.0s, max 40.0s | DECLINED — exceeds the budget and the hard 30s timeout |
enkidu-api-production: a cold start on this user-facing GraphQL API is a fully user-visible multi-second-to-multi-minute hang, not a background cost — declined. Reopen condition: a future measurement showing cold start reliably ≤2s (e.g. a smaller/faster boot path), or a bounded canary trial at minScale: 0 with acceptable UX metrics over a sustained window.
enkidu-litellm-production: measured cold start already exceeds Ai::ServiceClient::TIMEOUT = 30 (rails_api/app/services/ai/service_client.rb:8), which wraps the synchronous AI-mutation path (GetAdvice, GenerateGoalDraft, RefineDescription, GenerateMilestones, GetInsight, RefineGoal, SuggestGoals — all under rails_api/app/graphql/mutations/ai/, called directly inside GraphQL resolve, blocking the user-facing request). That timeout is rescued to a silently-swallowed nil (service_client.rb:83-85), so the failure mode is a silent request failure, not a slow one — re-introducing the exact shape of the already-realized OBJ-3166 incident (enkidu-litellm-staging's 2026-09-01 scale-from-zero thundering herd) for ~$13/mo of savings. Independently corroborated: the check_litellm/OBJ-3164 gotcha in docs/development/gotchas.md documents scale-from-zero cold starts on this same LiteLLM codebase measured up to ~97s on staging — worse than the deploy-triggered sample measured here, consistent with true scale-from-zero costing more than a revision rollout. Reopen condition: only if (a) OBJ-3166's root cause is mitigated at the platform/quota level, and (b) Ai::ServiceClient::TIMEOUT is raised well above measured cold-start latency or the synchronous AI-mutation path is converted to async (matching the Agent Runner's existing job+webhook pattern).
Full measurement methodology (the startup_latencies query, the warm-state anchor, and the sensitivity checks): OBJ-3469.
Re-checked against measured billing-export data, 2026-09-11 → 2026-09-17 (Phase 6b.2b-iii, OBJ-3941). Both DECLINED verdicts above were re-checked against the BigQuery billing export (enkidu-488723.billing_export, Cloud Run service costs, 7 complete UTC days) once it held enough history to query.
Neither verdict reopens. Both reopen conditions above are latency-based (a cold-start bound, or an incident mitigation plus a timeout/async change) — the billing export is a cost source, and no measured dollar figure can cross a latency threshold. No cost-based reopen condition is introduced here for either verdict.
What the export does settle is the size of the prize each decline forgoes:
| Service | Modeled (Aug 2026, Per-Service Cost Model) | Measured (2026-09-11 → 2026-09-17 run rate) |
|---|---|---|
enkidu-api-production | $10.65/mo | $13.21/mo — measured |
enkidu-litellm-production | $13.60/mo | $12.68/mo — measured |
enkidu-litellm-production: measured confirms the "~$13/mo of savings" claim above — $12.68/mo measured, at/below the $13.60/mo it was modeled against.enkidu-api-production: measured runs modestly above its $10.65/mo modeled figure — consistent with the duty-cycle growth already noted above (1.05% → 1.349% between the Aug 2026 model and the 2026-09-07 re-verification). Traffic grew between the model and this measurement window, so the prize this decline forgoes is, if anything, larger than modeled here, not smaller.
Neither direction changes the verdict: the decline was never primarily a cost call (cost "doesn't settle the tradeoff on its own," above), and a standing-rent cost running above or below its model has no bearing on a latency-based reopen condition.
4. Crono throttling mode — RECOMMENDED, not implemented (OBJ-3506)
enkidu-crono-production + enkidu-crono-staging together cost $52.19 + $52.35 ≈ $104.54/mo to run Crono — a scheduler with essentially no work to do outside firing scheduled jobs. deploy/crono.production.yaml:26-34 documents why cpu: 1 can't be reduced:
IMPORTANT: minScale and maxScale are both "1" — Crono is not multi-process safe. [...] CPU: cpu-throttling: false (always-allocated, so Crono reliably wakes and fires on its own clock rather than in response to requests) requires >=1 full vCPU on Cloud Run — sub-1-vCPU is rejected outright regardless of concurrency (rollback #3). cpu: 1 is the legal floor for this workload shape [...]
So cpu: 1 is a hard floor under cpu-throttling: false — the lever is the throttling mode itself (whether Crono needs always-allocated CPU at all, vs. request-based throttling with some other wake mechanism), not the CPU size. This is unchanged from the original analysis.
Disposition: RECOMMENDED, not implemented (2026-09-08, Phase 5b, OBJ-3469) — filed as its own implementation-sized issue, OBJ-3506, deliberately out of v4.54's own scope. No config was changed in this phase.
The gate: does flipping to request-based throttling recover a large fraction of $104.54/mo, or a small one? CPU utilization (not cpu/allocation_time, which is definitionally 100% under cpu-throttling: false and can't reveal real work done) measured production 0.2415% mean / staging 0.5486% mean over August — modeled post-flip cost $11.85 / $12.03, recoverable ≈$80.66/mo (≈77%). A 10× sensitivity check (to absorb I/O-wait undercounting in the utilization proxy — a real wake-triggered request bills for its full wall-clock duration including I/O wait, not just CPU-busy cycles) still leaves ≈$76.57/mo (≈73%) recoverable — large enough that the architectural question had to actually be evaluated, not declined on cost grounds alone.
Verdict: recommend. Concrete mechanism: Cloud Scheduler → authenticated HTTP wake, one Scheduler job per existing Crono.perform(...) entry in rails_api/config/cronotab.rb (28 entries, finest granularity every 5.minutes — Cloud Scheduler's 1-minute floor covers all of them), reusing crono_jobs' existing durable state (rails_api/db/schema.rb:590-598, which persists job_id/last_performed_at/healthy/log independent of the in-process loop already) and the existing HMAC-SHA256 webhook auth pattern rather than inventing a new one. containerConcurrency: 1 (unchanged) naturally serializes overlapping wake calls, matching the "not multi-process safe" constraint. Net recoverable after the wake mechanism's own cost (28 jobs × 2 environments = 56 Scheduler jobs, sharing one project-wide free tier of 3 → 53 billable × $0.10 ≈ $5.30/mo — this Cloud Scheduler rate was not cross-checked against the live Billing Catalog API the way the Cloud Run rates above were; it's the one unverified number in this section, flagged rather than silently trusted): ≈$75.36/mo (≈$71/mo under the sensitivity case).
Precondition, not optional hardening: a positive liveness check — a Cloud Monitoring alerting policy reading crono_jobs.last_performed_at per job_id, threshold 2× that job's own configured period (a 5-minute job pages if stale >10 minutes; a daily job pages if stale >48h) — must ship before any implementation, routed the same way Dave's existing Crono/health autopilots already page. This directly targets the parse_at failure precedent: an hourly at: crashed Crono at boot and silently ran no scheduled job for ~2 months, caught only by luck, because nothing at the time monitored per-job liveness — only a green deploy, which isn't proof. Cloud Scheduler's own built-in retry-with-backoff on failed wake calls is also a net reliability improvement over the current bare in-process timer, which has no retry concept at all.
Not decided here: shrinking Crono's CPU below 1 vCPU. cpu-throttling: true technically permits fractional CPU, but this decision was scoped to throttling mode only — flagged as a possible follow-up for a future phase, not decided here.
Nothing above contradicts the milestone's hard constraints: maxScale stays "1" on both crono services (minScale is schedule-derived as of OBJ-3726 / v4.54 Phase 3a, not a constant); cpu: 1 isn't touched (only the throttling flag would change); enkidu-api-staging's maxScale: "4" isn't referenced by this decision at all.
Full rate table, utilization methodology, and the wake-mechanism design: OBJ-3469. Implementation tracking: OBJ-3506.
Related debt (not a saving, out of scope): enkidu-worker-*'s maxScale: "1" singleton pin is vestigial
deploy/worker.production.yaml:22-26 documents this directly:
NOTE: minScale and maxScale are both "1", inherited from when Crono ran in this container. Crono no longer runs here (see above, OBJ-1484) — the not-multi-process-safe constraint that originally pinned this service to a singleton no longer applies to THIS service. A future maxScale bump is unblocked by this change but is NOT being performed here.
This is a scaling-headroom finding, not a cost-saving one, and it is explicitly out of scope for v4.54 — be precise about that: raising maxScale would let the worker autoscale under load, which would cost more, not less, if it ever needed a second instance. It's flagged here because it's a real piece of technical debt adjacent to the cost model (the worker is currently a single point of failure with no autoscaling headroom), not because it saves money. Nothing in this milestone touched it.
5. Crono egress anomaly — INVESTIGATED (OBJ-3502)
enkidu-crono-production + enkidu-crono-staging produced 40.0 of the measured 58.6 GiB (68%) of Cloud Run egress in August, despite serving no public traffic — see Egress above for the corrected kind-classification finding (>99.8% is genuine internet-bound traffic, not internal/regional as the original doc stated; the dollar conclusion survives at ~$4.80/mo, on different grounds).
Disposition: INVESTIGATED, defect filed as OBJ-3502 (2026-09-08, Phase 4) — diagnosed, not fixed, per this phase's scope. Root cause traced to source, not hypothesized: two Crono-scheduled hourly jobs make an uncached, per-candidate-user live PostHog flag check before their own cheap in-memory filters narrow the set:
rails_api/app/jobs/social/generate_habit_checkin_reminder_job.rb:61—smart_timing_enabled?→FeatureFlagService.enabled?('smart_notification_timing_enabled', user:), invoked for every unique user among all users with an active habit goal (Goal.active_habits.includes(:user).find_each, line 26) — before the local-hour/due-today/already-reminded filters that would narrow the set.rails_api/app/jobs/critical_path/send_daily_reminders_job.rb:45—rollout_enabled?→FeatureFlagService.enabled?('critical-path-daily-reminder', user:), invoked for every user withcritical_path_reminder_enabled: true— before timezone/local-hour/played-today filters.
Both route through FeatureFlagService.enabled? → POSTHOG_CLIENT.is_feature_enabled (rails_api/app/services/feature_flag_service.rb:27). rails_api/config/initializers/posthog.rb omits personal_api_key:, so posthog-ruby's local flag evaluation never initializes — every single flag check falls through to a live HTTPS round-trip to https://us.i.posthog.com, with no caching between hourly runs and no batching across users in the same run. Fixing OBJ-3502 would materially shrink this egress figure without touching Crono's scheduling semantics at all — independent of the throttling-mode decision in item 4.
Ruled out as the driver, not assumed: Redis/Memorystore (private-kind bytes <20 MiB/service for the month), Cloud SQL (unix socket via the Cloud SQL Auth Proxy sidecar, not counted here), Sentry telemetry (Crono jobs are invoked via a raw .perform call, never through ActiveJob's instrumented path, so they generate zero Sentry transactions), and log shipping (structured logs go to stdout for Cloud Logging's collector, not an app-initiated network call). Best-supported but not fully closed: a staging-vs-production candidate-user population asymmetry (independent evidence: Sentry span volume 9.07M staging vs. 766K production over 14 days, ~11.8×, consistent with docs/development/gotchas.md's documented un-purged-smoke-account gap) — direct DB row counts to confirm this were not available in the verifying runtime. Full trace: OBJ-3468.
Ruled out: Compute Engine was already deleted before the window
An earlier draft of this analysis carried enkidu-preview-db (gcp_setup/04_preview_db_vm.sh) as a likely-orphaned e2-small worth roughly $15-20/mo to decommission, and as a partial explanation for the reconciliation gap. Live verification closes it as a non-item: gcloud compute instances list --project=enkidu-488723 returns Listed 0 items., disks list returns the same (re-confirmed 2026-09-07, no change), and Cloud Audit Logs record the v1.compute.instances.delete at 2026-07-17T19:16:25Z — five and a half weeks before the August window began. There is nothing left to decommission and $0 to recover. Full lifecycle and the evidence trail that made the VM look live are in Compute Engine above.
Negative finding: the database, logging, and storage are not the problem
Cloud SQL (db-f1-micro, 10 GB, ~$10.51/mo), Cloud Storage (~$0/mo), and Cloud Logging ($0/mo, under the 50 GiB free allowance) are all non-factors in this bill. This inverts the natural assumption that the database is the expensive part of running this stack — combined, these three services are under 2.2% of the modeled total. The cost here is overwhelmingly Cloud Run compute (always-on instance-seconds) and Artifact Registry storage, not the data layer.
Out of scope for v4.54
Named explicitly, not silently dropped: enkidu-worker-*'s vestigial maxScale: "1" debt (real technical debt, but raising it costs more, not less — see Related debt above), Compute Engine ($0, nothing left to decommission), Cloud SQL, Cloud Storage, and Cloud Logging (jointly under 2.2% of the modeled total, measured and ruled out as the cost driver — see the negative finding above). None of these were touched by this milestone's remediation work.
Measured window: 2026-08-01 → 2026-08-31. Last confirmed against live GCP: 2026-09-07 (Phase 1a, Cloud Run config/duty-cycle/egress/Memorystore/Compute Engine), 2026-09-08 (Phase 4/5, crono egress mechanism and the minScale/throttling decisions), 2026-09-09 (Phase 2, Artifact Registry cleanup-policy dry-run baseline), 2026-09-11 (Phase 6b.1, Artifact Registry post-prune re-measurement), 2026-09-13 (Phase 6b.2a, billing export enablement + coverage, OBJ-3714), 2026-09-18 (Phase 6b.2b-i, billing-export reconciliation against 7 complete UTC days, OBJ-3936). Last updated: 2026-09-18.