Objectuve Docs

Appearance

v4.5 — Runtime Least-Privilege ​

No Cloud Run workload, job, or VM in the project runs as roles/editor anymore — and a CI guard now fails the build before that regresses.

Summary ​

Every Cloud Run service, every Cloud Run job, and the enkidu-preview-db GCE VM ran as the project's default compute service account — and that account held roles/editor, a full-project write grant, plus roles/firebase.admin and roles/secretmanager.secretAccessor. The API is internet-facing and runs an LLM agent loop, the highest-exposure surface in the project, so a single compromised revision or job carried project-wide write across both staging and production, which shared the one identity with no IAM-layer isolation between them.

v4.5 provisioned nine dedicated, least-privilege, per-workload-per-environment service accounts (four Cloud Run workload classes × two environments, plus the preview-db VM), cut every workload over onto them one blast-radius tier at a time — staging, then production, then the VM — and only then stripped roles/editor, roles/firebase.admin, and roles/secretmanager.secretAccessor from the default compute SA. That strip is the milestone's entire security payoff; everything before it is preparation, and the ROADMAP says so explicitly to keep anyone from declaring victory early.

The milestone's defining trait wasn't the final IAM diff — it was how many times a plan-time assumption got overturned by a live GCP read or a look at the actual deploy code instead of a repo-grep or a status field. The preview-db VM's disposition flipped from delete to migrate after a live check found it backing 10 live preview PRs a name-grep couldn't see. Two legacy enkidu-agent-runner services flipped from "provision new SAs" to "just delete them" after reading the deploy workflows found the real workload already ran elsewhere, least-privileged, as a sidecar. A pre-strip live sweep (DEPRIV-0) caught four workloads no phase plan had ever counted before the strip could touch them. Each catch happened before it shipped, not after.

Goal ​

Move every runtime workload in the enkidu-488723 project off the default compute service account (confirmed holding roles/editor — a full-project write grant on every workload) and onto dedicated, least-privilege, per-workload-per-environment service accounts, then strip roles/editor from the default compute SA so no runtime identity carries project-wide write by default.

Scope — What Shipped ​

  • gcp_setup/08_runtime_service_accounts.sh — provisions all 9 dedicated SAs (enkidu-run-{api,worker,jobs,litellm}-{production,staging} + enkidu-preview-db-vm) with grants derived from the real deploy manifests and workflow --set-secrets flags, not hand-written, with a trailing assertion that fails the script if any generated grant list is empty or a manifest-referenced secret is missing
  • deploy/service.{staging,production}.yaml and deploy/worker.{staging,production}.yaml — each pins its dedicated serviceAccountName
  • .github/workflows/{staging,production,preview,preview_teardown,rollback}.yml — every gcloud run jobs create/update and gcloud run deploy invocation passes --service-account
  • enkidu-preview-db GCE VM — migrated onto enkidu-preview-db-vm@… in Phase 4 on 2026-07-16 (no PR), then deleted on 2026-07-17, alongside the ~20 orphaned enkidu-api-pr-* / enkidu-db-setup-pr-* preview resources cleaned up manually that same day (see .github/workflows/ci.yml's lint-workflows guard comment). The branch-based preview architecture it backed had already been retired on 2026-04-05 (39415e33d), so Phase 4's "10 live preview PRs" sighting was that leaked cohort, not real consumers. gcp_setup/04_preview_db_vm.sh still carries --service-account + --no-scopes for any future re-provision, and its enkidu-preview-db-vm@ service account — a confirmed zero-blast-radius orphan (full five-surface check in docs/operations/deployment.md § Runtime service accounts) — was deleted on 2026-09-05; recoverable for 30 days (until ~2026-10-05) via gcloud iam service-accounts undelete 105943331296695590995 --project=enkidu-488723, after which a re-run of gcp_setup/08_runtime_service_accounts.sh would provision a brand-new SA with a different uniqueId, not restore this one
  • Two dormant Cloud Run jobs (enkidu-clerk-migrate, enkidu-make-admin) and two legacy standalone Cloud Run services (enkidu-agent-runner-{production,staging}) deleted — the live agent-runner workload already runs as a least-privileged sidecar inside enkidu-worker-{env}
  • roles/editor, roles/firebase.admin, and roles/secretmanager.secretAccessor removed from the default compute SA (github-actions-deployer's separate firebase.admin binding, needed for Firebase Hosting deploys, was left untouched)
  • New scripts/check-runtime-service-account.mjs (+ .test.mjs) — CI guard that fails the build if any deploy/*.yaml lacks serviceAccountName or any workflow job/service deploy lacks --service-account
  • docs/operations/deployment.md — new runtime-IAM section: the workload-class role table, the gcloud run services replace reset trap, the two-firebase.admin-bindings distinction

Phases ​

PhaseNameStatusPlansHighlights
1Provision least-privilege SAs (additive)Shipped19 SAs live with derived, resource-scoped grants; Roy caught a production-bucket leak and a tautological coverage assertion pre-live-run
2Staging Cloud Run cutoverShipped1Both manifests + all jobs + LiteLLM wired; image upload+delete proven against a staging bucket the milestone discovered had never actually existed
3Production Cloud Run cutoverShipped1Gate corrected mid-phase — the real cutover was the v4.1.1 tag, not the inert master merge; 24h clean soak before Phase 5
4Preview DB VM SA migrationShipped1Delete rescope overturned live — the VM backs 10 live preview PRs by raw IP; migrated instead
5De-privilege default compute SA + CI guard + docsShipped1Pre-strip sweep found 4 uncounted workloads; agent-runner pair deleted, not migrated; Editor/firebase.admin/secretAccessor stripped; CI guard live

Key Decisions ​

  • Strip last, not first. Phases 1-4 create identities and move workloads onto them, but the default compute SA keeps roles/editor the entire time — the strip is Phase 5 and only Phase 5. The ROADMAP calls this out explicitly so no earlier phase gets mistaken for the finish line.
  • Gate on the artifact that actually deploys, not the PR that merges. Phase 3's PR merged to master inertly — production.yml only reads those files on a v* tag. The real cutover event, and the thing the 24h soak actually gates, is the tag cut.
  • A repo-grep "nothing references this" is not the same as "nothing depends on this." Both major disposition reversals this milestone — the preview-db VM (delete → migrate) and the agent-runner pair (provision-new-SA → delete) — came from finding something a static search couldn't see: a raw-IP consumer in one case, a documented-but-never-run cleanup step in the other. Both times, gating the action on a live GCP or deploy-code read caught it before it shipped.
  • A merged script is not applied infrastructure. Phase 1 hit this twice — the 9 SAs didn't exist until someone actually ran the script against live GCP, and a staging GCS bucket the manifests depended on had been declared in a setup script for two days but never (re-)created. Both were caught by a live run, not by CI.
  • Resource-scoped grants are invisible to a project-level audit. The GDPR-exports bucket grant (07_create_gdpr_exports_buckets.sh) targeted the default compute SA specifically and would have silently broken GDPR export post-strip if Phase 1's follow-on PR hadn't redirected it — a gcloud projects get-iam-policy read alone would never have shown it.
  • The CI guard shipped before the risky phase, deliberately. Landing preview_teardown.yml/rollback.yml's wiring separately and early (OBJ-1392) meant Phase 5's guard authored against an already-clean tree, instead of going red for the first time inside the milestone's least-reversible step — where the pressure would have been to narrow the guard rather than fix the gap it found.

Requirements Coverage ​

27 / 27 requirements satisfied — SA 6/6, STAGE 3/3, PROD 2/2, VM 3/3, DEPRIV 7/7, GUARD 2/2, DOCS 4/4.

CategoryCountStatus
SA-1 – SA-66All satisfied
STAGE-1 – STAGE-33All satisfied
PROD-1, PROD-22All satisfied
VM-1 – VM-33All satisfied
DEPRIV-0 – DEPRIV-67All satisfied
GUARD-1, GUARD-22All satisfied
DOCS-1 – DOCS-44All satisfied

Full requirements list and per-phase detail: v4.5-runtime-least-privilege-ROADMAP.md on GitHub. No separate REQUIREMENTS.md file exists for this milestone — requirements are defined inline per phase in the ROADMAP (Phase 5's expanded DEPRIV-5/6 and DOCS-3/4 are defined in the dispatched OBJ-1358 issue body).

Outcomes ​

The default compute service account holds zero project-level primitive roles — confirmed by a live gcloud projects get-iam-policy read (etags BwZW48BbxkM= / BwZW48EPpyU= / BwZW48Gj6vw= on the three removed bindings). Every Cloud Run workload runs under a dedicated identity scoped to exactly what that workload needs, generated from the real deploy manifests rather than hand-maintained. (The preview-db VM was migrated too, then deleted on 2026-07-17 — see Scope above.) Staging and production no longer share one identity — a compromise in one environment no longer implies write access to the other. A regression that omits a service account from a manifest or workflow now fails CI instead of silently defaulting back onto the compute SA.

Tech Debt ​

  • (Phase 5) No independent second-identity verification of the IAM strip — the crew's own runtime service account lacks permission to re-run get-iam-policy itself, so two of the ten Phase 5 acceptance criteria rest on Josh's first-party report rather than a cross-checked second read. Non-blocking given Josh executed as project Owner, but worth a lightweight read-only IAM-audit role if this class of milestone recurs.
  • (Milestone-wide) No automated drift check between what gcp_setup/*.sh declares and what actually exists in enkidu-488723 — this bit Phase 1 twice (the SAs, a missing staging bucket) before a live run or luck caught it either time. Flagged as a candidate CI check, explicitly not yet scoped.
  • (Phase 5) docs/operations/deployment.md's pre-existing Cloud Run Services/Jobs inventory lists (2 services / 3 jobs) are stale against the full 6-service/22-job tier this milestone actually migrated — out of scope for the DOCS-1..4 pass, flagged for a future docs-lint sweep.
  • 0d1f038aa — Phase 1: provision 9 least-privilege runtime service accounts (#1495)
  • 2cc25e311 — Phase 1 follow-on: GDPR-exports bucket grants for dedicated SAs (#1535)
  • 6c5d45d18 — Phase 3: wire production Cloud Run workloads to least-privilege service accounts (#1501)
  • 5871cf744 — feat(ci): add runtime service-account guard for Cloud Run manifests + deploys (#1579)
  • 471ab9bd6 — [Dori] docs(v4.5): document runtime least-privilege IAM model (#1580)

Last updated: 2026-07-18 Version: v4.1.1

Loading…