Agent Runner Operations

Operational guide for deploying, monitoring, and troubleshooting the AI Workforce Agent Runner service. For technical architecture, see architecture/agent-runner.md.

Not to be confused with the Multica crew daemon host — the separate Mac host that runs the named crew agents (Maggie, Codi, Dori, etc.), distinct from this Cloud Run service. That host has its own disk-exhaustion failure mode (unbounded task-dir accumulation, not anything below) — see Multica Daemon Disk Hygiene if that's what you're looking for.

Service Overview

Property	Value
Cloud Run service (prod)	`enkidu-agent-runner-production`
Cloud Run service (staging)	`enkidu-agent-runner-staging`
Port	4001
Health endpoint	`GET /health`
Auth	OIDC (prod) / shared secret (dev)
Stateless	Yes (in-memory runMap lost on restart)

Deployment

Docker Build

3-stage build from repo root (needs .claude/skills/ context):

bash

docker build -f agent_runner/Dockerfile -t agent-runner .

Skills are baked into the image at /app/skills from .claude/skills/. Updating skills requires a rebuild and redeploy.

Local Development

bash

cd agent_runner
cp .env.example .env
npm install
npm run dev          # tsx watch on :4001

Requires:

Rails API on :3000 (webhook target)
LiteLLM proxy on :4000 (or direct Anthropic API)

Docker Compose

bash

docker-compose up agent_runner

The docker-compose.yml service config:

Port: 4001
Skills: Mounted read-only from ./.claude/skills
LLM: Routes through http://litellm:4000
Webhook: Posts to http://api:3000/webhooks/ai-workforce
Auth: AUTH_MODE=secret with shared RAILS_WEBHOOK_SECRET

Production (Cloud Run)

Deployed via the deploy-agent-runner job in .github/workflows/staging.yml and .github/workflows/production.yml. Authentication to GCP uses Workload Identity Federation; the service itself is IAM-gated (--no-allow-unauthenticated).

Deploy triggers:

Staging — every push to master deploys enkidu-agent-runner-staging.
Production — a v* tag deploys enkidu-agent-runner-production (gated by the validate job that confirms the tag is on master and the staging image exists).

Build context is the repo root, not agent_runner/:

bash

docker build -f agent_runner/Dockerfile -t <image> .

The Dockerfile bakes the .claude/skills/ tree into the image, so the build needs the whole repo as context.

Secrets and env vars at deploy time:

The only value mounted from GCP Secret Manager is RAILS_WEBHOOK_SECRET (--set-secrets="RAILS_WEBHOOK_SECRET=rails-webhook-secret:latest").
ANTHROPIC_API_KEY is not a Secret Manager secret. It is passed as a throwaway placeholder env var (ANTHROPIC_API_KEY=unused-litellm-is-oidc-gated) because the Claude Agent SDK requires some value, but the real auth boundary is Cloud Run IAM: the runner reaches LiteLLM (ANTHROPIC_BASE_URL) via OIDC, and LiteLLM ignores the x-api-key header entirely. Do not provision an anthropic-api-key secret for this service — it would have no effect.
Other env vars set inline: NODE_ENV, AUTH_MODE=oidc, LITELLM_AUTH_MODE=oidc, MAX_RUN_BUDGET_USD, ANTHROPIC_BASE_URL (the LiteLLM URL), RAILS_WEBHOOK_URL, and the Sentry trio (SENTRY_DSN, SENTRY_ENVIRONMENT, SENTRY_RELEASE). AUTH_MODE controls the Rails webhook hop; LITELLM_AUTH_MODE controls the LiteLLM hop. Both must be oidc in production — they are independent knobs.

Environment Variables

Variable	Required	Default	Description
`PORT`	No	`4001`	Express server port
`ANTHROPIC_BASE_URL`	Yes	—	LiteLLM proxy or Anthropic API URL
`ANTHROPIC_API_KEY`	Yes	—	API key for LLM access
`RAILS_WEBHOOK_URL`	Yes	—	Webhook endpoint for results
`RAILS_WEBHOOK_SECRET`	Yes (secret mode)	—	HMAC signing key
`AUTH_MODE`	No	`secret`	`secret` (dev) or `oidc` (prod) — controls Rails webhook hop
`LITELLM_AUTH_MODE`	No	(unset)	`oidc` to fetch OIDC bearer for LiteLLM; leave unset for local dev or non-IAM-gated endpoints
`SKILLS_DIR`	No	`./skills`	Path to skill SKILL.md files
`MAX_RUN_BUDGET_USD`	No	`1.0`	Per-run token budget cap
`GITHUB_PAT`	No	—	GitHub MCP tool access
`POSTHOG_API_KEY`	No	—	PostHog analytics access
`SENTRY_AUTH_TOKEN`	No	—	Sentry issue querying (MCP tool)
`SENTRY_DSN`	Yes (staging/prod)	—	Sentry ingest DSN for crash reporting; no-op when unset (local dev)
`SENTRY_ENVIRONMENT`	Yes (staging/prod)	`NODE_ENV`	Explicit environment label so staging events aren't mislabelled as production
`SENTRY_RELEASE`	No	—	Release tag for per-release attribution in Sentry (e.g. `v3.9.300`)
`MAILTRAP_API_TOKEN`	No	—	Email template management
`SLACK_WEBHOOK_URL`	No	—	Slack message posting

Monitoring

Health Check

bash

curl http://localhost:4001/health
# { "status": "ok", "timestamp": "..." }

Cloud Run uses this as the liveness probe. Returns 200 if the Express server is responsive.

Run Status

bash

curl http://localhost:4001/runs/{run_id}/status
# { "run_id": "...", "status": "running" | "cancelled" }

Note: Status is in-memory only. After restart, all in-flight runs are lost. Rails AiRun table is the source of truth.

Budget Monitoring

Two independent enforcement layers:

Agent Runner: Per-run cap via MAX_RUN_BUDGET_USD (default $1.00)
LiteLLM: Monthly per-employee cap (default $30/month)

AiBudgetAlertJob runs hourly via Crono and sends Slack alerts at 80% and 100% thresholds.

Troubleshooting

Stale Runs (Stuck in "running")

TimeoutStaleRunsJob automatically marks runs stuck >15 minutes as failed (runs every 15 min via Crono).

Manual cleanup:

ruby

# Rails console
AiRun.where(status: 'running').where('started_at < ?', 15.minutes.ago).update_all(status: 'failed')

Webhook Delivery Failures

The webhook sender retries 3 times with linear backoff (1s, 2s, 3s) and 15-second timeout per attempt.

Check Agent Runner logs for:

[webhook] POST failed — network or auth error
[webhook] all retries exhausted — persistent failure

Common causes:

Rails API down or unreachable
HMAC secret mismatch (RAILS_WEBHOOK_SECRET must match between Agent Runner and Rails)
OIDC token expired or metadata server unreachable (Cloud Run only)

Skill Loading Errors

[skillLoader] skill not found: {name}

Check:

SKILLS_DIR env var points to the correct directory
Skill directory exists: {SKILLS_DIR}/{name}/SKILL.md
In Docker: Skills were baked in at build time — rebuild if updated

Agent SDK Errors

Error	Cause	Fix
`error_max_turns`	Agent exceeded `maxTurns`	Increase maxTurns or simplify task
`error_max_budget_usd`	Per-run budget exceeded	Increase `MAX_RUN_BUDGET_USD`
Connection refused	ANTHROPIC_BASE_URL unreachable	Check LiteLLM proxy status
401 Unauthorized from LiteLLM	OIDC bearer missing or misconfigured	See "401 from LiteLLM" below

Note: error_max_turns and error_max_budget_usd are NOT thrown exceptions — they're result messages with is_error=true. The runner handles them and reports status: 'failed' to Rails.

401 from LiteLLM

Trigger: Agent runs fail; LiteLLM Cloud Run logs show HTTP 401 from the /chat/completions route.

Confirming OIDC is being sent: look for Authorization: Bearer eya… in LiteLLM Cloud Run logs. Absence of this header on a request that hits the IAM-gated endpoint means the runner is not injecting the OIDC token.

bash

gcloud logging read \
  'resource.type="cloud_run_revision" AND resource.labels.service_name="enkidu-litellm-production" AND jsonPayload.route="/chat/completions"' \
  --project=enkidu-488723 --limit=10 --freshness=10m

Diagnosis checklist:

LITELLM_AUTH_MODE=oidc must be set in the Agent Runner deploy YAML (check the deploy-agent-runner job in .github/workflows/production.yml and staging.yml).

IAM binding: the Agent Runner service account must have roles/run.invoker on the LiteLLM Cloud Run service:

bash

gcloud run services add-iam-policy-binding enkidu-litellm-production \
  --region=us-central1 \
  --member="serviceAccount:<agent-runner-sa>@enkidu-488723.iam.gserviceaccount.com" \
  --role="roles/run.invoker" \
  --project=enkidu-488723

AUTH_MODE (Rails webhook hop) and LITELLM_AUTH_MODE (LiteLLM hop) are independent — setting one does not affect the other.

Cancellation Not Working

Cancellation is cooperative — the agent checks runMap.cancelRequested every 500ms via polling interval. If the agent is in a long-running tool call, cancellation may be delayed until the tool returns.

Scaling Considerations

Stateless: Safe to run multiple instances. Each instance has its own runMap.
Concurrency: Express handles concurrent requests. Each POST /runs spawns an async agent execution.
Memory: runMap grows with concurrent runs. Each entry is small (~100 bytes).
Cold starts: Node.js Alpine image starts in ~2 seconds on Cloud Run.
Cost: Per-run budget cap prevents runaway spending. Monthly LiteLLM caps provide a second safety net.

Last updated: 2026-08-31 (cross-link to Multica Daemon Disk Hygiene)

Coach Personas

Agent Runner Operations

Service Overview

Deployment

Docker Build

Local Development

Docker Compose

Production (Cloud Run)

Environment Variables

Monitoring

Health Check

Run Status

Budget Monitoring

Troubleshooting

Stale Runs (Stuck in "running")

Webhook Delivery Failures

Skill Loading Errors

Agent SDK Errors

401 from LiteLLM

Cancellation Not Working

Scaling Considerations

Objectuve Docs

Agent Runner Operations ​

Service Overview ​

Deployment ​

Docker Build ​

Local Development ​

Docker Compose ​

Production (Cloud Run) ​

Environment Variables ​

Monitoring ​

Health Check ​

Run Status ​

Budget Monitoring ​

Troubleshooting ​

Stale Runs (Stuck in "running") ​

Webhook Delivery Failures ​

Skill Loading Errors ​

Agent SDK Errors ​

401 from LiteLLM ​

Cancellation Not Working ​

Scaling Considerations ​

Objectuve Docs

Agent Runner Operations

Service Overview

Deployment

Docker Build

Local Development

Docker Compose

Production (Cloud Run)

Environment Variables

Monitoring

Health Check

Run Status

Budget Monitoring

Troubleshooting

Stale Runs (Stuck in "running")

Webhook Delivery Failures

Skill Loading Errors

Agent SDK Errors

401 from LiteLLM

Cancellation Not Working

Scaling Considerations