Objectuve Docs

Appearance

AI Operations Runbook ​

Operational procedures for Objectuve's AI stack (LiteLLM proxy, Rails AI client, cost controls). Every section is a self-contained scenario with triggers, steps, verification, and rollback.

Last reviewed: 2026-04-09 Runbook owner: Ops rotation (see CODEOWNERS)

Kill-Switch Flag Flips ​

When to use ​

  • Slack budget alert fires and an immediate spend halt is needed.
  • A specific AI feature is producing harmful or embarrassing output and must be stopped.
  • Incident response: LiteLLM or Gemini upstream is degraded and graceful degradation (cached/nil responses) is preferred over continued retries.
  • The Phase 3 two-tier kill-switch operates at two layers: the mutation-layer (returns an AI_DISABLED error extension) and the service-layer (Ai::ServiceClient.chat returns nil, background jobs degrade gracefully).

Procedure ​

There are three flags, each controlling one "virtual key" category:

  • ai.coaching_enabled — controls coaching/default + coaching/milestones (user-facing mutations).
  • ai.moderation_enabled — controls moderation/screen (ScreenContentJob).
  • ai.batch_enabled — controls batch/insights + batch/check-ins + batch/description.

The feature-to-category map is the CATEGORY_FOR_FEATURE constant in rails_api/app/services/ai/service_client.rb. Unknown features default to batch — the most-restrictive safe bucket.

Flipping a flag requires a commit + CI deploy. There is no runtime override (Phase 3 D-15).

  1. Identify which category to disable from the Slack alert body, the admin dashboard per-feature pie chart, or the incident report.

  2. Edit the target settings file:

    bash
    # Production
$EDITOR rails_api/config/settings/production.yml

# Staging (e.g., testing a disable before production)
$EDITOR rails_api/config/settings/staging.yml

  3. Set the flag to false. Example disabling coaching in production:

    yaml
    ai:
  use_litellm: true         # leave as-is
  coaching_enabled: false   # <-- flip this
  moderation_enabled: true
  batch_enabled: true
  monthly_budget_cents: 50000
  budget_alert_thresholds: [80, 100]

  4. Commit and push. Staging deploys on push to master:

    bash
    git add rails_api/config/settings/production.yml
git commit -m "ops: disable ai.coaching_enabled in production"
git push origin master

  5. For production, tag a release:

    bash
    release   # or: release <version> — see docs/operations/deployment.md

  6. Wait for Cloud Run to finish rolling out. Staging typically takes ~5 minutes, production ~8 minutes. Watch the GitHub Actions run for the staging deploy or the production deploy workflow.

Verification ​

  1. GraphQL-level check. From a mutation that hits the disabled category, expect an AI_DISABLED error extension:

    bash
    curl -X POST https://api.objectuve.com/graphql \
  -H 'Content-Type: application/json' \
  -H "SessionToken: $TOKEN" \
  -d '{"query":"mutation { generateMilestones(goalId: \"GOAL_PUBLIC_ID\") { milestones errors } }"}'
# Expect errors[0].extensions.code == "AI_DISABLED"

  2. LiteLLM request log. Confirm LiteLLM Cloud Run has received zero /chat/completions requests for the disabled category since the flag flipped:

    bash
    gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="enkidu-litellm-production" AND jsonPayload.route="/chat/completions"' \
  --project=enkidu-488723 \
  --limit=20 \
  --freshness=15m

  3. Admin dashboard. After ~1 minute (the data is real-time, no cache), open the admin "AI" tab and confirm the per-feature pie chart for the disabled feature stops growing.

Rollback ​

  1. Flip the flag back to true in the same settings file.
  2. Commit, push, deploy (same as the flip procedure).
  3. Verify with the same GraphQL curl — expect a normal response (no AI_DISABLED error).

User-Visible Degraded States ​

When AI features are disabled or unavailable, users see fallback behavior instead of AI-generated insights. This section documents what users see on each surface when AI is degraded.

Query vs. Mutation Asymmetry ​

The me.currentInsights query (used on all major pages) behaves differently from mutations like getInsight when AI is disabled:

  • Queries (currentInsights): Return [] (empty array, no error). Allows frontend to render fallback rung silently.
  • Mutations (getInsight, generateMilestones): Return AI_DISABLED error. Used in discrete user-triggered actions (milestone generation, on-demand insight).

Why the asymmetry? Queries fire on every page mount. A hard error would surface as a toast on every navigation, degrading UX. Mutations are discrete actions — a single error message is appropriate.

Per-Surface Degraded States ​

When Settings.ai.batch_enabled = false, rate-limited, or upstream error:

SurfaceAI DisabledRate LimitedUpstream ErrorFallback Source
Dashboard insight cardPersona-aware fallback copy¹Persona-aware fallback copyPersona-aware fallback copyAi::Fallbacks::Insights
Goals insight cardPersona-aware fallback copy¹Persona-aware fallback copyPersona-aware fallback copyAi::Fallbacks::Insights
Goal Detail insight cardPersona-aware fallback copy¹Persona-aware fallback copyPersona-aware fallback copyAi::Fallbacks::Insights
Achievements insight cardPersona-aware fallback copy¹Persona-aware fallback copyPersona-aware fallback copyAi::Fallbacks::Insights
Communities insight cardPersona-aware fallback copy¹Persona-aware fallback copyPersona-aware fallback copyAi::Fallbacks::Insights
Goal Coach modalStatic welcome + persona-aware chipsStatic welcome + persona-aware chipsCoachUnavailableCard (Apollo error boundary)Depends on error path
Generate MilestonesAI_DISABLED error → toastAI_RATE_LIMITED error → toastError toastMutation error handling
Generate Goal DraftAI_DISABLED error → toastAI_RATE_LIMITED error → toast{ draft: null }, no error — hero stays in form phase, user proceeds manuallyMutation error handling (GenerateGoalDraft)
Suggest Goals (quiz)No mutation-layer guard — falls through to { suggestions: null, errors: [] } (quiz empty state), not an AI_DISABLED error²AI_RATE_LIMITED error → quiz error state{ suggestions: null, errors: [] }, quiz empty stateMutation error handling (SuggestGoals)

¹ AI Disabled + Cold Cache note: When batch_enabled = false and the cache is empty (batch job hasn't run or cache expired), currentInsights returns []. The frontend then renders the third-rung fallback (generateMockInsight). When the cache is primed (batch has populated it), users see the cached AI response or persona-aware fallback copy from the batch job. This asymmetry ensures fallback copy is always persona-aware: either from the backend batch (Rung 1 or 2) or from the mock generator (Rung 3), never silent nils.

² suggestGoals has no mutation-layer AI_DISABLED check. GenerateGoalDraft and GenerateMilestones both call Ai::SettingsGuard.enabled?('coaching_enabled', ...) before invoking the AI service and raise a typed AI_DISABLED error if it's off (D-14 tier-1). SuggestGoals (rails_api/app/graphql/mutations/ai/suggest_goals.rb) skips that guard entirely and relies solely on Ai::ServiceClient.chat's tier-2 category_enabled? check, which returns nil rather than raising — so a disabled-category request surfaces to the quiz as an empty result, not an explicit error. Not a bug filed as of this writing; noted here because it's a real asymmetry an on-call engineer could otherwise mistake for a suggestGoals-specific outage.

Key invariant: Users never see garbled or partial AI output. The fallback ladder ensures every surface has clean, persona-aware copy or an explicit error message.

Persona-Aware Fallback Copy ​

The fallback library (Ai::Fallbacks::Insights) has 6 personas (captain, spark, mirror, rival, sage, analyst) × 4 surface types (welcome_back, streak_break, check_in, generic_insight). Example:

Persona: spark
Surface: generic_insight
Fallback: "Momentum starts small. Pick one thing and do it — right now counts."

Fallback copy is action-forward, tone-of-voice consistent with Coach brand, and requires no LLM context.

Budget Alert Response + Threshold Adjustment ​

When to use ​

  • Slack #ai-budget (or the configured channel) receives an 80% or 100% alert from AiMetrics::AiBudgetAlertJob.
  • Monthly spend is trending above plan without an alert (admin dashboard shows budgetUtilizationPercent > 80% but no alert arrived — see troubleshooting below).

Procedure ​

  1. Open the admin AI dashboard. Note the top-spending feature from the per-feature pie chart and the per-model breakdown (which model is most expensive).

  2. Decide the response. Choose one:

    • (a) Disable the top-spending category via the Kill-Switch Flag Flips procedure above. This stops the bleeding immediately.

    • (b) Lower that category's rate limit. See "Rate-limit Tuning" below. This preserves access but slows burn.

    • (c) Raise the monthly budget for the remainder of the month. Edit rails_api/config/settings/production.yml:

      yaml
      ai:
  monthly_budget_cents: 75000  # was 50000 — raises the cap from $500 to $750

      Commit, push, tag, deploy. The next hourly AiBudgetAlertJob run recomputes against the new ceiling.

  3. Notify stakeholders in the same Slack channel describing which option was chosen and why.

Verification ​

  • After the next hourly run of AiMetrics::AiBudgetAlertJob (wait up to 60 minutes or re-trigger — see below), confirm no duplicate 80% or 100% alerts fire if you raised the budget. If you disabled a category, confirm the admin dashboard stops growing in that category.

  • Check the Crono job ran successfully:

    bash
    gcloud logging read 'resource.type="cloud_run_revision" AND resource.labels.service_name="enkidu-api-production" AND jsonPayload.job_class="AiMetrics::AiBudgetAlertJob"' \
  --project=enkidu-488723 \
  --limit=5 \
  --freshness=90m

Re-firing alerts within the same month (clearing dedup) ​

If the alert must re-fire in the same calendar month (e.g., you raised the budget and then spend crossed the new 80%), clear the Rails.cache dedup key:

bash
# Replace 202604 with the current YYYYMM (UTC) and 80 with the threshold to re-fire.
gcloud run jobs execute enkidu-rails-console-production --region=us-central1 --wait
# From inside the console session:
Rails.cache.delete("ai:budget_alert:#{Time.current.utc.strftime('%Y%m')}:80")

Alternative (if the rails-console job is unavailable): open a Cloud Shell Rails console via the Cloud SQL proxy and run the same Rails.cache.delete call. The key will be re-written on the next threshold crossing.

Rollback ​

  • If you lowered the budget by mistake, edit production.yml back to the previous value, commit, deploy.
  • If you disabled a category by mistake, follow the Kill-Switch rollback.
  • If you cleared a dedup key and got an unwanted duplicate alert, there is no action to take — the next cache write re-establishes dedup for the month.

Rate-limit Tuning ​

When to use ​

  • Users report "You're sending too fast — try again in a minute" errors (or the equivalent AI_RATE_LIMITED extension in GraphQL responses).
  • Spend is trending above budget but disabling a category is too heavy-handed.
  • Load test or incident analysis shows the current limits are wrong for real traffic.

Procedure ​

Phase 3 enforces rate limits in rails_api/app/services/ai/service_client.rb via the RATE_LIMITS constant. Current values (authoritative — confirm in the file):

  • coaching → 10 requests per minute per user
  • milestones → 5 requests per minute per user
  • goal_draft → 5 requests per minute per user (generateGoalDraft mutation; enforced since OBJ-1661/PR #1728 — previously present in the mutation's guard code but never actually enforced, since RATE_LIMITS had no entry for this feature and would_exceed? read a Redis key nothing ever wrote)
  • goal_suggestions → 5 requests per minute per user (suggestGoals mutation; same OBJ-1661 fix)
  • Other categories (moderation, insights, check_ins, description) are unlimited per user — rate_limited? returns false because RATE_LIMITS.key?(feature) is false.

Cache hits via CoachService never reach ServiceClient.chat, so they never consume rate-limit budget. Only cache misses are counted.

  1. Edit the constant:

    bash
    $EDITOR rails_api/app/services/ai/service_client.rb
# Find `RATE_LIMITS = { ... }` and adjust the integer values.

  2. Run the rate-limiter specs locally:

    bash
    cd rails_api && bundle exec rspec spec/services/ai/rate_limiter_spec.rb spec/services/ai/service_client_spec.rb

  3. Commit and deploy:

    bash
    git add rails_api/app/services/ai/service_client.rb
git commit -m "ops: adjust coaching rate limit from 10/min to 5/min"
git push origin master
# For production:
release

Verification ​

  1. Staging smoke test. Exercise the new limit intentionally from the staging frontend:
    • Rapidly click "Generate Milestones" more than the new limit within 60 seconds.
    • Expect the (limit + 1)th click to return the rate-limit error.
  2. Production check. Watch the Sentry error rate for AI_RATE_LIMITED events after deploy. Compare to the pre-deploy baseline.
  3. Spec coverage. spec/services/ai/rate_limiter_spec.rb and any mutation request specs exercising the limit must still pass.

Rollback ​

  1. Revert the commit:

    bash
    git revert <sha> --no-edit
git push origin master

  2. For production, tag and deploy the revert commit.

  3. Alternatively, edit the constant back to the previous value and commit it as a forward fix.

Switching the Coach Model ​

When to use ​

  • Product/ops wants to move the Coach off the current Gemini model (cost, quality, or capability reasons).
  • Testing a newly-added model id (see "Adding a genuinely new model" below) before it becomes the default.

As of v4.53, this is an admin-app action, not a deploy. Before this milestone, changing the model meant editing the model: value for the six Gemini aliases across infra/litellm/config.staging.yaml and infra/litellm/config.production.yaml and redeploying the LiteLLM Cloud Run service (deploy-litellm in staging.yml/production.yml) — the same mechanism .claude/skills/litellm/SKILL.md's "Add a new model alias" section still uses for other kinds of LiteLLM config changes. The Coach model no longer needs that: LiteLLM now carries a gemini/* wildcard route (infra/litellm/config.production.yaml, infra/litellm/config.staging.yaml) that passes any gemini/* id through unchanged, and Rails resolves which concrete id to send on every request via Ai::CoachModel.current — an app_settings-backed value an admin can change at runtime.

Procedure ​

  1. Open the admin dashboard's Monitoring view — the Coach model card (admin_dashboard/src/components/CoachModelPanel.vue) sits alongside the AI usage panel (admin_dashboard/src/views/MonitoringView.vue).
  2. This requires super admin. The card renders read-only ("Switching the Coach model needs a super admin") for any other role — the frontend gates the control on the update_coach_model permission, which only super_admin holds (admin_dashboard/src/composables/useRoles.ts), and the backend's updateCoachModel mutation independently enforces admin auth (require_admin!, rails_api/app/graphql/mutations/update_coach_model.rb).
  3. Pick a model from the "Switch to" list and confirm. This calls the updateCoachModel GraphQL mutation, which runs Ai::SetCoachModel — it rejects any id that isn't routable (Ai::ModelCatalog.routable?: allowlisted and carrying a rate window that covers right now — see "Adding a genuinely new model" below) before writing anything, then writes the app_settings row and busts the cache.
  4. The change is live within about a minute, not instantly. Ai::CoachModel.current (rails_api/app/services/ai/coach_model.rb) reads through a 60-second-TTL Rails cache (CACHE_TTL = 60.seconds). The mutation calls Ai::CoachModel.bust_cache! on success, so most requests pick up the new value almost immediately — but any request mid-flight against a still-warm cache entry, or landing before the bust propagates, can still see the previous model for up to that window.
  5. Nothing to deploy, no Cloud Run rollout to wait on, no LiteLLM redeploy.

The moderation side-effect ​

One setting controls all six Gemini aliases, not just the ones users think of as "Coach": coaching/default, coaching/milestones, moderation/screen, batch/insights, batch/check-ins, batch/description (Ai::ModelCatalog::GEMINI_ALIASES). Switching "the Coach model" also switches the model doing content moderation (moderation/screen, used by ScreenContentJob). There is no separate control for moderation — a model change is an all-six-aliases change, always.

Adding a genuinely new model ​

The admin picker only ever offers routable ids — allowlisted and carrying a rate window that covers right now, the same definition Ai::ModelCatalog.routable? encodes (Resolvers::AdminQueries.coach_model_options filters on the allowlist plus rate_for inline, not yet calling routable? directly, but the effective behavior is identical). Adding a model Coach has never run before still needs a Rails PR — this is a deliberate guard (D-02), not a gap:

  1. Verify the candidate id against LiteLLM's model_prices_and_context_window.json and the provider's own pricing page — Ai::ModelCatalog's header comment states this as a standing rule: never seed the catalog from an unverified id.
  2. Add the id to Ai::ModelCatalog::ALLOWLIST and give it an effective-dated entry in RATE_SCHEDULE (rails_api/app/services/ai/model_catalog.rb).
  3. No LiteLLM config change is needed for a Gemini id — the gemini/* wildcard already routes it.

Allowlisted is not the same fact as routable — both the write and read paths gate on the stronger one.Ai::ModelCatalog::ALLOWLIST is the registry of ids we can still price (it includes retired ids kept only so historical resolved_model rows still resolve to a rate); routable? is the subset Coach may actually run. gemini/gemini-2.5-flash stays in ALLOWLIST purely for that historical-pricing reason — its own rate window closed 2026-09-04, so it is allowlisted but no longer routable. Both Ai::SetCoachModel (the write path, rails_api/app/interactions/ai/set_coach_model.rb) and Ai::CoachModel.resolve (the read path, covering a stored id whose rate window lapses on a calendar date after being set) gate on routable?, not allowlisted? alone (closed as OBJ-3392) — so an allowlisted-but-lapsed id is rejected by the mutation with a distinct error ("is allowlisted but has no rate covering now — catalog freshness problem, not a typo", separate from the plain "not an allowlisted Gemini model" rejection) and, on the read side, falls back to the default rather than ever being served or billed at $0.00.

Fail-safe ​

A missing app_settings row, a stored value that isn't routable (not allowlisted, or allowlisted but its rate window has lapsed), or any read failure (e.g. Redis unavailable) all fall back to Settings.ai.default_coach_model (rails_api/config/settings.yml) rather than raising — Coach degrades to its prior default behavior, never to an exception (Ai::CoachModel.resolve). A live operator-chosen id whose rate window later lapses on a calendar date reverts to the default this way with no mutation involved — the app_settings row is left intact (so it's recoverable), and the error log plus the id dropping out of the admin picker's options are what surface it.

Currently routable models ​

Two ids carry a rate window covering the present right now: gemini/gemini-3.8-flash and gemini/gemini-2.5-pro — so the picker has a real choice and the switch is genuinely exercisable end-to-end. If a catalog change ever leaves only one (or zero) routable ids, the card falls back to a dedicated "nothing to switch to right now" state rather than a broken one-entry dropdown (isOnlyOption, CoachModelPanel.vue) — that state still exists and still fires in that condition, it's just not today's condition.

Audit trail ​

Every successful change lands in admin_actions as action_type: 'UPDATE_COACH_MODEL', wrapped in Admin::Logged (rails_api/app/interactions/ai/set_coach_model.rb), which records the actor and the before/after AppSetting state automatically. The admin card's "Recent changes" list shows the last five (Resolvers::AdminQueries.build_coach_model_setting, ordered newest-first). A rejected switch (id not routable — either not allowlisted, or allowlisted but its rate window has lapsed) writes neither an AppSetting row nor an AdminAction — the guard runs before Admin::Logged is ever called.

Verification ​

  1. Admin card. After switching, the "Running now" value updates and the new entry appears at the top of "Recent changes" with your name and "just now."
  2. Wait ~60 seconds, then confirm a real Coach interaction (chat, milestone generation, a moderation check) records the new model in ai_usage_events.resolved_model — ai_usage_events.model stays the fixed alias name (e.g. coaching/default) for every request regardless of which concrete model answered it, and the response body itself won't tell you either.

Rollback ​

Switch back to the previous model the same way — it's the identical procedure in either direction, and takes the same ~60-second window. There is nothing to redeploy or revert.

API Key Rotation + Provider Swap ​

When to use ​

  • Key rotation — scheduled (e.g., 90-day) or emergency (key leaked, revoked by provider).
  • Provider swap — adding Anthropic or OpenAI to the fallback chain. Phase 4 ships Gemini-only; future providers need new secrets + config edits.
  • Provider outage — Gemini is down and you want to fail over to a backup provider once one has been wired in.

Procedure: Rotating google-api-key ​

  1. Generate a new key in Google Cloud Console (Credentials → Create credentials → API key). Restrict the new key to the Gemini API only.

  2. Add a new version to Secret Manager. Do NOT delete the old version yet:

    bash
    printf '%s' 'NEW-KEY-VALUE' | \
  gcloud secrets versions add google-api-key \
    --data-file=- \
    --project=enkidu-488723

  3. Force the LiteLLM service to pick up the new version. Cloud Run caches :latest across container starts, but a fresh revision re-reads it:

    bash
    gcloud run services update enkidu-litellm-production \
  --update-env-vars="SECRET_ROTATION=$(date +%s)" \
  --region=us-central1 \
  --project=enkidu-488723

    The SECRET_ROTATION env var is a no-op for the app but forces a new revision, which re-reads :latest.

  4. Verify the new key works. Make a test call via the admin dashboard (e.g., generate a milestone) and confirm the response.

  5. Disable the old key version in Google Cloud Console after a 24-hour window in which no errors appear in Sentry.

    bash
    # List versions:
gcloud secrets versions list google-api-key --project=enkidu-488723
# Destroy the old secret version once confident:
gcloud secrets versions destroy <OLD_VERSION_NUMBER> --secret=google-api-key --project=enkidu-488723

Procedure: Adding a new provider (Anthropic or OpenAI) ​

  1. Create the provider account and generate an API key.

  2. Create a GCP Secret Manager secret for the key:

    bash
    # Example: Anthropic
gcloud secrets create anthropic-api-key \
  --replication-policy=automatic \
  --project=enkidu-488723

printf '%s' 'sk-ant-...' | \
  gcloud secrets versions add anthropic-api-key --data-file=- --project=enkidu-488723

  3. Wire the secret into the LiteLLM Cloud Run deploy in .github/workflows/staging.yml and .github/workflows/production.yml. Find the deploy-litellm job's --set-secrets="GOOGLE_API_KEY=google-api-key:latest" and append ,ANTHROPIC_API_KEY=anthropic-api-key:latest.

  4. Edit the LiteLLM config to add the provider to each capability alias and set up the fallback chain:

    bash
    $EDITOR infra/litellm/config.production.yaml
$EDITOR infra/litellm/config.staging.yaml

    For each capability alias, add the new provider as a fallback:

    yaml
    model_list:
  - model_name: coaching/default
    litellm_params:
      model: gemini/gemini-3.8-flash
      api_key: "os.environ/GOOGLE_API_KEY"
  - model_name: coaching/default
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: "os.environ/ANTHROPIC_API_KEY"

router_settings:
  fallbacks:
    - coaching/default: ["coaching/default"]

    Consult the LiteLLM docs for the exact fallback-chain syntax at the time of the edit — this has changed across LiteLLM versions.

  5. Add a pricing entry in rails_api/app/services/ai/pricing.rb for the new model so AiUsageEvent.cost_cents is accurate.

  6. Commit and deploy. Staging first, verify, then production tag.

Which secret goes where (cheat sheet) ​

SecretUsed byLocation
google-api-keyLiteLLM (Gemini calls)deploy-litellm --set-secrets in both workflows
anthropic-api-key (future)LiteLLM (Claude calls)deploy-litellm --set-secrets in both workflows
openai-api-key (future)LiteLLM (GPT calls)deploy-litellm --set-secrets in both workflows
slack-budget-alert-webhook-urlRails API (budget alerts)deploy-backend + migrate job + demo-reset job --set-secrets in both
clerk-secret-key[-dev]Rails API (JWT verification)All Rails API jobs --set-secrets
google-api-key (Rails) — NEVER—Rails calls LiteLLM, not Gemini directly. Rails must never see this key.

Capability alias → model mapping ​

After a swap, these aliases continue to resolve to LiteLLM's routing layer, not directly to a provider:

  • coaching/default → LiteLLM routes to the first healthy provider in the fallback list
  • coaching/milestones → same
  • moderation/screen → same
  • batch/insights → same
  • batch/check-ins → same
  • batch/description → same

Rails never knows which provider answered — it only sees the OpenAI-compatible response shape. The model column in ai_usage_events records whichever provider/model actually answered (from the LiteLLM response's model field).

Verification ​

  1. LiteLLM health check — the LiteLLM Cloud Run service is IAM-gated, so this needs an OIDC token. From a machine with gcloud auth:

    bash
    TOKEN=$(gcloud auth print-identity-token \
  --audiences=https://enkidu-litellm-production-h26eaxs77q-uc.a.run.app)
curl -H "Authorization: Bearer $TOKEN" \
  https://enkidu-litellm-production-h26eaxs77q-uc.a.run.app/health/liveliness
# Expect HTTP 200

  2. Rails smoke test. curl https://api.objectuve.com/health should return 200 with every component check reporting ok.

  3. End-to-end. Make a real AI call via the admin dashboard or GraphQL Playground and verify the response body.

  4. Admin dashboard. After a few calls, confirm the per-model pie chart shows the new provider's model name.

Rollback ​

  1. Key rotation gone wrong. Enable the old secret version:

    bash
    gcloud secrets versions enable <OLD_VERSION> --secret=google-api-key --project=enkidu-488723

    Then force a Cloud Run revision (the same SECRET_ROTATION trick) to re-read :latest.

  2. New provider broken. Revert the config edit in a new commit:

    bash
    git revert <sha> --no-edit
git push origin master

    Or manually edit to remove the fallback entries and redeploy.

  3. Emergency: disable LiteLLM entirely. Flip ai.coaching_enabled, ai.moderation_enabled, and ai.batch_enabled to false per the Kill-Switch procedure. All AI features degrade gracefully (ServiceClient.chat returns nil, mutations return AI_DISABLED, background jobs skip silently).

AI Workforce (Agent Runner) ​

Service Overview ​

The Agent Runner (agent_runner/) is a TypeScript Cloud Run service that executes AI employee runs using the Claude Agent SDK. It communicates with Rails via HMAC-authenticated webhooks and routes LLM calls through LiteLLM.

  • Health check: GET /health on port 4001
  • Run status: GET /runs/:id/status (in-memory, lost on restart)
  • Cancel: POST /runs/:id/cancel (cooperative — agent checks every 500ms)

401 from LiteLLM (Agent Runner → LiteLLM OIDC auth) ​

Trigger: AI Workforce runs fail; Agent Runner logs show LiteLLM returned 401 or the stream errors immediately.

LiteLLM Cloud Run is IAM-gated (--no-allow-unauthenticated). The Agent Runner authenticates via OIDC bearer set as ANTHROPIC_AUTH_TOKEN — controlled by LITELLM_AUTH_MODE=oidc in the deploy YAML (separate from AUTH_MODE, which controls the Rails webhook hop).

Quick check: verify Authorization: Bearer eya… appears in LiteLLM Cloud Run request logs. Absence means the runner is not injecting the OIDC token.

  1. Confirm LITELLM_AUTH_MODE=oidc is in the deploy-agent-runner job env in the relevant workflow file.
  2. Confirm the Agent Runner service account has roles/run.invoker on enkidu-litellm-production (and enkidu-litellm-staging).
  3. Full diagnosis steps: see "401 from LiteLLM" in the Agent Runner ops runbook.

Agent Runner Not Responding ​

Trigger: Health check fails, or runs stay in running status beyond 15 minutes.

  1. Check Cloud Run logs: gcloud run services logs read enkidu-agent-runner-production --limit=50
  2. Verify the service is deployed: gcloud run services describe enkidu-agent-runner-production
  3. TimeoutStaleRunsJob automatically marks runs stuck >15 minutes as failed (runs every 15 min via Crono)
  4. If the service is down, redeploy: gcloud run deploy enkidu-agent-runner-production --image=...

Run Stuck in "Running" ​

Trigger: AiRun record has status: running and started_at > 15 minutes ago.

  1. TimeoutStaleRunsJob handles this automatically — check if it's running: bundle exec rake crono:status
  2. Manual fix: AiRun.where(status: 'running').where('started_at < ?', 15.minutes.ago).update_all(status: 'failed', error_message: 'Manual timeout', finished_at: Time.current)
  3. The employee can be re-triggered via "Run Now" in the admin dashboard

Budget Alert Fires ​

Trigger: Slack notification from BudgetAlertJob at 80% or 100% threshold.

  1. Check the employee's monthly spend: admin dashboard > AI Workforce > employee card shows budget bar
  2. At 80%: informational — no action required unless unexpected
  3. At 100%: runs are blocked automatically by TriggerRun interaction. No manual intervention needed.
  4. To increase budget: admin dashboard > employee detail > update monthly budget

Delivery Failure ​

Trigger: Artifact approved but delivered_at is nil, or delivery_metadata contains error.

  1. Check Sidekiq dashboard for failed DeliverArtifactJob jobs
  2. Common causes:
    • Missing env var (GITHUB_ACCESS_TOKEN, MAILTRAP_SANDBOX_INBOX_ID, SLACK_WEBHOOK_URL)
    • GitHub API rate limit (check X-RateLimit-Remaining in logs)
    • Slack webhook URL invalid (must be hooks.slack.com — host validation enforced)
  3. Re-delivery: re-approve the artifact or manually enqueue DeliverArtifactJob.perform_later(artifact.public_id)
  4. Delivery is idempotent — safe to retry

Brand-Voice Filter Auto-Rejecting Everything ​

Trigger: All artifacts from an employee are auto_rejected, none reach pending queue.

  1. Check the employee's post_filter_skill — verify the skill file exists in .claude/skills/
  2. Check recent_rejections in the employee's memory (admin dashboard > Memory tab)
  3. The post-filter uses conservative pass-on-error — if the LLM call fails, the artifact passes through
  4. If the skill is producing false negatives: temporarily clear post_filter_skill on the employee via UpdateAiEmployee mutation
  5. The post-filter is a second query() call — check LiteLLM logs for the workforce/content model alias

Memory Corruption ​

Trigger: Employee produces nonsensical output, or memory tab shows unexpected data.

  1. View memory in admin dashboard > employee detail > Memory tab
  2. To clear a specific key: AiEmployeeMemory.find_by(ai_employee_id: employee.id, memory_key: 'key_name')&.destroy
  3. To clear all memory: employee.ai_employee_memories.destroy_all
  4. Memory is capped at ~2,000 tokens — if an entry is too large, the oldest content is truncated

Disabling an Employee ​

  1. Admin dashboard > AI Workforce > click "Pause" on the employee card
  2. Or via GraphQL: pauseAiEmployee(employeeId: "...", active: false)
  3. Paused employees are excluded from ScheduleRunsJob but retain their configuration and memory

Cross-references ​

  • Deployment mechanics — deployment.md (release tags, rollback, Cloud Run specifics)
  • Observability — observability.md (Sentry, Lograge, health checks, log queries)
  • Testing procedures — testing.md

Last updated: 2026-09-05 (added "Switching the Coach Model" — v4.53 Runtime Coach Model Control, OBJ-3385 — replacing the old edit-LiteLLM-config-and-redeploy procedure with the admin-app runtime switch; refreshed same day for OBJ-3392, which closed while this PR sat unmerged — both write and read paths now gate on Ai::ModelCatalog.routable?, not allowlisted? alone, and a second model, gemini/gemini-2.5-pro, is now live-priced)

Loading…