Objectuve Docs

Appearance

v4.53 — Runtime Coach Model Control ​

An operator switches the Gemini model behind Coach in the admin app — no deploy, no LiteLLM restart — and the cost ledger follows the switch automatically.

Summary ​

Before this milestone, changing the Gemini model behind Coach meant editing the model: value for six aliases across three LiteLLM YAML configs (config.production.yaml, config.staging.yaml, config.local.yaml), hand-syncing Ai::Pricing's billing rates — which were keyed by alias, not model id, with no mechanism forcing the two to agree — and redeploying both LiteLLM and Rails. The model identity was replicated across four independent places with no way to keep them consistent, which is how master could carry a live Gemini 3.8 Flash pin while Ai::Pricing still billed at the old 2.5 Flash rate for 33 minutes during one release.

Six phases closed that gap. Phase 1 severed the alias-to-rate coupling by introducing Ai::ModelCatalog, a single source of truth for the Gemini allowlist and an effective-dated rate schedule, and encoded the Gemini 3.8 Flash introductory-rate cliff (a real, published, dated pricing change on 2027-01-01) for the first time. Phase 2 made the model a runtime app_settings value, resolved once per request through a 60-second cache with a fail-safe default, and gave LiteLLM a gemini/* wildcard route so it never needs a config edit for a model change again. Phase 3 built the admin-gated GraphQL surface. Phase 4 designed the admin card, adding an unplanned "nothing to switch to right now" state that turned out to matter. Phase 5 shipped CoachModelPanel.vue next to the existing AI usage panel. Phase 6 documented the new operator procedure — and caught, mid-review, that a concurrent correctness fix (OBJ-3392) had changed the exact semantics the docs described, refreshing the claims before merge rather than shipping a runbook that told operators a fixed bug was still open.

That correctness fix is part of this milestone's story, not a footnote. The original plan assumed co-locating the allowlist and the rate table in one class would prevent a model from silently pricing at $0.00. It didn't — the allowlist conflates "ids Coach may run" with "ids we can still price historically," so gemini/gemini-2.5-flash stayed allowlisted after its own rate window closed, and the write guard checked allowlist membership alone. OBJ-3392 introduced Ai::ModelCatalog.routable? — allowlisted and carrying a rate window that covers right now — as the real gate on both the write and read paths, and added gemini/gemini-2.5-pro with a live rate. That second routable model is what makes this milestone's headline claim — an operator has something real to switch to — genuinely true today, not just true in principle.

Goal ​

Make the Gemini model behind Coach an operator decision made in the admin app, not a three-file code change plus two deploys — and make the cost ledger follow the switch automatically, so a model change can never silently mis-price itself.

Scope — What Shipped ​

  • Ai::ModelCatalog — the Gemini model allowlist plus an effective-dated rate schedule (rails_api/app/services/ai/model_catalog.rb), replacing a flat, undated rate table.
  • resolved_model column on ai_usage_events, populated alongside the existing model (alias) column — AdminAiUsageService#per_resolved_model is additive; #per_model is unchanged.
  • app_settings table and Ai::CoachModel — a 60-second-cache-TTL runtime read of the operator-chosen Gemini model, failing safe to Settings.ai.default_coach_model on a missing row, a non-routable value, or any read error.
  • Ai::SetCoachModel — the validated, audited write path (Admin::Logged, action_type: 'UPDATE_COACH_MODEL'), rejecting any id that isn't routable? before writing anything.
  • A gemini/* wildcard route in all four LiteLLM configs, so Rails can route to any routable Gemini id without a LiteLLM config change.
  • Admin GraphQL surface: coachModelSetting query (current model, allowlisted options with rates, last 5 changes) and updateCoachModel mutation, both require_admin!-gated.
  • CoachModelPanel.vue + Storybook stories, mounted in admin_dashboard's MonitoringView.vue alongside the existing AI usage panel.
  • Ai::ModelCatalog.routable? (OBJ-3392) — closes the allowlisted-but-unpriced gap and adds gemini/gemini-2.5-pro as a second live-priced, switchable model.
  • Documentation: docs/operations/ai-runbook.md's "Switching the Coach Model" section replaces the old edit-and-deploy procedure; .claude/skills/litellm/SKILL.md and docs/development/testing-coach-locally.md updated for the runtime model and the wildcard route.

Phases ​

PhaseNameStatusPlansHighlights
1Model-accurate cost attributionShipped1Ai::ModelCatalog + effective-dated rates + resolved_model column, no behaviour change (Codi)
2Runtime setting + alias resolution + LiteLLM passthroughShipped1app_settings-backed Ai::CoachModel/Ai::SetCoachModel, gemini/* wildcard, SET-05 single-resolution fix (Codi)
3Admin GraphQL surfaceShipped1coachModelSetting query + updateCoachModel mutation, both admin-gated (Codi)
4UI-SPEC for the Coach model cardShipped1Design contract + the load-bearing "nothing to switch to" state (Desi)
5CoachModelPanel.vue + Storybook + wire-upShipped1Admin component mounted beside AiUsagePanel, GraphQL-collision with Phase 3 resolved (Codi)
6DocsShipped1Runbook, LiteLLM skill, local-testing docs, CHANGELOG — refreshed mid-review for OBJ-3392 (Dori)

Key Decisions ​

  • Rails resolves the model once, per request; LiteLLM never picks a model (D-01) — /model/update would require a Postgres-backed LiteLLM deployment this repo doesn't run, so the routing decision stays entirely in Rails, sent as an explicit model: string.
  • LiteLLM gets one gemini/* wildcard route instead of pre-registering every allowlisted id (D-02) — trades "any arbitrary model id with zero deploys" for "switch among already-allowlisted models with zero deploys." A genuinely new model still needs a one-file Rails PR to add its allowlist entry and its rate — a deliberate guard against silently pricing an unverified model at $0.00.
  • One setting governs all six Gemini aliases, including content moderation (D-03) — the issue's non-goals ruled out per-alias controls; switching "the Coach model" also switches what moderates content, and the admin UI and runbook both say so explicitly.
  • ai_usage_events.model keeps its pre-milestone meaning; resolved_model is additive (D-04) — prevents every historical per-model chart from silently splitting in half at the switchover date.
  • routable?, not allowlisted?, is the real gate on both write and read paths (OBJ-3392) — allowlisted-for-historical-pricing and safe-to-route-to-now are different facts, and conflating them is exactly how a lapsed model could have kept accepting traffic at $0.00.
  • The admin card degrades honestly when fewer than two models are routable — an unplanned Phase 4 addition (options.length <= 1) that made it safe to ship Phase 5 before a second routable model existed, rather than shipping a broken one-entry dropdown.

Requirements Coverage ​

40/40 phase-checklist items satisfied (this milestone has no standalone REQUIREMENTS.md; see the Milestone Audit for the full item-by-item verification).

CategoryCountStatus
COST-* (Phase 1 — cost attribution)9All satisfied
SET-* (Phase 2 — runtime setting)10All satisfied
ADMIN-* (Phase 3 — GraphQL surface)6All satisfied
UI-* (Phase 4 — UI-SPEC)5All satisfied
Phase 5 (component + wire-up)5All satisfied
DOC-* (Phase 6 — docs)5All satisfied

Full phase-checklist detail: v4.53-runtime-coach-model-control-ROADMAP.md on GitHub.

Outcomes ​

A super admin can open the admin dashboard's Monitoring view, see the Coach model currently running and its per-token rate, switch to another routable model, and see that reflected in the next Coach request's ledger entry within about a minute — with no deploy and no LiteLLM restart. A non-routable id (unallowlisted, or allowlisted but its rate window has lapsed) is rejected server-side before anything is written. A missing or corrupted setting fails safe to the YAML default rather than breaking Coach. The historical per-model cost chart is unchanged; a new per-resolved-model view sits alongside it. Rate changes with known future effective dates — like Gemini 3.8 Flash's 2027-01-01 price increase — are now expressible as data, not a future code change.

Tech Debt ​

  • (Phase 1) workforce/content and workforce/image have no entry in Ai::Pricing::RATES_CENTS_PER_MILLION and book at $0.00 today — the same silent-zero failure class this milestone fixed for the Gemini Coach aliases, deliberately deferred to its own issue because pricing them is a behaviour change, not a refactor.
  • (Phase 3) Resolvers::AdminQueries.coach_model_options duplicates routable?'s filter logic inline rather than calling it directly — functionally identical today, but a future change to routable?'s definition would need to be mirrored by hand at this call site.
  • 7c01458bf — Phase 1: model-accurate cost attribution, Ai::ModelCatalog (PR #2914)
  • 8a0806903 — Phase 2: runtime setting + alias resolution + LiteLLM passthrough (PR #2918)
  • 7b2a7ac40 — Phase 3: admin GraphQL surface (PR #2919)
  • b5ee422a2 — Phases 4+5: UI-SPEC + CoachModelPanel.vue + Storybook (PR #2921)
  • 9ff861fb7 — OBJ-3392: gate Coach model writes/reads on routable?, not allowlisted? (PR #2922)
  • 293545718 — Phase 6: docs, refreshed for OBJ-3392 mid-review (PR #2923)

Last updated: 2026-09-05

Loading…