Objectuve Docs

Appearance

v1.0 — Dedicated AI Service ​

Every AI call in Objectuve now flows through a dedicated LiteLLM proxy — local, production, rate-limited, cost-capped, and monitored.

Summary ​

Objectuve's first major AI integration shipped inside Rails as an in-process langchainrb + Gemini SDK dependency. It worked for demos, but it tied LLM behavior to the Rails deploy cycle, made per-feature cost attribution impossible, and left no way to kill an AI feature without a full deploy.

v1.0 extracted that surface into a dedicated LiteLLM proxy service — first as a Docker Compose service for local development, then as a private Cloud Run service for production with OIDC-authenticated service-to-service calls. All six AI feature categories (coaching, milestones, insights, check-ins, description refinement, moderation) now route through Ai::ServiceClient with virtual-key isolation, budget caps, per-user rate limits, and response caching. The old langchainrb gem and every Gemini-specific code path were removed. Admins can see total spend, per-feature breakdown, and per-model distribution from a dashboard, and a Slack alert fires when spend crosses 80% of the monthly budget.

This milestone was the operational foundation every subsequent AI milestone (Meet Coach, AI Workforce) built on.

Goal ​

Extract the in-process LangChain/Gemini integration into a standalone LiteLLM proxy service. Flow in three delivery boundaries: build and test everything locally first (no GCP changes), then deploy to production with a feature-flag-controlled staged rollout, then enforce cost controls and clean up the old code path once the new path has proven stable for 7 days.

Scope — What Shipped ​

  • LiteLLM proxy service as a Docker Compose service for local dev and a private Cloud Run service for staging + production.
  • Ai::ServiceClient HTTP wrapper with WebMock-backed specs; all AI calls route through it.
  • 6 versioned prompt modules with a shared Base preamble — coaching, milestones, insights, check-ins, description, moderation.
  • Settings.ai.use_litellm feature flag controls cutover; flipped true permanently after 7-day soak.
  • OIDC service-to-service auth — Rails → LiteLLM via GCP IAM identity tokens.
  • Virtual keys per feature category (sk-coaching, sk-moderation, sk-batch) — disabling one halts calls for that category instantly.
  • Per-user rate limits (10/min coaching, 5/min milestones) returning 429 on excess.
  • Response caching for milestone and description features (1-hour TTL).
  • Admin dashboard view for token usage, cost, and model distribution (shipped later into the v1.3 admin dashboard).
  • Slack budget alert at 80% of monthly spend via AiMetrics::AiBudgetAlertJob.
  • Removed: langchainrb gem, legacy CoachService Gemini code paths, ScreenContentJob instance_variable_get hack.
  • Added: LiteLLM reachability in /health endpoint; operations runbook at docs/operations/ai-runbook.md.

Phases ​

PhaseNameStatusPlansHighlights
1Local FoundationShipped5LiteLLM in Docker Compose, Ai::ServiceClient, 6 prompt modules, feature flag, full test coverage
2Production DeploymentShipped4LiteLLM Cloud Run service, OIDC auth, staged rollout, zero-downtime cutover
3Cost & Rate ControlShipped5Virtual keys, per-user rate limits, response caching, kill switches
4Operational MaturityShipped5langchainrb removal, admin dashboard, budget alerts, AI runbook

Key Decisions ​

  • LiteLLM over OpenAI gateway or DIY proxy — LiteLLM gave us multi-model routing, virtual-key rate limiting, and response caching out of the box without writing any of it.
  • Private Cloud Run service, OIDC-only — not public-internet reachable; every call is authenticated with an identity token verified by the receiving service.
  • Three settings-file toggles, not per-feature DB flags — ai.coaching_enabled, ai.moderation_enabled, ai.batch_enabled. Flipping requires a deploy (intentional — AI changes are production changes). Runtime flag UI deferred to a later milestone.
  • Feature flag gates use of the new client, not the old path — we shipped both paths in parallel for a week, then flipped the flag, then deleted the old path.

Requirements Coverage ​

All 43 v1 requirements satisfied across 4 phases. Categories: PROX-* (LiteLLM proxy), RAIL-* (Rails client), DEPL-* (deployment), COST-* (cost controls), OPS-* (operations), ADMN-* (admin surface), TEST-* (testing).

Full list: v1.0 in ROADMAP.md.

Outcomes ​

  • Single call site. Every AI interaction in Rails goes through Ai::ServiceClient. Adding a new feature or swapping a model is a one-line change, not a dependency migration.
  • Observability. Per-feature cost and token counts are visible to admins in near-real-time.
  • Kill switches. Disabling a feature category or a virtual key halts calls within seconds — no deploy required for the virtual-key path.
  • Foundation unlocked. v1.1 (auth hardening), v1.2 (Meet Coach personalization), and v1.4 (AI Workforce) all built directly on Ai::ServiceClient and the prompt-module pattern shipped here.
  • 2da27c561 — chore: complete v1.0 Dedicated AI Service milestone
  • 585182c45 — Add Dedicated AI Service PRD (LiteLLM extraction)

Last updated: 2026-05-23

Loading…