Dedicated AI Service -- Feature Exploration & PRD

Product: Objectuve (codename: Enkidu) Feature: Dedicated AI Service -- LiteLLM proxy extraction Status: ✅ Shipped (v1.0) Date: April 2026

1. Problem Statement

Why Extract AI Now?

The Meet Coach PRD (approved April 2026) introduces 6 coaching personas, a 5-point tone spectrum, 4 focus area priorities, and 5 communication frequency presets. This creates a combinatorial prompt space of 120+ personality combinations that must be injected into every CoachService method. Building this on the current architecture will break the monolith.

What Breaks If We Don't

1. CoachService becomes unmaintainable. Currently 179 lines with 5 public methods, a provider cascade in initialize_llm, and Gemini-specific format branching in 4 places. Meet Coach requires adding build_personality_preamble(user) and a user: keyword to every method. Each method already contains inline prompt text, JSON parsing, and provider-specific formatting. After Meet Coach, CoachService will be 300+ lines of tangled prompt templates, JSON parsing, provider branching, and business logic.

2. Content moderation is held together with duct tape. ScreenContentJob (line 59) calls coach.instance_variable_get(:@llm) to steal the LLM instance because there's no public way to access it. This breaks when Meet Coach changes CoachService's internals. Moderation and coaching have completely different requirements: moderation should use the cheapest model; coaching benefits from expressive, expensive models. Currently they share the same provider with no differentiation.

3. Costs are invisible and uncontrollable. Every Ai::CoachService.new creates a fresh LLM client with no connection pooling. GenerateAiCheckInPromptsJob iterates over all eligible users -- hundreds of LLM calls per run. Zero token counting, zero cost tracking, zero rate limiting, zero budget enforcement. As Meet Coach drives engagement (target: 2+ weekly interactions per user), costs scale invisibly.

4. Provider lock-in via langchainrb. Google Gemini's message format (parts vs content) is handled in 4 separate inline conditionals. Adding any new provider means touching every method. Langchainrb provides no fallback, routing, or caching.

5. No operational levers. Swapping models requires changing an env var and redeploying the entire Rails API. No fallback if a provider goes down. No way to A/B test models. No cache for repeated prompts. No circuit breaker.

What "Good" Looks Like After Extraction

Rails treats AI as an HTTP API call. Prompts are managed as versioned template modules. Models are assigned per-capability. Costs are tracked per-user. Providers swap without deployment. Moderation runs on a cheap model independently. Meet Coach's 120+ prompt combinations are handled by template composition, not code branching.

2. Goals & Success Metrics

Primary Goals

Unblock Meet Coach -- architecture that handles persona/tone/focus prompt composition cleanly
Separate coaching from moderation -- independent model selection and cost profiles
Add cost visibility -- per-user, per-feature token and spend tracking
Remove provider lock-in -- single integration point, no format branching in app code
Enable operational control -- model swapping, fallbacks, rate limiting without Rails redeployment

Success Metrics

Metric	Target	Measurement
CoachService line count after Meet Coach	Under 60 lines (HTTP wrapper only)	Code review
Time to swap default model	Under 5 minutes, no deploy	Operational test
Per-user cost visibility	100% of LLM calls tracked with user_id and feature	LiteLLM dashboard
Provider outage impact	Automatic fallback within 30 seconds	Synthetic monitor
Content moderation cost	50%+ reduction via cheaper model routing	LiteLLM spend report
AI latency p95	No regression vs. current (under 3s for coaching)	Sentry APM
Local dev experience	Works without API keys via mock mode	Developer testing

Guardrails

Zero downtime during migration -- feature flag controls old vs. new path
No user-facing behavior changes in Phase 1 (identical prompts, identical responses)
Rails API latency must not increase by more than 50ms (HTTP hop overhead)
LiteLLM must not become a single point of failure -- graceful degradation required

3. Architecture Design

3a. Service Architecture

Deployment Topology

                                    +------------------+
                                    |  Firebase Hosting |
                                    |  (Vue 3 Frontend) |
                                    +--------+---------+
                                             |
                                             v
+-------------------+    HTTP/JSON    +------+---------+    SQL/Redis    +-----------+
|   LiteLLM Proxy   | <------------- |  Rails API     | ------------> | Cloud SQL  |
|   (Cloud Run)     |                 |  (Cloud Run)   |               | PostgreSQL |
|                   |                 |                |               +-----------+
|  - Model routing  |                 |  - GraphQL     |
|  - Cost tracking  |                 |  - Interactions|    +----------+
|  - Rate limiting  |                 |  - Sidekiq jobs|    | Redis 7  |
|  - Fallbacks     |                 +----------------+    | (Shared) |
|  - Caching       |                                       +----------+
+--------+----------+
         |
         v  (upstream LLM calls)
+--------+----------+
| LLM Providers     |
| - OpenAI          |
| - Anthropic       |
| - Google Gemini   |
| - Ollama (dev)    |
+-------------------+

LiteLLM runs as a separate Cloud Run service (enkidu-ai-proxy-production / enkidu-ai-proxy-staging) in the same GCP project and region (us-central1). Rails communicates via internal Cloud Run service-to-service HTTP calls using IAM-based authentication (no public endpoint).

Why a separate Cloud Run service:

LiteLLM is Python; Rails is Ruby -- no in-process option
Service-to-service calls within same region add ~5-15ms -- acceptable given LLM calls take 500ms-3s
Independent scaling based on LLM request volume
Independent deployment for model config changes

Database: New database litellm_production / litellm_staging on the existing Cloud SQL instance. No new infrastructure cost. LiteLLM tables isolated by database.

Redis: Share existing Memorystore Redis using a different database number (redis://10.234.96.43:6379/2 -- Rails uses /1). LiteLLM Redis usage is lightweight (TTL cache, rate limit counters).

Docker Compose Addition (Local Development)

yaml

litellm:
  image: ghcr.io/berriai/litellm:main-latest
  ports:
    - "4000:4000"
  volumes:
    - ./litellm/config.yaml:/app/config.yaml
  environment:
    LITELLM_MASTER_KEY: "sk-dev-master-key"
    DATABASE_URL: "postgresql://postgres@db:5432/litellm_development"
    REDIS_HOST: redis
    REDIS_PORT: 6379
    REDIS_DB: 2
  depends_on:
    - db
    - redis
  command: ["--config", "/app/config.yaml", "--port", "4000"]

Rails gains LITELLM_BASE_URL: http://litellm:4000 and LITELLM_API_KEY: sk-dev-master-key.

3b. Prompt Management

Where Prompts Live

Layer 1: System-level config -- in LiteLLM (YAML). Model assignment per capability, fallback chains, rate limits, cache TTL per route.

Layer 2: Dynamic prompt content -- constructed in Rails. Actual prompt text (user context, persona preamble, tone modifier, focus instructions) assembled in Rails and sent as the messages array. This is where Meet Coach's prompt composition lives.

Why this split: LiteLLM is a proxy, not a prompt engine. It controls which model handles the request, whether it's cached, and how it's rate-limited. Prompt text must be constructed by the caller.

Prompt Modules in Rails

rails_api/app/services/ai/
  prompts/
    coaching.rb          # Persona preamble, tone modifiers, focus instructions
    milestones.rb        # Milestone generation prompt
    insights.rb          # Insight card generation prompt
    check_ins.rb         # Check-in notification prompt
    description.rb       # Goal description refinement prompt
    moderation.rb        # Content screening prompt
  service_client.rb      # HTTP client for LiteLLM

Each module defines a VERSION constant (e.g., "coaching-v2") sent as metadata on every request for tracking.

How Meet Coach Personas Map to Prompt Templates

Ai::Prompts::Coaching contains:

Persona descriptions -- hash mapping identifiers to behavioral instruction blocks
Tone modifiers -- 5 instruction blocks adjusting intensity
Focus instructions -- primary and secondary text blocks
Core rules -- universal guardrails (2 sentences max, no cliches, no guilt)
build_preamble(preferences) -- composes all into a single system message

3c. Model Routing

Per-Capability Model Assignment

LiteLLM config defines model aliases that abstract the provider/model:

yaml

model_list:
  - model_name: coaching/default
    litellm_params:
      model: anthropic/claude-sonnet-4-20250514
      api_key: os.environ/ANTHROPIC_API_KEY

  - model_name: coaching/default       # fallback
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: moderation/screen
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: coaching/milestones
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

Capability	Model Alias	Why
`get_advice`	`coaching/default`	Highest-quality; persona fidelity matters
`generate_insight`	`coaching/default`	Needs personality + structured JSON
`generate_check_in_prompt`	`coaching/default`	Short but persona-critical
`generate_milestones`	`coaching/milestones`	Structured output; cheaper model handles well
`refine_description`	`coaching/milestones`	Text transformation; no personality needed
`screen_content`	`moderation/screen`	Classification task; cheapest viable model

Fallback Chains

Multiple entries with the same model_name create automatic fallback. LiteLLM tries the next on 429 (rate limit), 500, or timeout.

A/B Testing Models

LiteLLM supports weighted load balancing:

yaml

- model_name: coaching/default
  litellm_params:
    model: anthropic/claude-sonnet-4-20250514
    weight: 80          # 80% of traffic

- model_name: coaching/default
  litellm_params:
    model: openai/gpt-4o
    weight: 20          # 20% of traffic

Rails sends model: "coaching/default". LiteLLM routes by weight and logs which model was used. No app code changes for A/B tests.

4. Migration Plan

New Rails Files

File	Purpose
`app/services/ai/service_client.rb`	Faraday HTTP client wrapping LiteLLM
`app/services/ai/prompts/coaching.rb`	Persona preamble, tone, focus composition
`app/services/ai/prompts/milestones.rb`	Milestone generation template
`app/services/ai/prompts/insights.rb`	Insight card template
`app/services/ai/prompts/check_ins.rb`	Check-in template
`app/services/ai/prompts/description.rb`	Description refinement template
`app/services/ai/prompts/moderation.rb`	Content screening template
`config/initializers/ai_service.rb`	Singleton client config

Ai::ServiceClient Design

ruby

module Ai
  class ServiceClient
    def chat(model:, messages:, metadata: {}, temperature: 0.7, max_tokens: 500)
      response = @conn.post('/v1/chat/completions') do |req|
        req.headers['Authorization'] = "Bearer #{@api_key}"
        req.body = {
          model: model,
          messages: messages,
          temperature: temperature,
          max_tokens: max_tokens,
          metadata: metadata
        }
      end
      handle_response(response)
    end
  end
end

One HTTP call. No langchainrb. No provider branching. No Gemini format handling.

What Happens to CoachService

Replaced. The new version:

Accepts user: keyword on every public method
Delegates prompt construction to Ai::Prompts::*
Calls Ai::ServiceClient.chat() with appropriate model alias
Parses the response
Contains zero LLM provider knowledge

Estimated: 40-60 lines. Down from 179 today, far simpler than the 300+ lines it would become under Meet Coach.

GraphQL Mutation Changes

Minimal -- the 4 mutations continue creating Ai::CoachService.new. Only change: pass current_user:

ruby

# After
service = ::Ai::CoachService.new
advice = service.get_advice(user_context, user: context[:current_user])

Background Job Changes

GenerateAiCheckInPromptsJob: Passes user coaching preferences into the call
ScreenContentJob: Calls Ai::ServiceClient directly with moderation/screen. The instance_variable_get hack is eliminated entirely

Backward Compatibility

Feature flag (Settings.ai.use_litellm) controls which path:

Deploy LiteLLM to staging
Enable flag on staging, verify identical behavior
Enable flag on production
Remove old code path and langchainrb gem

5. Cost & Usage Tracking

Per-User Token Tracking

Every request includes metadata:

json

{
  "model": "coaching/default",
  "messages": [...],
  "metadata": {
    "user_id": "usr_abc123",
    "feature": "get_advice",
    "prompt_version": "coaching-v2",
    "persona": "spark",
    "tone": 3
  }
}

LiteLLM logs this with token counts and cost to its PostgreSQL database.

Virtual Keys

Virtual Key	Scope	Budget	Purpose
`sk-coaching`	Coaching features	$500/month	All coaching model calls
`sk-moderation`	Content moderation	$50/month	Screening calls only
`sk-batch`	Background jobs	$200/month	Check-in generation

Rate Limiting Strategy

Scope	Limit	Rationale
Per-user RPM (coaching)	10 req/min	Normal usage is 1-2/min
Per-user RPM (milestones)	5 req/min	Less frequent
Global RPM (all coaching)	200 req/min	Protects provider limits
Global RPM (moderation)	100 req/min	Async, can queue
Per-user daily token budget	50,000 tokens	Prevents runaway costs

6. Operational Controls

Model Swapping Without Deploy

Update LiteLLM config and redeploy only the proxy (30-second Cloud Run deploy). Or use LiteLLM's runtime API:

bash

curl -X POST https://ai-proxy.internal/model/update \
  -H "Authorization: Bearer $MASTER_KEY" \
  -d '{"model_name": "coaching/default", "litellm_params": {"model": "openai/gpt-4o"}}'

Kill Switch (Three Levels)

LiteLLM: Disable a virtual key. Immediate 429 for all calls using that key.
Rails: Settings.ai.coaching_enabled flag. Mutations return fallbacks without calling LiteLLM.
Frontend: Feature flags hide Coach UI entirely.

Health Checks

LiteLLM exposes GET /health. Rails' GET /health extends to check LiteLLM reachability.

Monitoring & Alerting

Signal	Alert Threshold
LiteLLM error rate	>5% of requests in 5 min
LiteLLM latency p95	>5 seconds
Provider fallback triggered	Any occurrence (info)
Budget threshold crossed	80% and 95%
Virtual key exhausted	Immediate
Service unhealthy	2 consecutive failures

7. Meet Coach Integration

Preference Flow

User selects persona/tone/focus in MeetCoach.vue
  -> UpdateCoachingPreferences mutation
  -> UserDetail.data JSONB updated
  -> (later) GetAdvice mutation called
  -> Mutation passes context[:current_user] to CoachService
  -> CoachService reads user.coaching_preferences
  -> Ai::Prompts::Coaching.build_preamble(preferences) constructs system message
  -> Ai::ServiceClient.chat(model: "coaching/default", messages: [system, user])
  -> LiteLLM routes to Claude with metadata {persona: "spark", tone: 3}
  -> Response returned

Preamble Constructed in Rails, Not LiteLLM

LiteLLM is a proxy, not a prompt engine. Persona preferences live in Rails' database. Prompt construction is testable with RSpec.

A/B Testing Personas

Two approaches:

Prompt variants: Rails selects between prompt text versions, tracks via prompt_variant metadata
Model variants: LiteLLM weighted routing. Same prompt, different model. Config change only.

8. Content Moderation

Separate Model and Budget

Moderation uses moderation/screen alias routed to GPT-4o-mini (~$0.15/M input tokens vs Claude Sonnet ~$3/M). Separate virtual key (sk-moderation) with its own budget.

json

{
  "model": "moderation/screen",
  "messages": [{"role": "user", "content": "<screening prompt>"}],
  "temperature": 0.1,
  "max_tokens": 200,
  "metadata": {"feature": "content_moderation"}
}

ScreenContentJob After Migration

ruby

def screen_with_ai(content)
  prompt = Ai::Prompts::Moderation.build(content: content.truncate(1000))
  response = Ai::ServiceClient.instance.chat(
    model: 'moderation/screen',
    messages: [{ role: 'user', content: prompt }],
    temperature: 0.1,
    max_tokens: 200,
    metadata: { feature: 'content_moderation' }
  )
  parse_ai_response(response.dig('choices', 0, 'message', 'content'))
end

The instance_variable_get hack, Gemini format branching, and CoachService dependency are all gone.

Cost Profile

~$0.00003 per moderation call. At 10,000 calls/month: ~$0.30/month. Effectively free.

9. Security

API Key Management

Rails to LiteLLM: GCP IAM service-to-service auth (Cloud Run OIDC identity tokens). No shared API keys.
LiteLLM to providers: Provider API keys in GCP Secret Manager, injected as env vars. Never touch Rails.
LiteLLM master key: Admin API only. Stored in Secret Manager. Not used by Rails.

Prompt Injection Defense

Input sanitization in Ai::ServiceClient before sending
System/user role separation (persona in system, user content in user)
Strict JSON schema validation on structured outputs
Content truncation before prompt inclusion

PII in Prompts

Current prompts include: user first name, goal names, descriptions, streak counts, milestone names, mood data, free-text content. Mitigations:

LiteLLM request logging configured to redact PII
Provider DPAs in place
Privacy policy discloses AI processing
Consider user-level AI opt-out

Audit Logging

LiteLLM logs every request with: timestamp, model, token counts, cost, latency, user_id, feature. Stored in PostgreSQL with 90-day retention.

10. Local Development

Mock Mode (Replaces MockLlm)

LiteLLM supports a fake/ model prefix returning deterministic responses:

yaml

# config.dev.yaml
model_list:
  - model_name: coaching/default
    litellm_params:
      model: fake/gpt-4o
  - model_name: moderation/screen
    litellm_params:
      model: fake/gpt-4o-mini

docker compose up starts LiteLLM with mock mode. All AI features work without API keys.

Testing

RSpec tests use WebMock/VCR to stub Ai::ServiceClient HTTP calls (no LiteLLM process needed)
Integration tests can optionally start LiteLLM via Docker Compose

11. Edge Cases & Open Questions

Edge Cases

#	Scenario	Resolution
EC-1	LiteLLM service down	`ServiceClient` catches connection errors, returns fallback responses
EC-2	All upstream providers down	LiteLLM exhausts fallback chain, returns 503. Rails returns fallback UI
EC-3	User exceeds rate limit	LiteLLM returns 429. Rails shows "Coach is busy" message
EC-4	Budget exhausted mid-month	429 for that virtual key. Moderation budget should have higher buffer
EC-5	LiteLLM config has invalid YAML	Fails to start. Cloud Run health check fails. Previous revision serves
EC-6	Feature flag on but LiteLLM not deployed	Connection refused. Flag should check connectivity first
EC-7	Redis under pressure from Sidekiq	LiteLLM keys are small + TTL-bounded. Low risk but monitor

Open Questions

LiteLLM versioning: Pin to monthly release tag for stability? Update quarterly?
Public endpoint: No for MVP. Later: evaluate if frontend needs direct streaming for Coach chat.
Streaming responses: Current Coach modal fetches complete responses. Meet Coach may benefit from SSE streaming for longer persona-driven responses. Defer to post-MVP.
Multi-turn conversation memory: Session-scoped in frontend composable. Server-side persistence is a Rails feature, not LiteLLM. Out of scope.
LiteLLM vs. alternatives: Why not a thin proxy ourselves? LiteLLM provides cost tracking, rate limiting, fallbacks, 100+ providers out of the box. Months to build. Risk: open-source dependency. Mitigated by: widely adopted, standard OpenAI protocol (easy to replace), forkable.
Shared Cloud SQL instance: Recommended for MVP. Revisit if AI volume exceeds 10,000 requests/day.
Prompt caching: Milestones and descriptions are good candidates (similar goals). Personalized advice is not (every context differs). Enable per-route.

12. Implementation Phases

Phase 1: Minimum Viable Proxy (1 sprint)

Unblocks Meet Coach. Local dev only.

litellm/ directory with config.yaml and config.dev.yaml
LiteLLM in docker-compose.yml
Ai::ServiceClient (Faraday HTTP wrapper)
Ai::Prompts::* modules (all 6: coaching, milestones, insights, check_ins, description, moderation)
Rewritten Ai::CoachService using ServiceClient and prompt modules
Rewritten ScreenContentJob using ServiceClient directly
Feature flag with old path preserved
RSpec tests for ServiceClient, prompt modules, rewritten service
langchainrb retained but unused when flag is on

Phase 2: Production Deployment + Cost Tracking (1 sprint)

Cloud Run service, Cloud SQL database, Secret Manager entries
GitHub Actions workflow for LiteLLM deployment
IAM service-to-service auth
Virtual keys with budget configuration
Slack webhook budget alerts
LiteLLM admin UI behind IAP
Health check extension
Enable on staging, validate, enable on production
Smoke tests updated

Phase 3: Operational Maturity (1 sprint)

Per-user rate limiting
Response caching for milestones and descriptions
Sentry integration for LiteLLM
Model A/B testing configuration
Remove feature flag and langchainrb gem
PII redaction in request logging
Operational runbook

Phase 4: Advanced Capabilities (future)

Semantic prompt caching for insights
Streaming responses for Coach chat via SSE
Dedicated Cloud SQL instance (if volume warrants)
Fine-tuned model evaluation pipeline
Prompt A/B testing framework

13. Cost Analysis

Current Estimated Costs

Feature	Calls/Month	Model (assumed)	Cost
get_advice	500-1,000	Gemini Flash	$0.50-1.00
generate_insight	2,000-5,000	Gemini Flash	$1.00-2.50
generate_milestones	200-500	Gemini Flash	$0.20-0.50
refine_description	100-300	Gemini Flash	$0.10-0.30
generate_check_in_prompt	500-2,000/day	Gemini Flash	$5.00-20.00
screen_content	200-1,000	Gemini Flash	$0.10-0.50
Total			$7-25/month

Post-Meet Coach Projection (500 active users)

Feature	Calls/Month	Model	Cost
get_advice	4,000+	Claude Sonnet	$12-20
generate_insight	10,000+	Claude Sonnet	$30-50
generate_check_in_prompt	15,000+	Claude Sonnet	$45-75
generate_milestones	500	GPT-4o-mini	$0.50
refine_description	300	GPT-4o-mini	$0.30
screen_content	2,000	GPT-4o-mini	$0.20
Total			$90-150/month

How LiteLLM Reduces Costs

Model routing: Milestones, descriptions, moderation on GPT-4o-mini ($0.15/M input) instead of Claude Sonnet ($3/M). ~40-50% savings on those calls.
Response caching: 10-20% cache hit rate on milestones/descriptions. ~$5-15/month savings.
Budget enforcement: Hard caps prevent runaway costs from bugs.
Token visibility: Per-prompt-version tracking reveals wasteful prompts.

LiteLLM Infrastructure Cost

Resource	Monthly Cost
Cloud Run (min 0, max 2 instances)	$5-15
Cloud SQL (shared instance)	$0 incremental
Redis (shared instance)	$0 incremental
Total overhead	$5-15/month

Negligible compared to the LLM API costs it manages.

Critical Files

File	Change
`rails_api/app/services/ai/coach_service.rb`	179-line file being replaced with 40-60 line HTTP wrapper
`rails_api/app/jobs/content_moderation/screen_content_job.rb`	`instance_variable_get` hack eliminated; clean ServiceClient call
`rails_api/Gemfile`	`langchainrb` removed (Phase 3)
`docker-compose.yml`	Gains `litellm` service
`rails_api/app/graphql/mutations/ai/`	Pass `current_user` for persona preferences
`rails_api/app/jobs/ai_coaching/generate_ai_check_in_prompts_job.rb`	Uses ServiceClient, passes user preferences
`docs/product/meet-coach-prd.md`	Companion PRD driving extraction timeline

Coach Features — feature doc for the shipped implementation

Last updated: 2026-07-16 — Phase 100 audit. PRD for shipped feature (v1.0) archived to completed/.

Dedicated AI Service -- Feature Exploration & PRD ​

1. Problem Statement ​

Why Extract AI Now? ​

What Breaks If We Don't ​

What "Good" Looks Like After Extraction ​

2. Goals & Success Metrics ​

Primary Goals ​

Success Metrics ​

Guardrails ​

3. Architecture Design ​

3a. Service Architecture ​

Deployment Topology ​

Docker Compose Addition (Local Development) ​

3b. Prompt Management ​

Where Prompts Live ​

Prompt Modules in Rails ​

How Meet Coach Personas Map to Prompt Templates ​

3c. Model Routing ​

Per-Capability Model Assignment ​

Fallback Chains ​

A/B Testing Models ​

4. Migration Plan ​

New Rails Files ​

Ai::ServiceClient Design ​

What Happens to CoachService ​

GraphQL Mutation Changes ​

Background Job Changes ​

Backward Compatibility ​

5. Cost & Usage Tracking ​

Per-User Token Tracking ​

Virtual Keys ​

Rate Limiting Strategy ​

6. Operational Controls ​

Model Swapping Without Deploy ​

Kill Switch (Three Levels) ​

Health Checks ​

Monitoring & Alerting ​

7. Meet Coach Integration ​

Preference Flow ​

Preamble Constructed in Rails, Not LiteLLM ​

A/B Testing Personas ​

8. Content Moderation ​

Separate Model and Budget ​

ScreenContentJob After Migration ​

Cost Profile ​

9. Security ​

API Key Management ​

Prompt Injection Defense ​

PII in Prompts ​

Audit Logging ​

10. Local Development ​

Mock Mode (Replaces MockLlm) ​

Testing ​

11. Edge Cases & Open Questions ​

Edge Cases ​

Open Questions ​

12. Implementation Phases ​

Phase 1: Minimum Viable Proxy (1 sprint) ​

Phase 2: Production Deployment + Cost Tracking (1 sprint) ​

Phase 3: Operational Maturity (1 sprint) ​

Phase 4: Advanced Capabilities (future) ​

13. Cost Analysis ​

Current Estimated Costs ​

Post-Meet Coach Projection (500 active users) ​

How LiteLLM Reduces Costs ​

LiteLLM Infrastructure Cost ​

Critical Files ​

Related Documentation ​

Objectuve Docs

Dedicated AI Service -- Feature Exploration & PRD

1. Problem Statement

Why Extract AI Now?

What Breaks If We Don't

What "Good" Looks Like After Extraction

2. Goals & Success Metrics

Primary Goals

Success Metrics

Guardrails

3. Architecture Design

3a. Service Architecture

Deployment Topology

Docker Compose Addition (Local Development)

3b. Prompt Management

Where Prompts Live

Prompt Modules in Rails

How Meet Coach Personas Map to Prompt Templates

3c. Model Routing

Per-Capability Model Assignment

Fallback Chains

A/B Testing Models

4. Migration Plan

New Rails Files

Ai::ServiceClient Design

What Happens to CoachService

GraphQL Mutation Changes

Background Job Changes

Backward Compatibility

5. Cost & Usage Tracking

Per-User Token Tracking

Virtual Keys

Rate Limiting Strategy

6. Operational Controls

Model Swapping Without Deploy

Kill Switch (Three Levels)

Health Checks

Monitoring & Alerting

7. Meet Coach Integration

Preference Flow

Preamble Constructed in Rails, Not LiteLLM

A/B Testing Personas

8. Content Moderation

Separate Model and Budget

ScreenContentJob After Migration

Cost Profile

9. Security

API Key Management

Prompt Injection Defense

PII in Prompts

Audit Logging

10. Local Development

Mock Mode (Replaces MockLlm)

Testing

11. Edge Cases & Open Questions

Edge Cases

Open Questions

12. Implementation Phases

Phase 1: Minimum Viable Proxy (1 sprint)

Phase 2: Production Deployment + Cost Tracking (1 sprint)

Phase 3: Operational Maturity (1 sprint)

Phase 4: Advanced Capabilities (future)

13. Cost Analysis

Current Estimated Costs

Post-Meet Coach Projection (500 active users)

How LiteLLM Reduces Costs

LiteLLM Infrastructure Cost

Critical Files

Related Documentation