Coach Model Ownership — Open Source LLM/SLM Fine-Tuning for Coach
Product: Objectuve (codename: Enkidu) Feature: Own the model powering Coach — fine-tuned open source SLM Status: Penny-proposed; needs human greenlight Author: Penny (Product Lead) Date: May 2026
1. Problem Statement
Coach is the most differentiated feature Objectuve has. Six personas, a 5×5 tone/depth matrix, focus areas, and a UDG-enriched context pipeline — no competitor combines this level of personalization with gamified goal tracking. The north-star doc explicitly names "AI Coach quality" as a data moat: "Models trained on goal completion patterns unique to Objectuve get better with more data. Aggregated, anonymized insights about what coaching approaches drive completion in which contexts — this data doesn't exist anywhere else."
Today, that moat is theoretical. Coach routes every request to a commercial provider (OpenAI, Anthropic, Gemini). Two problems compound as the user base grows:
1. Cost scales linearly with usage. At 5,000 MAU with 10 Coach interactions/week/user, GPT-4o costs ~$5,000–7,000/month for Coach alone. At 25,000 MAU, it's $25,000–35,000/month. For a bootstrapped PBC with a $200K ARR target, this is a structural problem — AI costs will eat 30–50% of gross margin before the model is fully scaled.
2. The data moat never materializes. Every interaction trains the commercial provider's model, not ours. We're generating proprietary coaching signals — which phrasing styles correlate with goal completion, which personas resonate for which goal categories, what tone drives check-in after a streak break — and handing them to OpenAI. When we eventually want a model trained on Objectuve data, we'll have no training set and no baseline to improve from.
This PRD addresses both problems by proposing a phased program to own the model powering Coach: build the data flywheel now, train and serve a fine-tuned open source SLM when scale justifies it, and use the outcome correlation data to continuously improve it over time.
2. Opportunity
Why Now Is the Right Time to Start (But Not to Ship)
The window to build this competitively is open. No consumer goal-tracking app owns its coaching model. The tools to fine-tune and serve a 3–8B parameter SLM efficiently have matured in 2024–2025: QLoRA makes fine-tuning a 7B model possible on a single A100 in 4 hours; Cloud Run with L4 GPUs makes serving a sub-$400/month inference layer feasible; DPO (Direct Preference Optimization) lets us align the model on goal completion outcomes without human raters.
The right action in May 2026 is not to train a model — it's to instrument the data pipeline that makes training possible in 12–24 months. Start the flywheel now, fire the engine when the scale justifies it.
Alignment With North Star and Mission
- Goals completed: A Coach model trained on which advice patterns correlate with actual goal completion will outperform a generic commercial model on our specific task. Better coaching → more goals completed.
- Anti-addictive design: A model we control can be tuned explicitly against engagement-maximizing behaviors. We can embed our mission into the model weights themselves — not just the prompts.
- PBC data obligations: User coaching data stays within our infrastructure. No personal goal data shared with third-party training pipelines.
- Cost structure: Reductions at scale fund the free tier and extend the bootstrap runway.
PBC Check (Preliminary)
- Free tier impact: None. This is an infrastructure cost optimization. Coach remains free.
- Engagement vs. goals: Improvement is measured against goal completion rates, not session time or Coach interaction frequency.
- Dark-pattern check: Clean. Model is aligned to support users' stated goals, never to create dependency on the Coach itself.
3. Technical Landscape
Model Candidates
All candidates are commercially permissive (Apache 2.0 or MIT unless noted).
| Model | Params | License | Strengths for Coach | Weaknesses |
|---|---|---|---|---|
| Phi-3.5 Mini | 3.8B | MIT | Best-in-class quality/size ratio; CPU-runnable; fast inference | Limited context window (128K, but performance degrades past 4K) |
| Llama 3.2 3B | 3B | Llama 3.2 Community | Meta ecosystem support; strong instruction following; very fast | Shorter context, less expressive than 7B variants |
| Llama 3.1 8B | 8B | Llama 3.1 Community | Strong reasoning; very fine-tunable; huge community | Requires GPU for responsive inference |
| Mistral 7B v0.3 | 7B | Apache 2.0 | Excellent style adherence; robust JSON output; strong persona differentiation | Slightly heavier than Phi class |
| Gemma 2 2B | 2B | Gemma (commercial OK) | Smallest viable model; Google-optimized; handles structured outputs well | Quality floor; persona fidelity at risk |
| Qwen 2.5 7B | 7B | Apache 2.0 | Strong multilingual (future i18n); high instruction compliance | Less established fine-tuning community |
Recommended primary candidate: Phi-3.5 Mini for serving (CPU-friendly, fast) with Llama 3.1 8B as the quality benchmark during evaluation. Coach responses are short (2–3 sentences), well-defined in structure, and highly persona-constrained — exactly the type of narrow task where a fine-tuned small model matches a large commercial model.
Fine-Tuning Approach
Phase A — Supervised Fine-Tuning (SFT) with QLoRA
QLoRA trains adapter weights (1–5% of model parameters) rather than the full model. A 7B model fine-tune on a single A100 GPU takes 2–6 hours. A 3.8B model takes 1–3 hours. Training cost: $1–15/run on spot instances (Modal, RunPod, Lambda Labs at $0.50–2.50/hr).
The adapter captures persona voice, tone spectrum compliance, structural output (2-sentence advice, JSON insight cards, milestone arrays), and Objectuve-specific context (streak logic, XP, badge language).
Phase B — Direct Preference Optimization (DPO)
After sufficient real-world data accumulates, we align the model on outcomes rather than just stylistic quality. DPO trains the model to prefer responses that were followed by:
- Goal check-in within 48 hours of advice
- Streak continuation after a break
- Milestone completion after auto-generation
This is the competitive moat. No commercial provider trains on whether their advice led to a goal being completed. We will.
Phase C — Continual Learning
Quarterly LoRA adapter retraining on fresh interaction + outcome data. The model improves as the platform grows. This is the flywheel.
Infrastructure
Training (cloud burst):
- Modal or RunPod with A100 80GB (SXM4) — $1.50–2.50/hr
- Training run cost: $3–15 per training cycle
- Cadence: monthly initially, then quarterly once stable
Serving (persistent):
- GCP Cloud Run with L4 GPU (NVIDIA L4, 24GB VRAM) — available in Cloud Run Jobs/Services as of 2025
- Phi-3.5 Mini serves at ~40–80 tokens/sec on L4 → ~150–300ms per Coach response
- Cost: ~$0.55/GPU-hr. At 12 hrs/day average load: ~$200/month
- Alternative at lower scale: Fly.io GPU Machines (~$0.50/hr, easier cold-start management)
- Serving stack: Ollama (already in the CoachService provider cascade — zero new Rails integration required) or vLLM for higher throughput
Integration path: Zero Rails code changes. The existing Ollama provider branch in CoachService (triggered by OLLAMA_BASE_URL) already routes to a local or remote Ollama server. Pointing OLLAMA_BASE_URL to our serving endpoint switches the provider with one env var. The LiteLLM proxy layer (planned in dedicated-ai-service-prd.md) adds a clean routing layer for A/B testing between commercial and owned models.
4. Data Requirements and Flywheel
What We Need to Train
The fine-tuning dataset has three components, one per Coach task:
| Task | Input | Output | Volume target |
|---|---|---|---|
| get_advice | UDG context string + user message + persona/tone/depth config | 2-sentence advice response in persona voice | 1,000–5,000 per persona = 6,000–30,000 total |
| generate_insight | Page context JSON + persona config | {type, title, message, cta_label} structured insight | 500–2,000 per page type = 3,000–12,000 total |
| generate_milestones | Goal name + description | JSON array of 3–5 milestone strings | 2,000–10,000 total |
Phase 1 (synthetic, pre-launch): Use GPT-4o to generate the initial training corpus from structured prompts. Cost: ~$50–150 in API calls for the full dataset. Quality control: human review sample (10%) + automated structural validation (JSON parsing, sentence count, persona keyword checks).
Phase 2 (real interactions, post-launch): Instrument the advice and insight endpoints to log:
- Input context (UDG string, page context)
- Generated response
- User's implicit feedback signal (did they follow up? did they complete a check-in within 24h?)
- Explicit feedback (thumbs up/down on insight cards — already planned in Phase 5)
Consent: covered under privacy policy with anonymous aggregation opt-out. High-quality signal is only the get_advice responses where the user continued the conversation (implicit quality signal) or where a check-in was logged within 48h.
Phase 3 (outcome-labeled, 12–18 months post-launch): Join interaction logs with goal completion events. This is the DPO training signal: advice response → did the user complete more goals in the following 30 days? Requires ~500+ outcome-labeled pairs per persona to be meaningful. At 1,000 MAU with 10 Coach interactions/week, this accumulates in 3–4 months.
Data Privacy Architecture
- All training data is anonymized:
user_id→ hash, goal names → category only, no PII in training corpus - Users can opt out of anonymized model training (preference in settings, stored in
UserDetail.data) - Training data never leaves GCP infrastructure
- No sharing with any commercial provider's training pipeline
- Covered under PBC data commitments
5. Cost Analysis
Current State (commercial API)
Blended input/output cost per Coach request at typical UDG context depth (~1,500 input tokens, ~200 output tokens):
| Provider | Cost/1M tokens (in/out) | Cost per request | 5K MAU cost/month | 25K MAU cost/month |
|---|---|---|---|---|
| GPT-4o | $2.50 / $10.00 | ~$5.75 per 1K requests | ~$5,750 | ~$28,750 |
| GPT-4o Mini | $0.15 / $0.60 | ~$0.345 per 1K requests | ~$345 | ~$1,725 |
| Claude Haiku 3.5 | $0.80 / $4.00 | ~$1.88 per 1K requests | ~$1,880 | ~$9,400 |
| Claude Sonnet 4.6 | $3.00 / $15.00 | ~$7.50 per 1K requests | ~$7,500 | ~$37,500 |
Assumes 10 Coach interactions/week/MAU.
Owned Model (served)
| Scale | GPU infra/month | Training/quarter | Total AI cost/month | vs. GPT-4o Mini | vs. Claude Haiku |
|---|---|---|---|---|---|
| 500 MAU | $200 | $15 | $205 | $170 more | $115 more |
| 5K MAU | $200–400 | $15 | $215–415 | $70 less to $130 savings | $1,465–1,665 savings |
| 25K MAU | $400–800 | $30 | $430–830 | $895–1,295 savings | $8,570–8,970 savings |
| 100K MAU | $1,200–2,400 | $50 | $1,250–2,450 | $4,450–5,650 savings | $35,550–36,750 savings |
Break-even with GPT-4o Mini: ~3,000–4,000 MAU
Break-even with Claude Haiku: ~700–1,500 MAU (Haiku is more expensive than Mini)
The smart strategy: stay on GPT-4o Mini or Claude Haiku until 2,000–3,000 MAU, then migrate to owned model. The data flywheel starts from day one regardless.
6. Quality Expectations
SLM fine-tuning for narrow, well-defined tasks consistently achieves 85–95% quality parity with frontier models when:
- The output structure is fixed (2-sentence advice, JSON card, string array)
- The style space is well-defined (6 personas × 5-tone spectrum = learnable from examples)
- The context is structured (UDG is machine-generated, consistent format)
- Training data volume is adequate (1,000+ examples per task type)
Studies on task-specific fine-tuning (Meta's Llama 2 paper, Microsoft's Phi-3 technical report, Mistral's evaluation suites) consistently show that a 7B model fine-tuned on domain-specific data outperforms GPT-4 on that specific task while being 10–50x cheaper to serve.
Coach is a narrow task. The output is short. The context is structured. This is the textbook case for fine-tuning.
Quality evaluation harness (required before production switch):
- Automated: structural validation (sentence count, JSON validity, CTA presence), persona keyword compliance (Captain ≠ "I feel like maybe...", Rival ≠ "wonderful progress!")
- Human: blind comparison panel (Josh + 2-3 users) rating response pairs on helpfulness, persona accuracy, and motivational quality. Target: owned model preferred in ≥75% of comparisons.
- Outcome: 30-day A/B test (50% traffic split) measuring check-in rate within 48h of Coach interaction. Target: owned model ≥ commercial model ± 5%.
7. Risks
| Risk | Likelihood | Impact | Mitigation |
|---|---|---|---|
| Quality degradation on rare persona/tone combinations | Medium | Medium | Synthetic data specifically targets edge combinations (Rival + Detailed + 5/5 Demanding) |
| GPU infra operational burden on small team | High | Medium | Use managed GPU platforms (Modal, Fly.io) to avoid self-managed hardware; start with 1 env var swap |
| Insufficient training data pre-launch | High (pre-launch) | Low (because we defer training) | Use synthetic data to bootstrap; real data starts accumulating day one |
| Model drift as product evolves | Medium | Medium | Quarterly retraining cadence; eval harness catches regressions |
| Persona fidelity at smaller model sizes (2B) | Medium | High | Don't use sub-3.8B models; Phi-3.5 Mini is the floor |
| User perception of quality drop | Low (if eval passes) | High | Gate behind A/B test; never ship without human eval sign-off |
| LiteLLM proxy dependency (routing layer) | Low | Low | Rollback path: env var switch back to commercial provider |
8. What This Is Not
- Not a general-purpose LLM. The model only powers Coach. It doesn't moderate content, generate product copy, or handle any other AI task.
- Not a replacement for frontier models everywhere. Milestone generation and insight cards may migrate later; Coach advice is the beachhead.
- Not a pre-launch priority. Zero users means zero real training data. The correct action in May 2026 is instrumentation, not training.
- Not a moonshot. SLM fine-tuning for narrow tasks is a solved engineering problem in 2026. The risk is execution and timing, not feasibility.
9. Viability Score
7.5 / 10
| Dimension | Score | Rationale |
|---|---|---|
| Technical feasibility | 9/10 | Ollama already in CoachService; LiteLLM proxy en route; fine-tuning tooling mature |
| Strategic alignment | 10/10 | Directly named as data moat in north-star doc; PBC-aligned; mission-embedded |
| Economic viability | 7/10 | Strong positive at 3K+ MAU; negative ROI pre-1K MAU; timing-sensitive |
| Data availability | 5/10 | Pre-launch means no real data yet; synthetic bootstrap required |
| Team execution risk | 6/10 | Small team + GPU infra = operational load; use managed platforms to mitigate |
| Quality confidence | 8/10 | Narrow task, structured output — well within SLM fine-tuning sweet spot |
The score would be 9/10 at 2,000+ MAU with 6 months of interaction logs. The limiting factor today is data, not technology or economics.
10. Phase Recommendation
This program has four phases. Each phase is a prerequisite for the next, and each has a clear trigger to advance.
Phase 0 — Instrument (Start Now, No Dependencies)
Timeline: May 2026 — Launch
Cost: ~0 (logging infrastructure only)
Work: ~1–2 days of backend engineering
- Add a
CoachInteractionLogmodel (or use PostHog events) to record:user_id_hash,persona,tone,depth,focus_areas,page,request_type(advice/insight/milestone),input_token_count,output_token_count,provider,latency_ms - Add implicit feedback signal: log a
coaching_followthroughevent when a user checks in within 48h of a Coach advice interaction - Add explicit feedback: thumbs up/down on insight cards (already in Phase 5 roadmap — align timing)
- Add opt-out preference to user Coach settings
Gate to Phase 1: Launch + 30 days of data.
Phase 1 — Synthetic Dataset (Post-Launch, Pre-Scale)
Timeline: Launch + 30–90 days (Jul–Oct 2026)
Cost: ~$150–300 (API calls for data generation + 20 hours engineering)
Trigger: App is live; basic interaction logging is running
- Write a data generation script: for each of 6 personas × 5 tone levels × 3 depth levels × 5 focus areas × 20 representative goal contexts × 3 task types, generate a training example via GPT-4o
- Total target: ~18,000–50,000 training examples across all task types
- Run structural validation on every generated example (sentence count, JSON validity, persona keyword audit)
- Human review sample: Josh reviews 200 examples (3–4 hours) to calibrate quality threshold
- Store dataset as JSONL in GCS, versioned
Gate to Phase 2: Dataset > 10,000 validated examples; 500+ real interaction logs; eval harness running.
Phase 2 — Fine-Tune and Evaluate (After Traction Signal)
Timeline: 6–18 months post-launch (~Nov 2026 – Apr 2027)
Cost: ~$500–2,000 one-time (training runs + evaluation time)
Trigger: 500+ MAU with consistent Coach usage; 5,000+ real interaction logs; eval harness built
- Fine-tune Phi-3.5 Mini with QLoRA on the combined synthetic + real dataset using Modal or RunPod
- Fine-tune Llama 3.1 8B as quality benchmark (larger model = quality ceiling check)
- Run automated eval: structural, persona compliance, JSON validity
- Run human eval: blind comparison panel (target: ≥75% preference for fine-tuned model)
- Deploy Phi-3.5 Mini to Cloud Run with L4 GPU or Fly.io GPU machine with Ollama serving
- A/B test: 10% traffic to owned model, 90% to commercial. Measure check-in followthrough rate.
- If A/B passes (owned ≥ commercial ± 5%): expand to 50%, then 100%
Gate to Phase 3: A/B test passed; owned model in production for 60 days with stable quality metrics; 2,000+ MAU.
Phase 3 — Outcome Alignment and Flywheel (Scale)
Timeline: 18–36 months post-launch (~Nov 2027 – Apr 2028)
Cost: ~$200–800/month infrastructure + ~$50/quarter training
Trigger: 2,000+ MAU; 6+ months of outcome-labeled interaction data; DPO training feasible
- Build DPO training pipeline: join
coaching_interactionswithgoal_completion_eventsto label preference pairs (advice that led to completion = preferred; advice that didn't = rejected) - Run DPO fine-tuning on top of the SFT adapter — aligns the model on outcome, not just style
- Establish quarterly retraining cadence: new LoRA adapter every 90 days
- Expand Coach to cover more surface area: welcome back messages, streak break messages, goal auto-adjustment suggestions (all in Phase 5 roadmap) — owned model powers all of it
- Retire the commercial API fallback for Coach (keep for content moderation, which has different requirements)
This is where the moat materializes. A model that improves as users complete goals, tuned on thousands of real coaching interactions from Objectuve's specific population, is not reproducible by a competitor without the same user base and the same years of outcome data.
11. Related Documents
docs/features/ai-coach.md— Current Coach runtime behaviordocs/features/meet-coach.md— Persona + tone + focus configurationdocs/product/coach-udg-prd.md— User Data Graph (context enrichment, in-progress)docs/product/dedicated-ai-service-prd.md— LiteLLM proxy extraction (Phase 2 prerequisite)docs/product/phase-5-intelligence-prd.md— Intelligence features (Phase 5, includes explicit feedback on insight cards)docs/product/north-star.md— AI Coach quality named as data moat (§ Competitive Moat)
12. Suggested Next Step
Phase 0 instrumentation is the correct immediate action. It costs ~0, requires 1–2 days of backend work, and starts the data flywheel that makes every subsequent phase possible. Without it, we arrive at 2,000 MAU with nothing to train on.
Do not train a model before launch. Do not allocate GPU budget before 500 MAU. Do instrument the interactions from day one.
Route to Orion to scope Phase 0 instrumentation as a discrete task attached to the Phase 5 intelligence milestone (natural home — PostHog already instrumented, Coach interaction logging fits there). Keep Phases 1–3 in backlog for triage after launch metrics are visible.
Penny — Product Lead, Objectuve
Confidence: high on strategic rationale; medium on economic timing (depends on actual MAU growth)
Last updated: 2026-05-05