Objectuve Docs

Appearance

v4.55 — Docs AI Assistant ​

A public visitor on help.objectuve.com and an @objectuve.com teammate on docs.objectuve.com can each ask a question in plain language and get a synthesized answer with working citations, instead of only keyword search results.

Summary ​

The OBJ-3612 spike proved the retrieval architecture worked — full-context stuffing for the small guide corpus, BM25 lexical retrieval for the large internal docs corpus — but deliberately shipped nothing production-ready: no auth, no rate limiting, no dedicated model alias, no enforcing cost ceiling, no CSP. This milestone built all five, plus two things the spike never touched: getting either corpus into the Rails runtime at all (the image build context is rails_api/, so neither guide_site/ nor docs/ existed there before this milestone), and tuning citation accuracy at a chosen retrieval depth.

Seven phases shipped guide-first. Phase 1 laid the shared platform plumbing, including the product's first enforcing AI budget ceiling — Ai::BudgetGuard — where a prior mechanism only alerted Slack. Phase 2 designed one widget for both VitePress themes, settling the milestone's central question by finding that the guide site already had a tuned, empty "Ask '<query>' in your own words" row waiting to be filled, rather than building a second search entry point. Phases 3–4 shipped the public surface: an unauthenticated askGuideQuestion operation defended by three independent layers (a per-IP throttle, the new budget guard, and a kill switch), plus a fourth Roy caught in review — an unbounded question length — closed before it could exhaust the production ceiling in minutes. Phases 5–6 shipped the internal surface: a Postgres-backed corpus for docs/'s 8.7 MB / 10,667 chunks, chosen on measured memory numbers rather than argument, and a Clerk-gated, @objectuve.com-restricted askDocsQuestion operation. Phase 7 reconciled the documentation — correcting the spike ADR's standing "neither has shipped" claim, writing the runtime-controls ops page, and closing a known retrieval gotcha as fixed.

Two things are true at close that the phase list alone wouldn't show, and both are stated in the milestone's own record rather than rounded up. First, the milestone's one soft metric — citation accuracy at the chosen retrieval depth — has only ever been measured against fixtures (76.9% @ k=5, from the spike). No agent in the delivery chain holds staging LiteLLM credentials to run the live re-measurement, so it's tracked as a fast-follow (OBJ-3654) rather than blocking the ship. Second, neither assistant is reachable by a real user yet: both Firebase targets deploy from master and both call the production API, but the Rails resolvers behind them only ship on the next version tag — confirmed by querying live production directly, where both operations still 404. The engineering is complete; the release is a scheduling step, not a build gap.

Goal ​

Ship a Mintlify-style "ask a question, get a synthesized answer with source citations" assistant on both Objectuve doc surfaces — help.objectuve.com (public guide) and docs.objectuve.com (Clerk-gated internal docs) — on the retrieval architecture the OBJ-3612 spike already validated, with the five production requirements that spike explicitly deferred actually built: auth, rate limiting, a dedicated model alias, an enforcing cost ceiling, and CSP.

Scope — What Shipped ​

  • Ai::BudgetGuard — the product's first enforcing per-feature AI spend ceiling, checked inside Ai::ServiceClient.chat alongside the existing category and rate-limit gates. Generic-with-opt-in: any feature with a configured ceiling gets enforcement; every existing feature is untouched.
  • A dedicated docs/assistant LiteLLM model alias, registered in Ai::ModelCatalog::GEMINI_ALIASES, AiUsageEvent::FEATURES, and CoachInteraction::AUDIT_EXEMPT_FEATURES — a new docs_assistant AiUsageEvent/rate-limit category distinct from coaching.
  • Settings.ai.docs_assistant_enabled — a config-gem static kill switch, ships false, flippable per environment via a commit + deploy (not a live override).
  • Ai::DocsAssistant::AnswerGuideQuestion — full-context-stuffing Interaction over a committed guide-corpus snapshot, behind the unauthenticated askGuideQuestion GraphQL operation. Defended by a rack-attack per-IP throttle, the budget guard, the kill switch, and a 1000-char question-length cap.
  • A Postgres-backed docs corpus — DocsCorpusChunk/DocsCorpusPosting, populated by a rake indexing task from docs/**/*.md — and Ai::DocsAssistant::AnswerDocsQuestion, a BM25-retrieval Interaction at k=5, behind the Clerk-gated, @objectuve.com-restricted askDocsQuestion operation.
  • GuideAskAssistant.vue and DocsAskAssistant.vue — the widget, in each site's VitePress theme, filling an ask-row inside the existing local-search modal rather than adding a new launcher. Nine states specified and shipped: idle, armed, thinking, answer, refusal, rate-limited, couldn't-reach, disabled-in-flight, disabled-at-load.
  • First-ever Content-Security-Policy headers (report-only) on both the enkidu-guide and enkidu-docs Firebase targets.
  • A CI-enforced retrieval regression check tracking recall and citation accuracy as separate metrics — recall@5 verified live at 100% (up from the spike's 90.9%); citation accuracy remains fixture-backed pending OBJ-3654.
  • Documentation: the spike ADR corrected, a new docs/operations/docs-assistant-ops.md, a new docs/features/docs-assistant.md, the BM25 gotcha closed as fixed, and a CHANGELOG entry worded to not imply production availability.

Phases ​

PhaseNameStatusPlansHighlights
1AI platform plumbingShipped1docs/assistant alias, docs_assistant category, the product's first enforcing Ai::BudgetGuard (Codi)
2UI-SPEC — assistant widget for both surfacesShipped1One design pass, nine states, fills an existing tuned search-modal row instead of a new entry point (Desi)
3Guide assistant backendShipped1Unauthenticated askGuideQuestion, three abuse-defense layers plus a fourth (question length) caught in review (Codi)
4Guide widget + CSP + shipShipped1help.objectuve.com live; A5 (live alias resolution) closed with real evidence (Codi)
5Docs corpus + BM25 retrieval + citation tuningShipped1Postgres-backed corpus, recall 90.9%→100% @ k=5; citation accuracy explicitly left unconfirmed pending a live model (Codi)
6Docs endpoint (Clerk-gated) + widget + CSPShipped1askDocsQuestion, @objectuve.com-restricted server-side; live citation re-measurement deferred to OBJ-3654; production-tag gap disclosed (Codi)
7DocumentationShipped1ADR corrected, ops page + feature doc, gotcha closed, CHANGELOG, zero copy drift found (Dori)

Key Decisions ​

  • Guide-first, both surfaces in one milestone — help.objectuve.com has the harder abuse story (first unauthenticated money-spending endpoint in the product); docs.objectuve.com has the harder corpus story (8.7 MB against a 512Mi Cloud Run ceiling). Sequencing this way gave each phase one hard problem instead of two.
  • Postgres-backed corpus chosen on measured numbers, not argument (Phase 5) — ~1.5 MB/query for the shipped design vs. +80–320 MB for the rejected in-memory alternative, against a service with documented OOM history. Keeps Rails memory flat regardless of docs/'s future growth.
  • The assistant fills an existing UI slot rather than adding a new one (Phase 2) — GuideSearchEnhancements.vue already reserved a bottom-pinned row for exactly this purpose; a separate launcher or an answer-row above keyword results were both argued down as competing entry points on a site whose search UX was already deliberately tuned.
  • Three independent abuse-defense layers on the public endpoint, all required — a per-IP throttle alone is trivially defeated by a distributed caller; the enforcing budget ceiling and the kill switch are what actually bound worst-case spend.
  • Recall and citation accuracy tracked as separate CI metrics, deliberately — conflating them would hide which failure mode a regression represents. Recall passed and improved; citation accuracy is the one metric this milestone could not fully close (see Outcomes).
  • Neither a live citation re-measurement nor a production tag blocked this ship — both were named, tracked, and explicitly disclosed rather than silently deferred. Josh made the call on the former; the latter is a release-timing decision, not an engineering gap.

Requirements Coverage ​

34/37 phase-checklist items satisfied — 2 partial (recorded, accepted substitutions), 1 deliberately deferred (OBJ-3654). This milestone has no standalone REQUIREMENTS.md; see the Milestone Audit for the full item-by-item verification.

CategoryCountStatus
PLAT-* (Phase 1 — platform plumbing)5All satisfied
UI-* (Phase 2 — UI-SPEC)5All satisfied
GUIDE-* (Phases 3–4 — public surface)1312 satisfied, 1 substituted (drift-guard demonstration, accepted by Roy)
DOCSRV-* (Phases 5–6 — internal surface)119 satisfied, 1 partial (citation-accuracy confirmation), 1 deferred (live re-measurement, OBJ-3654)
DOC-* (Phase 7 — documentation)3All satisfied

Full phase-checklist detail: v4.55-docs-ai-assistant-ROADMAP.md on GitHub.

Outcomes ​

On master today: a public visitor on help.objectuve.com can ask a question in the guide's search modal and get a cited answer, defended by three independent abuse layers; an @objectuve.com teammate on docs.objectuve.com can do the same over the internal engineering corpus, with the server verifying the Clerk session and email domain before any LLM call runs — verified against the exact failure case that matters most, a valid session on a non-Objectuve email. Both fail closed gracefully to an existing "couldn't reach" state on any error. Spend is bounded by an enforcing ceiling attributable separately from coaching traffic, and ops can disable either surface with a settings flip (requiring a commit + deploy, not a live override — a correction Phase 7 made to the milestone's own original framing).

Not yet true: a real user cannot reach either assistant yet — both Firebase targets deploy from master, but the Rails resolvers behind them only reach api.objectuve.com on the next version tag. Both operations 404 in live production as of milestone close. And the milestone's one soft metric, citation accuracy at k=5, is measured only against fixtures (76.9%) — the live number is a tracked fast-follow (OBJ-3654), not yet run.

Tech Debt ​

  • (Phase 6/close) Citation accuracy remains fixture-backed pending a live re-measurement — OBJ-3654, blocked on staging LiteLLM credentials reaching a crew agent's sandbox.
  • (close) Neither assistant is live in production — blocked on the next version tag, a release-timing decision rather than a build gap.
  • (Phase 2) Two pre-existing, out-of-scope findings surfaced incidentally and filed separately: docs_site's dark-mode brand-color contrast (OBJ-3626) and its VitePress motion-vocabulary gap (OBJ-3627).
  • (close) A crew-infra issue, not specific to this milestone's content: OBJ-3657, the shared bare repo's local git config attributing some commits to Codi <josh@objectuve.com> instead of the canonical identity.
  • 058d890a459 — Phase 1: AI platform plumbing, Ai::BudgetGuard (PR #3071)
  • 05d490ffd85 — Phase 2: UI-SPEC for both surfaces (PR #3072)
  • 3f579edc7fc — Phase 3: guide assistant backend + abuse controls (PR #3073)
  • 1b6192c906c — Phase 4: guide widget + CSP, help.objectuve.com live (PR #3091)
  • 5633cf5a80c — Phase 5: docs corpus + BM25 retrieval + citation tuning (PR #3081)
  • f58e1ba4332 — Phase 6: docs endpoint (Clerk-gated) + widget + CSP (PR #3097)
  • 129fbf1df73 — Phase 7: documentation (PR #3101)

Last updated: 2026-09-12

Loading…