ADR: Embedding an AI knowledge-retrieval assistant into the docs sites

Status: Go on the docs.objectuve.com lexical path, go on the help.objectuve.com full-context path — both since shipped. This page was originally a spike verdict, not a production decision; the v4.55 Docs AI Assistant milestone (OBJ-3616) built the productionization epic this ADR deferred. See Production status (as shipped, v4.55) below for what shipped against each requirement this ADR listed. The spike verdict, measured numbers, and findings below are historical record and unchanged — they were, and remain, the evidence baseline the milestone was built and judged against. Date: 2026-09-10 (spike decision). Production status added 2026-09-11 (v4.55 Phase 7, OBJ-3624). Related: scripts/README.md — docs-agent-spike.mjs, BM25 count-question distractor gotcha, Docs & guide assistant operations, Docs & guide assistants (feature doc)

Context

Both VitePress doc surfaces use search: { provider: 'local' } — a MiniSearch keyword index (docs_site/.vitepress/config.ts:81-83, guide_site/.vitepress/config.ts:69-71). Keyword search finds pages; it doesn't synthesize an answer with citations. Mintlify's assistant ("ask a question, get a synthesized answer with source citations") is the reference behavior this spike evaluated.

OBJ-3612's task package (Orion) scoped this as a throwaway, timeboxed spike — a CLI eval harness, not a shipped feature — because the two doc surfaces have radically different corpus sizes, and that difference is the finding:

Surface	Site	Access	Corpus
help.objectuve.com	guide_site/	public	22 real pages (404.md/AUTHORING.md excluded as non-user-facing), ~16k tokens
docs.objectuve.com	docs_site/ (srcDir: '../docs')	Clerk-gated to @objectuve.com (docs_site/.vitepress/theme/AuthGate.vue:8)	463 files, 10,667 heading chunks, ~2M tokens

The guide corpus fits entirely in one prompt — zero retrieval infrastructure needed. The docs corpus does not, and the hard constraint on this spike was: try the cheapest retriever (lexical/BM25) first, and if it fails the quality bar, that failure is itself the finding — no embeddings, no pgvector, no new deployed service, to be rescued inside this spike. Confirmed against infra/litellm/config.production.yaml and infra/litellm/config.local.yaml: no embedding model is configured in either.

What was built

scripts/docs-agent-spike.mjs (Codi, Tasks 1–2 of OBJ-3612) — a throwaway, zero-npm-dependency Node harness with two paths:

• --corpus guide: stuffs the entire guide_site/*.md corpus into the system prompt with per-page source markers.
• --corpus docs --k <N>: BM25 lexical retrieval over docs/**/*.md, heading-chunked, top-k chunks into the prompt.

Both paths call LiteLLM's /v1/chat/completions with model coaching/default (resolves to gemini/gemini-3.8-flash), and report per-question source-hit, latency, and cost, plus (docs-only) recall@k tracked separately from citation accuracy — a retrieval miss and a citation miss are different failures with different fixes. Full script/eval-set/test detail: scripts/README.md.

Measured results

Codi's own harness run used a local stub server standing in for LiteLLM (no GOOGLE_API_KEY in that sandbox) — those LLM-dependent numbers were not real. The tables below are from a live run against gemini-3.8-flash (Josh, via a local path-rewrite/alias stand-in for LiteLLM's OpenAI-compatible route — not a repo change, not deployed) and supersede the stub numbers. The BM25 retrieval numbers were real and unchanged in both runs.

--corpus guide (28 questions, 4 out-of-scope refusals)

Metric	Value
Source-hit rate	96.4% (27/28)
Latency p50 / p95	1782ms / 3645ms
Total run cost	34.7052¢ (~1.24¢/query)

Single miss: a Supporter-tier color-theme question. All 4 out-of-scope refusal probes passed. One run only — treat as a point estimate, not a settled number.

--corpus docs --k 5 (26 questions, 4 out-of-scope refusals, 22 with expected sources)

Metric	Value
Source-hit rate	76.9% (20/26), repeat run 69.2% (18/26)
Recall@5	90.9% (20/22), identical on repeat
Latency p50 / p95	1738ms / 4110ms
Total run cost	4.0755¢ (~0.16¢/query)

--corpus docs --k 15 (same eval set)

Metric	Value
Source-hit rate	65.4% (17/26), identical on repeat
Recall@15	100% (22/22), identical on repeat
Latency p50 / p95	1892ms / 6535ms
Total run cost	11.6581¢ (~0.45¢/query)

Two denominators are deliberate, not an inconsistency: 4 of the 26 docs questions (and the guide set's own out-of-scope subset) are refusal probes with no expectedSources — they count toward source-hit (a correct refusal is a pass) but are excluded from recall.

Findings

Lexical retrieval passes the quality bar. Orion's stop condition — "if lexical retrieval fails, that failure is the finding" — did not trigger. Recall@5 = 90.9%, recall@15 = 100%, both reproduced identically on repeat runs. Recall is deterministic; there is no case for embeddings or pgvector from this data.

Citation accuracy is the open variable, not retrieval. Source-hit rate is stochastic (±8 points across repeat runs at k=5) and drops as k grows (76.9% → 65.4%, k=5 → k=15) — the opposite direction from recall. More retrieved context gives the model more plausible-looking wrong sources to cite; this is a distractor effect, not a retrieval defect. A production build needs prompt/citation discipline tuning at a chosen k — that tuning is unscoped work, not scoped here.

Root cause on the two recall@5 misses: both are GraphQL field-count questions ("how many root mutation fields", "how many root query fields"). docs/architecture/graphql/index.md chunks into a short, answer-dense ## Contents chunk plus two much longer category-enumeration chunks; BM25's length normalization ranks the short chunk outside top-5 but inside top-15 on generic, repeated query vocabulary. This is a BM25 weakness on count/table-lookup questions, not a chunking defect — the chunker isolated the answer chunk correctly. Recorded as a gotcha below (flagged as inference: the harness doesn't log which competing chunks displaced the answer chunk).

Cost does not decide between the two architectures. ~1.24¢/query (guide, fixed ceiling — the whole ~16k-token corpus is stuffed every call) vs. ~0.16¢ (docs k=5) / ~0.45¢ (docs k=15). The docs path's cost and latency both grow with docs/'s size, unlike the guide path's fixed ceiling.

Decision

Go, per surface, on the retrieval architecture only — not a build order or a commitment to build:

• help.objectuve.com: full-context stuffing. No retrieval infrastructure needed; the corpus fits in one prompt.
• docs.objectuve.com: lexical (BM25) retrieval, no embeddings, no pgvector, no new deployed service. k and citation-prompt discipline are open tuning questions for whoever picks up the productionization epic, not settled here.

Neither surface ships from this issue. Productionizing either — a live HTTP endpoint, a UI widget, auth — is out of scope for the spike (per Orion's task package) and, per the Epic decision in that package, is a separate epic decision to be made on this evidence.

Production status (as shipped, v4.55)

The five requirements this ADR originally deferred to "the productionization epic" are now built. Full operational detail (kill switch, budget-trip behavior, rate limits, corpus refresh) lives in Docs & guide assistant operations; this section confirms each requirement shipped and how, so this ADR no longer asserts anything false about shipped-state.

• Auth story for the Clerk-gated internal corpus — shipped. Types::QueryType#ask_docs_question (rails_api/app/graphql/types/query_type.rb:839-878) verifies a real Clerk session server-side (require_auth!) and that current_user.email ends with @objectuve.com (DOCS_ASSISTANT_ALLOWED_EMAIL_DOMAIN, query_type.rb:9) — both checks run before the kill switch or any LLM call. docs_site/.vitepress/theme/AuthGate.vue's client-side ALLOWED_DOMAIN check is a UX affordance only, not this field's security boundary. #ask_guide_question (query_type.rb:799-830) stays unauthenticated by design — the guide corpus has no equivalent sensitivity.
• Rate limiting via Ai::RateLimiter — shipped, plus a second layer for the guide. docs_assistant has its own entry in both CATEGORY_FOR_FEATURE and RATE_LIMITS (20 req/min, rails_api/app/services/ai/service_client.rb:34,54) — a distinct category from coaching/batch, so disabling the assistant never touches Coach or background jobs. #ask_docs_question also enforces its own 20/min via Ai::RateLimiter.check_and_increment (real user_public_id); #ask_guide_question, being anonymous, uses an ad hoc Rails.cache counter keyed anon:<ip> at the same 20/min, plus a coarser 30/min-per-IP rack-attack throttle in front of it (rails_api/config/initializers/rack_attack.rb) — three independent layers, not one, because the guide endpoint spends money per unauthenticated request. Full numbers: Docs & guide assistant operations § Rate limits.
• A dedicated LiteLLM model alias — shipped. docs/assistant is a new entry in infra/litellm/config.production.yaml / config.local.yaml, added to Ai::ModelCatalog::GEMINI_ALIASES, and resolves through the same runtime-switchable Ai::CoachModel.current as the coaching aliases — cost and usage attribute to docs_assistant, never coaching.
• A cost ceiling — shipped, and enforcing, not just an alert. Ai::BudgetGuard (rails_api/app/services/ai/budget_guard.rb) is checked inside Ai::ServiceClient.chat before every LiteLLM call and returns the same nil contract as the kill switch and rate limiter when month-to-date docs_assistant spend reaches its configured ceiling (inclusive >=). Shipped generic-with-opt-in, not docs_assistant-specific — any feature with a Settings.ai.<feature>_monthly_budget_cents key gets the same enforcement. Ceilings: Docs & guide assistant operations § Budget ceiling.
• firebase.json CSP connect-src change — shipped, report-only. Both enkidu-guide (Phase 4) and enkidu-docs (Phase 6) now carry a Content-Security-Policy-Report-Only header with connect-src covering https://api.objectuve.com — the first CSP either target has ever had. Report-only, not enforcing, was the deliberate choice for both (neither site has a staging twin to ramp through first); see the ops page for the full reasoning and how to check for violations before ever flipping to enforcing.

Corpus-delivery annotation (v4.55 Phase 5) — the in-memory-snapshot recommendation below no longer holds for the docs/ surface. This ADR's spike never evaluated how a corpus reaches the Rails runtime — docs/ isn't in the rails_api/ build context (see .github/workflows/staging.yml's working-directory: rails_api), so productionizing the lexical-retrieval decision above still had an open delivery question. Phase 5 measured it directly rather than arguing it: a Postgres-backed corpus (DocsCorpusChunk/DocsCorpusPosting, populated by rails_api/lib/tasks/docs_corpus.rake) costs ~1.5 MB/query in live RSS, against +80–320 MB for an in-memory snapshot — a real risk on a 512Mi Cloud Run service with a 195 MB cold-boot baseline and documented OOM history (OBJ-1481). The guide corpus (79 KB, committed and loaded whole per Phase 3) is unaffected by this — it's small enough that the snapshot-vs-Postgres question never arises for it. No embeddings, no pgvector, no new deployed service either way, so this ADR's original guardrail (below) still holds.

Consequences

• No vector DB, no embedding model, and no new deployed service exist anywhere in the spike diff — the spike stayed inside its stated guardrails, and v4.55's production build did too (see the Postgres-backed corpus annotation above — still no embeddings, no pgvector, no new service).
• The "go" on productionizing either surface started from measured numbers, not a hunch: lexical retrieval was validated on the real 463-file corpus, and the open work was prompt/citation tuning plus the five production requirements above — not a retrieval-architecture redesign. v4.55 shipped exactly that scope.
• Rollback is already the default: scripts/docs-agent-spike* and this page remain throwaway/reference artifacts — nothing runtime depends on them. The shipped assistant's own rollback is a settings flip (docs_assistant_enabled: false), not a revert — see the ops page.

References

• OBJ-3612 — this spike (harness, eval sets, live-run numbers, this decision record).
• OBJ-3616 — v4.55 Docs AI Assistant, the productionization epic this ADR deferred. .planning/milestones/v4.55-docs-ai-assistant-ROADMAP.md has the full phase-by-phase build record.

---

Last updated: 2026-09-11 (OBJ-3624, v4.55 Phase 7: superseded the "Production requirements" section with "Production status (as shipped)" now that all five requirements have shipped; corrected the standing "Neither has shipped" claim; added the Phase 5 corpus-delivery annotation. Spike numbers and findings above are unchanged. Prior update 2026-09-10, OBJ-3612: initial decision record)