Docs & Guide Assistant Operations

Runtime controls for the two "Ask a question, get a cited answer" assistants shipped in v4.55 — Ask the guide on help.objectuve.com and Ask the docs on docs.objectuve.com. Both route through the docs_assistant feature key end to end; this page covers the kill switch, the enforcing budget ceiling, the rate limits on each surface, and how to refresh the docs corpus.

For what the assistants are and how they behave from a user's perspective, see Docs & guide assistants. For the retrieval-architecture decision and measured numbers behind this build, see the spike ADR.

Every claim below is verified against the named source file.

Kill switch

Settings.ai.docs_assistant_enabled (rails_api/config/settings.yml:15) gates both GraphQL operations — Types::QueryType#ask_guide_question and #ask_docs_question (rails_api/app/graphql/types/query_type.rb:803,848) both check it, via the shared Ai::SettingsGuard.enabled? fail-closed helper, before the rate limiter touches Redis and before any LLM call. One flag disables both surfaces at once — there is no separate flag per surface.

This is a static config value, not a live-editable one — flipping it needs a commit + CI deploy, the same as every other Settings.ai.*_enabled flag documented in the Kill-Switch Flag Flips procedure. It is not the runtime-switchable kind (compare Ai::CoachModel.current in Switching the Coach Model, which reads from an admin-editable app_settings row with a 60-second cache) — Ai::SettingsGuard reads Settings.ai directly, which is the config gem's file-based settings, resolved at boot.

Current values, per environment (rails_api/config/settings/{development,staging,production,test}.yml):

Environment	docs_assistant_enabled
Development	false
Test	false
Staging	true
Production	true

To flip it, follow the Kill-Switch Flag Flips procedure against rails_api/config/settings/production.yml (or staging.yml), editing docs_assistant_enabled instead of coaching_enabled/moderation_enabled/batch_enabled.

What a user sees when disabled: both resolvers return { state: 'DISABLED', answer: nil, citations: [] } — a normal (non-error) GraphQL response. The widget renders UI-SPEC's "Disabled (in flight)" or "Disabled (at load)" state — see Docs & guide assistants — never an error toast.

Budget ceiling and what happens when it trips

Ai::BudgetGuard (rails_api/app/services/ai/budget_guard.rb) is an enforcing per-feature monthly spend ceiling, checked inside Ai::ServiceClient.chat (rails_api/app/services/ai/service_client.rb:80) after the kill switch and rate-limit gates but before any LiteLLM call. It is generic-with-opt-in, not docs_assistant-specific — any feature with a Settings.ai.<feature>_monthly_budget_cents key gets the same enforcement; docs_assistant is simply the first feature to opt in.

• Boundary is inclusive: month-to-date spend >= the ceiling blocks the request that would reach or cross it, not just spend already over it.

• Spend is summed from AiUsageEvent for the feature, for the current calendar month (rails_api/app/services/ai/budget_guard.rb:53).

• Ceilings today (rails_api/config/settings.yml:16 as the default, overridden per environment):

  Environment	docs_assistant_monthly_budget_cents
  Development	500¢ ($5, inherited default — no override file)
  Test	1000¢ ($10)
  Staging	500¢ ($5)
  Production	5000¢ ($50)

  Each is deliberately ~10% of that environment's global monthly_budget_cents — the ceiling shouldn't be able to consume the whole environment's AI budget by itself while still leaving headroom for coaching/moderation/batch spend (rails_api/config/settings.yml:10-14 comment).

What happens when it trips

The request does not get a distinct "budget exceeded" error. Ai::ServiceClient.chat returns nil — the identical contract it returns for the kill switch and the rate limiter — and both Ai::DocsAssistant::AnswerGuideQuestion/AnswerDocsQuestion treat a blank response as an upstream failure (fail_with(error: 'No response from the guide assistant', code: :upstream_unavailable), rails_api/app/interactions/ai/docs_assistant/answer_guide_question.rb:41). The GraphQL layer then raises the generic DOCS_ASSISTANT_UNAVAILABLE error ("Couldn't reach the guide/docs assistant — try again in a moment.") — the same error a real LiteLLM outage would produce. The widget renders UI-SPEC's generic "couldn't reach" terminal state, not a budget-specific one.

There is no Slack alert for this ceiling tripping. AiMetrics::AiBudgetAlertJob (rails_api/app/jobs/ai_metrics/ai_budget_alert_job.rb) only tracks the globalSettings.ai.monthly_budget_cents across every feature — it has no per-feature awareness of docs_assistant_monthly_budget_cents. The only trace a tripped ceiling leaves is a Rails.logger.info("[Ai::ServiceClient] budget ceiling reached feature=docs_assistant") line and the plateau it produces in AiUsageEvent cost sums for the feature.

Diagnosing "the assistant stopped answering"

1. Check the kill switch first — confirm docs_assistant_enabled is still true in the relevant environment's settings file.

2. Check month-to-date spend against the ceiling:

   gcloud run jobs execute enkidu-rails-console-production --region=us-central1 --wait
   # From inside the console session:
   AiUsageEvent.where(feature: 'docs_assistant', created_at: Time.current.beginning_of_month..Time.current).sum(:cost_cents)

   Compare against the ceiling table above. If spend has reached the ceiling, that's the cause — not an outage.

3. If neither explains it, treat it as a real upstream failure and follow API Key Rotation + Provider Swap / the LiteLLM health check in that same runbook.

Raising the ceiling

Edit docs_assistant_monthly_budget_cents in the target environment's settings file (rails_api/config/settings/production.yml etc.), commit, and deploy — same commit + CI deploy mechanics as the kill switch above. There is no admin-UI control for this ceiling today.

Rate limits

The two operations are not rate-limited the same way — the guide is anonymous and the docs assistant is always authenticated, so they need different keying:

Layer	askGuideQuestion (guide, anonymous)	askDocsQuestion (docs, authenticated)
GraphQL-layer, graceful	20/min, keyed current_user&.public_id || "anon:<ip>" via an ad hoc Rails.cache counter (query_type.rb:807-815)	20/min, keyed by real user_public_id via Ai::RateLimiter.check_and_increment (query_type.rb:858) — the same shared limiter and 20/min limit Ai::ServiceClient's own docs_assistant category enforces
rack-attack, per-IP, coarse	30/min per IP (rails_api/config/initializers/rack_attack.rb, throttle guide-assistant-ip) — deliberately coarser than the 20/min above, so a well-behaved caller never reaches it; exists to reject a flood at the Rack boundary before Rails routes/parses/executes a GraphQL query at all	none — an authenticated caller doesn't need a per-IP layer the way an anonymous public endpoint does
Per-request size	MAX_QUESTION_LENGTH = 1000 chars, enforced inside each Interaction (answer_guide_question.rb:23, answer_docs_question.rb:23) — closes the abuse vector none of the above bound: request size, not request count	same, 1000 chars

Both a rate-limited GraphQL-layer request and an IP-throttled rack-attack request return a RATE_LIMITED-coded GraphQL error before any LLM call — neither consumes budget.

Why the guide gets three layers and the docs assistant gets two: help.objectuve.com is Objectuve's first unauthenticated endpoint that spends money per request (~1.24¢/query measured in the spike). An IP throttle alone is trivially defeated by a distributed caller — that's why the budget ceiling above is a hard gate, not just an alert, and why the rack-attack layer exists at all even though the GraphQL-layer limit already covers well-behaved callers.

Tuning a rate limit

Same procedure as Rate-limit Tuning — edit the relevant constant (RATE_LIMITS['docs_assistant'] in service_client.rb, the hardcoded 20/30 in query_type.rb/rack_attack.rb, or MAX_QUESTION_LENGTH in either Interaction), run the relevant specs, commit, and deploy.

Corpus refresh (docs.objectuve.com only)

The guide corpus (79 KB, guide_site/*.md) is committed as a generated snapshot and rebuilt by its own CI drift guard — no manual refresh step. The docs corpus (8.7 MB / 464 files / 10,667 heading chunks, backing askDocsQuestion) is different: it's a Postgres-backed index (DocsCorpusChunk / DocsCorpusPosting) that must be rebuilt whenever docs/ content changes. .github/workflows/docs-corpus-refresh.yml now does this automatically — it runs on every push to master touching docs/** (or the indexer/rake task files), plus manual workflow_dispatch.

cd rails_api && bundle exec rake docs_corpus:refresh

(rails_api/lib/tasks/docs_corpus.rake, calling DocsCorpusIndexer.call — rails_api/app/services/docs_corpus_indexer.rb.)

Hard constraint, stated in the task's own desc and the indexer's class comment: this must run from a full monorepo checkout, with docs/ present as a sibling of rails_api/ — never from the deployed container. docs/ is not in the Rails image's build context (rails_api/ is the Docker build context — see .github/workflows/staging.yml's deploy-backend job). Running the task where docs/ is absent raises rather than silently wiping the existing corpus — Roy's review (PR #3081) caught that an empty docs/**/*.md glob would otherwise delete every row inside the same transaction with no error.

The task is idempotent — it always replaces the full index inside one transaction, so re-running against an unchanged docs/ tree changes nothing. Run it after any deploy that changes docs/ content; nothing else keeps the index in sync with what's actually in the corpus.

One-time IAM prerequisite: the workflow authenticates as github-actions-deployer (the same service account production.yml uses) and needs roles/cloudsql.client on that account to reach the Cloud SQL Auth Proxy — granted via gcp_setup/03_service_accounts.sh, which is a bootstrap script, not something any CI workflow invokes automatically. If this hasn't been applied to the live project yet, the workflow's "Start Cloud SQL Auth Proxy" step fails, misleadingly, as a 30s readiness timeout rather than a permissions error.

Cross-references

• What the assistants do, from a user's perspective — Docs & guide assistants
• Retrieval architecture, spike numbers, and the corpus-delivery decision — Embedding an AI knowledge-retrieval assistant into the docs sites
• General AI kill switches, budget alerts, rate-limit tuning, model switching — AI Runbook
• The BM25 count-question retrieval gotcha, now resolved — Gotchas

Last updated: 2026-09-13 (PR #3134: corpus refresh workflow IAM prerequisite noted)