Objectuve Docs

Appearance

AiUsageEvent FEATURES allowlist drift gotcha ​

A validation allowlist duplicated between a model and its distributed callers drifts silently and permanently kills background jobs ​

Applies to: Any inclusion: { in: SOME_LIST } validation where SOME_LIST is hand-maintained in one file but the values that must satisfy it originate from call sites scattered across the codebase.

AiUsageEvent (rails_api/app/models/ai_usage_event.rb) records metrics for every LiteLLM response with a usage block, validated against a hardcoded allowlist:

ruby
class AiUsageEvent < ApplicationRecord
  FEATURES = %w[
    coaching milestones insights check_ins description moderation
    daily_insight_pack coach_synopsis coach_at_risk_nudge compassionate_break
    goal_draft goal_suggestions feedback_summary welcome_back weekly_digest
  ].freeze

  validates :feature, presence: true, inclusion: { in: FEATURES }
end

The feature: value isn't chosen from FEATURES — it's a free-text string literal passed by each caller into Ai::ServiceClient.chat(..., metadata: { feature: '...' }) (rails_api/app/services/ai/service_client.rb). Nothing ties a new caller's string to the allowlist at write time. FEATURES and its callers are two independent lists that happen to need to agree, with no compiler, linter, or test enforcing that until OBJ-1650 added one.

Why this fails permanently instead of just once ​

Ai::ServiceClient#record_usage enqueues AiMetrics::RecordAiUsageEventJob with the caller's raw feature string. The job calls AiUsageEvent.create!(feature: args[:feature], ...) (rails_api/app/jobs/ai_metrics/record_ai_usage_event_job.rb). If feature isn't in FEATURES, this raises ActiveRecord::RecordInvalid — a deterministic validation failure, not a transient one. Every retry fails identically, so Sidekiq's default ApplicationJob retry policy exhausts and the job lands in the dead set. It will sit there forever; nothing about the failure is retryable.

Because Ai::ServiceClient#safe_record_usage swallows the exception at enqueue time and the job only logs/Sentry-captures before re-raising (metrics recording must never break the user-facing AI response — see the job's own comment), the drift produces no user-visible symptom. It just quietly piles dead jobs into Sidekiq until the count crosses SIDEKIQ_DEAD_THRESHOLD (default 25) and trips the /health gate — see docs/operations/health-check-informational-gotcha.md for a related but distinct silent-drift failure shape (a check that's guaranteed to fail vs. an allowlist that drifted out of sync), and Sidekiq dead set runbook for how to clear the resulting dead set.

What happened (OBJ-1650, 2026-07-22) ​

Dave's scheduled staging health check caught /health reporting degraded (sidekiq.dead: 26, one over the threshold). Diagnosis traced it to AiUsageEvent::FEATURES never being updated as new feature: call sites were added to coach_service.rb and generate_daily_insight_pack_job.rb — 4 values (daily_insight_pack, coach_synopsis, coach_at_risk_nudge, compassionate_break) were already dying in production logs; 4 more (goal_draft, goal_suggestions, feedback_summary, welcome_back) were latent, not yet observed failing.

The fix PR (#1715) itself demonstrated the trap twice more during review:

  • Round 1 audit scoped to app/services/ai/ and app/jobs/ai_coaching/ only — missed app/interactions/weekly_digest/assemble_digest_data.rb, a 9th caller passing feature: 'weekly_digest', caught by review before merge rather than by a test.
  • First regression-guard spec scanned an enumerated directory list (app/{services/ai,jobs/ai_coaching}/**/*.rb) — the same shape of gap as the original bug, just moved into the test. It missed app/jobs/content_moderation/screen_content_job.rb too (using an already-allowlisted value, so not actively broken, but unguarded against future rename-drift).

Fix ​

Expand AiUsageEvent::FEATURES to include every live feature: value, and replace any enumerated-directory scan with one that finds callers by what they are, not where they live:

ruby
# rails_api/spec/models/ai_usage_event_features_drift_spec.rb
caller_files = Dir.glob(Rails.root.join('app/**/*.rb')).select do |path|
  File.read(path).include?('Ai::ServiceClient.chat(')
end
literal_features = caller_files.each_with_object(Set.new) do |path, set|
  File.read(path).scan(/feature:\s*'([^']+)'/).each { |(value)| set << value }
end

missing = literal_features - AiUsageEvent::FEATURES
expect(missing).to be_empty, "Found feature literal(s) not in AiUsageEvent::FEATURES: ..."

Scoping the scan to files that actually call Ai::ServiceClient.chat( — rather than a repo-wide feature:\s*'...' literal scan — matters too: app/interactions/user_identity/update_privacy_settings.rb has an unrelated PostHog feature: 'privacy_controls_v1' event key that would false-positive as "missing" under a naive repo-wide scan.

Note: even after the code fix ships, jobs already in Sidekiq's dead set do not auto-retry. They need a manual Sidekiq::DeadSet#retry_all (or discard) run against the affected environment before /health returns to ok — the Sidekiq dead set runbook covers the discard path via the sidekiq:dead:clear rake task, self-serve from the Actions tab.

Rule of thumb ​

If a validation allowlist's members are supposed to match values chosen independently by callers elsewhere in the codebase, the allowlist will drift — silently, because the failure is swallowed by design (metrics/logging paths often are), and permanently, because a validation failure is not a transient one Sidekiq will retry past. Guard it with a test that discovers callers by behavior (e.g. "calls this method") rather than by an enumerated directory list, which just relocates the same drift risk into the test itself.

  • rails_api/app/models/ai_usage_event.rb — FEATURES allowlist
  • rails_api/app/services/ai/service_client.rb — record_usage / safe_record_usage, where callers supply feature:
  • rails_api/app/jobs/ai_metrics/record_ai_usage_event_job.rb — where the validation failure actually raises and dies
  • rails_api/spec/models/ai_usage_event_features_drift_spec.rb — regression guard
  • rails_api/app/controllers/health_controller.rb — SIDEKIQ_DEAD_THRESHOLD gate that surfaces the drift
  • docs/operations/health-check-informational-gotcha.md — a related but distinct silent-drift failure shape gating the same /health endpoint
  • Sidekiq dead set runbook — self-serve remediation for the dead set this drift fills

Third sibling instance (OBJ-1660, 2026-07-22): CoachInteraction::FEATURES (rails_api/app/models/coach_interaction.rb) drifted the same way — missing 5 of the coach-voice features that reach Ai::ServiceClient.chat, with the same silent failure mode (an unlisted feature is skipped, not raised). Unlike this doc's allowlist, CoachInteraction::FEATURES is deliberately a curated subset of AiUsageEvent::FEATURES (it excludes non-coach-voice features like moderation), so a plain superset guard was the wrong shape; the fix added a sibling AUDIT_EXEMPT_FEATURES constant and a bidirectional partition spec instead. See ai-workforce.md — Vera.

Last updated: 2026-07-22

Loading…