Objectuve Docs

Appearance

Health check informational sub-checks gotcha ​

INFORMATIONAL_CHECKS in Rails code does not produce a neutral JSON signal ​

Applies to: Any health sub-check listed in HealthController::INFORMATIONAL_CHECKS that still performs a probe guaranteed to fail (e.g., a sidecar moved to a different service, a removed dependency).

HealthController (rails_api/app/controllers/health_controller.rb) supports a list of informational sub-checks:

ruby
INFORMATIONAL_CHECKS = %i[agent_runner].freeze

Checks in this list are excluded from the gating logic — they do not flip the overall status to "degraded" or the HTTP status to 503. The /health response will still return HTTP 200 with "status": "ok" even if an informational check fails.

However, the sub-check result is still written into the checks hash and emitted verbatim in the JSON body. If the underlying probe fails, the JSON will contain:

json
"agent_runner": {
  "status": "down",
  "message": "Failed to open TCP connection to localhost:4001 (Connection refused)"
}

Any monitoring system that reads raw checks.*.status values — including Penny's health-check autopilot — will see "status": "down" and fire an incident, regardless of the HTTP status code or the top-level "status" field.

The foot-gun: INFORMATIONAL_CHECKS is a Ruby-level classification. It has no effect on the JSON output shape. Monitoring systems cannot read Ruby.

What happened (OBJ-992, 2026-06-17) ​

PR #1173 (f9c43138f) correctly migrated the agent_runner sidecar from enkidu-api-staging (web) to enkidu-worker-staging (worker), and correctly added agent_runner to INFORMATIONAL_CHECKS so the overall /health verdict would not be affected. However, probe_agent_runner continued attempting a TCP connection to localhost:4001 — which is guaranteed to fail on the web service because the sidecar no longer runs there.

The result: every /health response on the web service included "agent_runner": { "status": "down", ... }. Penny's monitor, which scans individual checks.*.status values, fired a high-priority staging degradation incident. The staging environment was actually healthy; the AI Workforce was running on the worker service where it belongs.

Fix ​

For a sub-check that will always fail (permanently moved or removed dependency), return a neutral non-"down" status from the probe itself:

ruby
def probe_agent_runner
  { status: 'not_applicable', message: "agent_runner sidecar runs in enkidu-worker-#{Rails.env}" }
end

This removes the bad signal at the source. Penny's monitor ignores any status other than "down", so no monitoring changes are needed.

When returning not_applicable:

  • Keep the check wired through check_agent_runner — it still appears in the JSON body and on the status page.
  • Keep the check in INFORMATIONAL_CHECKS — this documents intent and prevents it from gating the overall verdict even if the probe is later updated to return a non-ok status.
  • Update the spec to expect "not_applicable":
ruby
it 'reports not_applicable because the sidecar runs in enkidu-worker-{env}' do
  get '/health'
  expect(response).to have_http_status(:ok)
  body = JSON.parse(response.body)
  expect(body['checks']['agent_runner']['status']).to eq('not_applicable')
end

Alternative: update the monitor ​

A more robust long-term fix is to update the monitoring logic to trust the HTTP status code and top-level "status" field rather than scanning individual sub-checks. Only alert when response.status >= 500 OR body.status == "degraded". This makes monitoring resilient to any future informational sub-check that returns a non-ok value.

The source fix (returning not_applicable) is faster and more targeted. The monitor fix is a systemic improvement. Both are valid; together they are comprehensive.

Rule of thumb ​

If a health sub-check is both informational AND will always fail, it should return a neutral status (e.g. not_applicable) rather than letting the underlying probe fail and emit "down". INFORMATIONAL_CHECKS prevents the check from gating HTTP 200 — but it does not change what the JSON body contains.

This incident also surfaced a latent fragility in the test suite. The CI failure on PR #1180 was an unrelated spec:

spec/interactions/user_identity/backfill_onboarding_completion_spec.rb:25

The failure was caused by TimeWithZone#iso8601 including the local timezone offset when Time.zone is non-UTC. If a prior spec in the same RSpec shard sets Time.zone and doesn't reset it, iso8601 produces "2026-06-17T17:19:32-04:00" instead of "2026-06-17T21:19:32Z", breaking timestamp comparisons.

Root cause: rails_api/spec/rails_helper.rb includes after(:each) hooks for DomainEvents.reset_for_testing! and Timecop.return, but not for Time.zone:

ruby
# rails_helper.rb — missing:
config.after(:each) { Time.zone = 'UTC' }

Fix applied in this incident: Changed .iso8601 to .utc.iso8601 in backfill_onboarding_completion.rb — this is invariant to Time.zone state and correct regardless.

Root cause fixed in OBJ-994 / PR #1191: config.after(:each) { Time.zone = 'UTC' } has been added to rails_api/spec/rails_helper.rb alongside the existing DomainEvents and Timecop resets. This closes the underlying leak — no future spec in a sharded run can contaminate Time.zone for a subsequent example.

  • rails_api/app/controllers/health_controller.rb — INFORMATIONAL_CHECKS, check_agent_runner, probe_agent_runner
  • rails_api/spec/requests/health_spec.rb — informational sub-check contract tests
  • rails_api/spec/rails_helper.rb — Time.zone = 'UTC' reset added in after(:each) (OBJ-994 / PR #1191)
  • docs/operations/observability.md — /health endpoint structure and component list
  • docs/operations/incident-response.md — Agent Runner failure mode and recovery
  • docs/operations/ai-usage-event-features-allowlist-gotcha.md — a related but distinct silent-drift failure shape gating the same /health endpoint (stale allowlist vs. probe-always-fails)

Last updated: 2026-07-22

Loading…