Sidekiq dead set runbook

Self-serve procedure for clearing a stuck Sidekiq dead set from the GitHub Actions tab — no local gcloud or cloud credentials required.

When to use this

• /health reports "status": "degraded" with checks.sidekiq.status == "degraded"
• The Playwright smoke pre-flight is failing at "Phase 4 (health redis+sidekiq warm)" on an unrelated PR
• checks.sidekiq.dead in the /health response is at or above SIDEKIQ_DEAD_THRESHOLD (default 25)

Symptom → cause chain

HealthController#check_sidekiq (rails_api/app/controllers/health_controller.rb:64-79) flips sidekiq.status to "degraded" (and the top-level /health status + HTTP code to 503) whenever Sidekiq::DeadSet#size exceeds SIDEKIQ_DEAD_THRESHOLD. The Playwright smoke pre-flight's Phase 4 step hard-asserts checks.sidekiq.status == "ok" (.github/workflows/playwright-smoke.yml:316-329) — so a full dead set fails a required check on every open PR, not just the one whose job actually failed.

This is shared staging/production infrastructure, so the PR under test is almost never the cause. On 2026-07-22, this fired for two unrelated PRs (OBJ-1655/#1716, OBJ-1656/#1718) in one day because a single job — AiMetrics::RecordAiUsageEventJob, failing identically since 2026-06-10 on a missing feature: "coach_synopsis" allowlist entry — had quietly filled the dead set past the threshold. See AiUsageEvent FEATURES allowlist drift gotcha for that specific root cause; this page covers the generic remediation.

Diagnose before clearing

Run mode: summary first and paste the breakdown into the blocking issue before clearing anything. Clearing destroys the only evidence of what actually failed — the OBJ-1650 root cause (the allowlist gap above) was only findable because the dead jobs were still sitting in the set. Treat summary as a required first step, not optional polish.

Do not just raise SIDEKIQ_DEAD_THRESHOLD. Bumping the threshold hides genuine, recurring job failures instead of fixing them — it's the same silent-drift smell this whole page exists to avoid, not a fix.

Step-by-step

1. Go to the repo's Actions tab → Sidekiq Dead Set workflow → Run workflow.
2. Fill in the inputs:
   • environment — staging or production (default staging)
   • mode — summary (read-only breakdown, default) or clear (deletes matching entries)
   • job_class — optional exact class filter, e.g. AiMetrics::RecordAiUsageEventJob (blank = no class filter)
   • older_than_days — optional; only entries older than N days (blank = no age filter)
   • confirm — set to yes to allow an unfiltered wipe (required only when both job_class and older_than_days are blank)
3. Run with mode: summary first. Read the per-job-class breakdown and the trailing JSON line in the run's logs (also echoed to the job summary) — total dead count, threshold, and whether it's currently over.
4. Paste that breakdown into the blocking issue as a durable record of what was in the dead set before anything was deleted.
5. Re-run with mode: clear, scoped to the job class you just identified as the offender (job_class: AiMetrics::RecordAiUsageEventJob, for example). Prefer this over an unfiltered wipe — it removes the confirmed problem class and preserves any other dead entries that might indicate a separate, still-unfixed bug.
6. Confirm the run's logs show deleted matching your expectation and dead_after back under the threshold. /health's sidekiq.status should return to ok on the next check.

An unfiltered wipe (both job_class and older_than_days left blank) requires confirm: yes — the workflow's underlying sidekiq:dead:clear rake task refuses with a non-zero exit otherwise. Use this only when you've already reviewed the full summary breakdown and have no reason to keep any of it around as evidence.

Filters reference (as shipped)

The workflow dispatches bundle exec rake sidekiq:dead:summary or sidekiq:dead:clear (rails_api/lib/tasks/sidekiq.rake) against the currently-deployed Cloud Run image for the chosen environment, passing these as env vars:

Workflow input	Rake env var	Behavior
job_class	JOB_CLASS	Exact class-name match; only matching entries are deleted
older_than_days	OLDER_THAN_DAYS	Only entries whose at is older than N days
confirm	CONFIRM	Must be yes to allow deletion when both filters above are blank
—	LIMIT	Safety cap on deletions per run, default 1000 (not exposed as a workflow input; set only if invoking the rake task directly)

Comma caveat: the workflow passes these through gcloud's --set-env-vars, which is comma-joined. A job_class or older_than_days value containing a literal comma will silently truncate the flag rather than error. Job class names don't normally contain commas, so this is a low-likelihood trap, not a blocking concern — but if a dispatch behaves unexpectedly with an unusual filter value, check for this first.

Why this can't be verified before merge

The workflow resolves the currently-deployed Cloud Run image for the target environment before running — it always operates against whatever is live, not against the PR that introduced it. Its own first end-to-end dispatch is necessarily a post-merge, post-redeploy step, not something CI can gate on pre-merge.

Verify against

• rails_api/app/controllers/health_controller.rb:64-79 — the threshold/degraded logic
• rails_api/lib/tasks/sidekiq.rake — sidekiq:dead:summary / sidekiq:dead:clear, the actual env-var names as shipped
• .github/workflows/sidekiq-dead-clear.yml — the workflow_dispatch inputs
• .github/workflows/playwright-smoke.yml:316-329 — Phase 4, the gate this remediates

Related

• Observability § Health check endpoint — the sidekiq check's degraded path
• AiUsageEvent FEATURES allowlist drift gotcha — the OBJ-1650 incident that filled the dead set on 2026-07-22
• Health check informational sub-checks gotcha — a related but distinct silent-drift shape on the same endpoint

---

Last updated: 2026-07-22