Stale payment record reconciliation runbook

Response procedure for the #alerts page from Billing::ReconcileStalePaymentRecordsJob — a PaymentRecord stuck pending for 2+ hours where Stripe's own Checkout session confirms paid. See Alerting § Stale Payment Record Reconciliation for the alert definition. This page covers the personal-Supporter reconciliation alert only; for Teams billing incidents (Stripe webhook misbehaving on a team checkout, a double-charged team, etc.) see the sibling Teams Billing Runbook — don't merge the two, they cover different PaymentRecord populations (team_subscription_id vs. personal).

When to use this

• #alerts receives :rotating_light: (<env>) N PaymentRecord(s) show Stripe payment succeeded but never got reconciled locally: <public_id, ...> from Billing::ReconcileStalePaymentRecordsJob.

Background: what this alert can and can't mean

The job pages only when Stripe's own livemode flag is true on the Checkout session (rails_api/app/jobs/billing/reconcile_stale_payment_records_job.rb:59-73, OBJ-2852) — a paid test-mode session is logged at warn and never pages. That gate means:

• Production: unchanged — this alert is only ever a real, unreconciled payment.
• Staging: this alert should never fire under normal operation, because every staging Stripe session is test-mode (cs_test_...). If it does fire on staging, treat it as a secret/config anomaly (staging holding a live Stripe key) rather than a payment incident — check STRIPE_SECRET_KEY in the staging deploy manifest for drift before doing anything else.

This section exists because OBJ-2852 found the opposite: nine identical staging pages in 30 days, all the same test-mode smoke-test artifact, misread at first glance as a production incident because the alert carried no environment tag. The env tag ((#{Rails.env}), matching Email::DeliveryCanaryVerifyJob's (staging) format) and the livemode gate both landed together specifically to prevent a repeat.

Step 1 — Read the environment off the alert

The alert message is prefixed (production) or (staging) — read this first; it tells you which of the two paragraphs above applies before you touch anything.

Step 2 — Confirm livemode on the Stripe Checkout session

For each public_id in the alert:

record = PaymentRecord.public_find('<public_id>')
record.stripe_checkout_session_id   # cs_live_... or cs_test_...

Look the session up in the Stripe dashboard (toggle to the matching Live/Test mode) and confirm payment_status == 'paid' and livemode on the session. The job already checked this before paging — this step is a human double-check, not a re-derivation — but it's also where you'd catch the staging-anomaly case from the background section above (a cs_live_ session id showing up in a staging alert).

Step 3 — Look up the record and the user's supporter state

record = PaymentRecord.public_find('<public_id>')
record.status                 # expect 'pending'
record.amount_cents
record.plan_id
user = record.user
user.email
user.supporter?                # false if the webhook never landed
user.supporter_tier            # nil / 'monthly' / 'yearly' / 'lifetime'
user.supporter_until

If user.supporter? is already true, a later event (retry, manual fix, or a duplicate webhook that did land) already reconciled the account — this is a stale-record cleanup, not a live incident. Skip to Step 4's dismiss path.

Step 4 — Reconcile or dismiss

Reconcile (the payment is real and the user still isn't upgraded)

Billing::Subscribers mutates User/PaymentRecord state only in response to the billing.payment_processed domain event, published by Billing::ProcessStripeWebhook.call(event:) (rails_api/app/interactions/billing/process_stripe_webhook.rb:19-37) from a real Stripe::Event. Do not hand-write a synthetic payload — the same rule the Teams Billing Runbook uses for its own webhook replays.

1. In the Stripe dashboard → Developers → Webhooks → the endpoint's event log, find the checkout.session.completed event for this session.
2. Resend it from the dashboard's event log. ProcessStripeWebhook#already_processed? guards on PaymentRecord.exists?(stripe_event_id: event.id), so a resend is idempotent — it will not double-process if it somehow already landed.
3. If no checkout.session.completed event exists in Stripe's log at all (the webhook was never sent, not just dropped in-flight), escalate — that's an endpoint-registration or delivery problem outside this runbook's scope, not something a resend can fix.

Verification

record.reload.status                 # 'completed'
record.stripe_event_id               # now present
user.reload.supporter?               # true
user.supporter_tier                  # matches the session's plan_slug

Dismiss (the record is a smoke/fixture artifact, not real money)

Only take this path when Step 2/3 confirm the session is test-mode and/or the record traces to a known smoke fixture (e.g. a +clerk_test checkout account) — never as a shortcut around Step 2 on a live-mode session. This procedure is exactly what was run for OBJ-2852's 8xDz9OOV staging artifact, recorded here as-run rather than reconstructed:

1. Guard before writing. A one-shot manual write against a known row needs redundant, human-verifiable conditions — every one of them must pass, and any single failure means stop:

   record = PaymentRecord.find_by(public_id: '<public_id>')
   guard = record.status == 'pending' &&
           record.stripe_checkout_session_id&.start_with?('cs_test_') &&
           record.user.email.include?('+clerk_test')
   raise "guard failed, do not proceed" unless guard

   This is deliberately a different (and stricter) check than the job's own livemode gate — a human confirming this specific row wants every corroborating signal available, where the job's automated gate wants exactly one authoritative signal (livemode) and treats every extra predicate as a way to silently miss a real payment. Don't try to reconcile the two by loosening this guard.

2. Soft-delete, not hard-delete.

   record.destroy   # acts_as_paranoid (paranoia gem): sets deleted_at, row retained

3. Confirm the page stops.

   PaymentRecord.pending.count   # should drop to reflect the removed record

4. Reversible if this was a mistake:

   PaymentRecord.with_deleted.find_by(public_id: '<public_id>').restore

Run this as a throwaway one-off (Rails console via gcloud run jobs execute, or a scoped Cloud Run job with maxRetries: 0, deleted after) — not as a standing rake task or a change to the job itself. This is a manual incident response action, not automated behavior.

What this runbook does not cover

• Escalating dedup so a genuinely stuck record doesn't re-page silently forever, and whether the job should auto-heal instead of only detecting — tracked separately, not yet resolved (OBJ-2859).
• Teams billing incidents — see Teams Billing Runbook.

Last updated: 2026-08-22 (OBJ-2852: initial version, written alongside the alert's env-tag and livemode gate)