Teams billing incident runbook

Internal 3-step procedure for a Teams billing incident (Stripe webhook misbehaving, a team double-charged, checkout provisioning a bad subscription, etc.). Teams billing is closed-beta (v4.0 Teams V1, Phase 6) — solo-operator, no on-call rotation. See Incident Response for the general platform runbook; this page covers the Teams-specific billing surface only.

If checkout is failing with "Stripe pricing has not been configured" (not an incident — a setup gap), see Phase 7 Teams § Billing & provisioning for the stripe_price_id wiring path and the "config existing ≠ config wired" gotcha.

When to use this

• Stripe webhook events for team checkouts are erroring, double-firing, or provisioning incorrect TeamSubscription rows
• A beta team reports being charged incorrectly (wrong seat count, double charge, charged after cancel)
• Webhooks::StripeController is throwing on checkout.session.completed for a team payload
• A refund request comes in from a beta participant

Step 1 — Kill teams_billing_v1

teams_billing_v1 gates all team-checkout webhook processing at the handler entry, before Teams::ProvisionTeamSubscription ever runs (rails_api/app/controllers/webhooks/stripe_controller.rb:24-42). Flipping it off does not touch already-provisioned TeamSubscription rows or in-app reads — it only stops new team checkout events from writing:

def teams_billing_enabled?(event)
  user = User.find_by(public_id: event.data.object.metadata['objectuve_user_id'])
  FeatureFlagService.enabled?('teams_billing_v1', user: user)
end

With the flag off, Stripe still gets a 200 OK (so Stripe doesn't retry-storm) but Objectuve makes no writes for that event.

1. Go to the PostHog feature flags UI (project 368400).
2. Open teams_billing_v1 and set Enabled to off (active: false). Takes effect within seconds, no deploy required — this is the documented kill-switch stage of the flag lifecycle. See Feature Flags § Step 3.
3. Confirm: trigger or wait for the next team checkout event and verify no new TeamSubscription/PaymentRecord writes occur (Rails logs or Sentry breadcrumbs for Teams::ProvisionTeamSubscription).

This only stops new checkout provisioning. The top-level teams_enabled Teams-visibility flag was retired at GA (PR #1498) and no longer exists in code — there is currently no lever to hide Teams entirely for all users. Teams itself stays visible regardless of this kill switch.

Step 2 — Inspect the Stripe dashboard

Team billing uses its own Stripe customer per team (created by Stripe from customer_email at Checkout — never the billing owner's personal Supporter customer ID; see StripeService.create_team_checkout_session, rails_api/app/services/stripe_service.rb:79-96). Look the team up on both sides:

1. In Rails console (or admin dashboard's Teams monitoring page), find the team and its subscription:

   team = Team.public_find('<team_public_id>')  # or Team.find_by(slug: '...')
   sub = team.subscription
   sub.stripe_subscription_id
   sub.stripe_customer_id
   sub.status            # trialing / active / past_due / grace / canceled
   sub.current_period_end

   'active' is set by Teams::ProcessInvoicePaid on invoice.paid (OBJ-1411), which transitions trialing/past_due/grace → active and refreshes current_period_end from Stripe on every event, including renewals. canceled is deliberately not a recovery source — a canceled team must re-subscribe via a fresh checkout rather than being silently reactivated by a delayed Stripe dunning success. See Phase 7 Teams PRD §7.1's webhook table.

2. In the Stripe dashboard, search by sub.stripe_customer_id or sub.stripe_subscription_id to see the actual invoice/payment history, failed charges, and webhook delivery log for that customer.
3. Cross-check against PaymentRecord for the same team_subscription:

   PaymentRecord.where(team_subscription: sub).order(created_at: :desc)

4. If webhook events are missing or errored, Stripe's dashboard → Developers → Webhooks → the endpoint's event log shows delivery attempts and response codes — cross-reference timestamps against Step 1's kill-switch window to rule out events dropped while teams_billing_v1 was off.

The failed-payment state machine (Teams::PaymentFailedJob, daily sweep) transitions past_due → grace → canceled automatically at 7 + 7 days (TeamSubscription::PAST_DUE_GRACE_DAYS / GRACE_CANCEL_DAYS, rails_api/app/jobs/teams/payment_failed_job.rb). Confirm whether a status change was this automatic sweep or a manual/webhook action before assuming an incident.

Step 3 — Refund procedure

Refund policy (PRD §12, decision #9): 7-day window, manual via Stripe dashboard. There is no in-app or GraphQL refund mutation — StripeService has no refund method by design.

1. Confirm the team is within the refund window — Teams::CancelTeamSubscription#within_refund_window? checks subscription.created_at >= 7.days.ago (rails_api/app/interactions/teams/cancel_team_subscription.rb:56-58).
2. If billing-owner-initiated cancellation is also needed, the cancelTeamSubscription mutation (owner-only, TeamAccessPolicy#manage_billing) already handles this correctly: it calls StripeService.cancel_subscription (cancel_at_period_end: true) and, if still inside the 7-day window, soft-deletes the Team in the same transaction. Outside the window the subscription is marked canceled but the team stays intact.
3. For the actual refund of already-captured payment, go to the Stripe dashboard → the customer's payment/invoice → Refund. Stripe refunds are not automated for Teams; issue the refund manually against the specific charge, not the subscription.
4. After refunding, verify locally: sub.reload.status should read canceled if the team also canceled; PaymentRecord for the refunded charge should be updated or annotated so support history stays accurate.
5. Notify the beta participant per the beta compensation terms (docs/product/phase-7-teams.md § open decisions #10) if the refund affects their locked beta discount.

What T3 changed: invoice.paid + invoice.payment_failed both shipped

This section is the follow-up pass this runbook predicted back when 'active' was still unreachable (OBJ-1411, OBJ-1412 — see the Phase 7 Teams PRD § 7.1 status notes). The whole TeamSubscription lifecycle is now Stripe-webhook-driven:

• invoice.paid → Teams::ProcessInvoicePaid moves trialing/past_due/grace → active, refreshes current_period_end from Stripe on every event (including plain renewals), and fires trial_converted to PostHog exactly once per conversion (never on a renewal of an already-active sub). canceled subs are deliberately never reactivated here — see Step 2 above.
• invoice.payment_failed → Teams::ProcessInvoicePaymentFailed moves trialing/active → past_due and stamps current_period_end = Time.current, so the 7+7-day grace/cancel timers start from the failure, not a stale date.
• Teams::TrialExpiredJob (daily) is now a safety-net alarm, not a writer — it no longer sets past_due. It logs + pages Sentry if a subscription is still trialing past trial_ends_at, which now means the invoice.payment_failed webhook is late or missing, not "the daily job caught up." Treat that Sentry alert as the first place to look if a conversion or a failure isn't landing.
• Teams::PaymentFailedJob (daily) still runs the past_due → grace → canceled sweep off current_period_end, but auto-cancellation now calls StripeService.cancel_subscription before writing canceled locally, and skips the local write entirely (fails closed, retries the next day) if Stripe still reports the subscription active — closing the "still being charged after we marked it canceled" gap.
• Neither the teams_billing_v1 kill switch (Step 1) nor teams_enabled gates any of the above — Step 1's kill switch only stops new checkout provisioning; conversions and failures for already-provisioned subscriptions process regardless of its state.

How to verify a conversion landed

1. Rails console: sub.reload.status == 'active' and sub.current_period_end is a future Stripe-sourced date (not the original trial end).
2. PaymentRecord.where(team_subscription: sub).order(created_at: :desc).first — a fresh row with a new stripe_event_id (never the original checkout.session.completed record's id — that one is never overwritten).
3. PostHog Activity / Events explorer (project 368400) — one trial_converted event, surface: 'teams', distinct_id = the billing owner's public_id. Absence of this event but status == 'active' on a genuine first conversion means the sub was already active when the event landed (a renewal, not a conversion) — check PaymentRecord history to tell the two apart.

If a conversion (or a failure transition) didn't land

1. Check Sentry for a [Teams::TrialExpiredJob] warning naming the team — its presence means the invoice.paid/invoice.payment_failed webhook hasn't arrived at all, so start with Stripe delivery, not Rails.
2. Stripe dashboard → Developers → Webhooks → the endpoint's event log: find the invoice.paid/invoice.payment_failed event for this subscription and check its delivery status and response code. A 400 means signature verification failed (see Step 2's kill-switch cross-reference); anything else delivered-but-no-effect points to step 3 or 4 below.
3. Confirm TeamSubscription.find_by(stripe_subscription_id: …) actually resolves for the invoice's subscription id — StripeService.invoice_subscription_id reads invoice.parent.subscription_details.subscription under the pinned API version, not the legacy top-level invoice.subscription. A subscription recreated in Stripe (new id) without updating the local row is a common cause of a silent no-match, which the handler treats as "not a team event" and routes to Billing::ProcessStripeWebhook instead — check Rails logs / Sentry breadcrumbs there too.
4. Check the sub's local status at the time the event arrived: Teams::ProcessInvoicePaid ignores (succeed_ignored, no-op) any status outside trialing/past_due/grace/active; Teams::ProcessInvoicePaymentFailed ignores anything outside trialing/active. A sub already canceled intentionally does not react to either — that's a fresh-checkout-required product decision, not a bug.
5. Rule out a replay: PaymentRecord.exists?(stripe_event_id: event.id) — a duplicate delivery of an already-processed invoice.paid event no-ops by design.
6. To force a fresh attempt in staging/test mode, use the Stripe CLI: stripe trigger invoice.paid (or invoice.payment_failed), or resend the specific event from the dashboard's event log — do not hand-write a synthetic payload for this surface (see OBJ-1411's live-fire verification requirement); replay the real event.

Escalation

Solo-operator, no on-call rotation — same posture as Incident Response. If the incident is isolated to Teams billing, Step 1 (kill switch) is enough to stop the bleeding — there is no broader "hide all of Teams" lever available or needed.

North Star metrics snapshot job (METRIC-4)

Not an incident procedure — routine ops for the daily job that feeds the 12-month North Star dashboard (PRD §10). See Phase 9 § Scoping findings for why this job emits point-in-time aggregates only, never a churn count.

What it emits

Teams::SnapshotNorthStarMetricsJob runs daily via Crono at 03:45 — deliberately after the billing sweeps at 03:00–03:30, so it reads post-sweep state — and calls Teams::ComputeNorthStarMetrics, then sends one team_metrics_snapshot event to PostHog:

Property	Definition
mrr_cents	Σ seat_count × monthly price over active + past_due + grace subscriptions (revenue-recognized; trialing is $0 and excluded). Annual plans normalize /12, mirroring AdminStatsService's Supporter-tier convention.
paying_teams	Count of revenue-recognized subscriptions
trialing_teams	Count of trialing subscriptions, reported separately — never folded into MRR or paying_teams
avg_seats_purchased	Mean seat_count (what paying teams are billed for) across revenue-recognized subscriptions
avg_seats_occupied	Mean TeamMembership count per paying team (what they actually filled) — the gap between this and avg_seats_purchased is itself a churn signal

Every event carries surface: 'teams' explicitly (PosthogNotifier defaults to surface: 'admin' otherwise) and distinct_id: 'system' — a synthetic ID, since a company-wide aggregate has no user and PosthogNotifier.capture silently no-ops on a blank distinct_id. A deterministic per-day uuid (derived from team_metrics_snapshot:<date>) means a same-day double-fire dedupes to one PostHog datapoint, not two.

Orphaned TeamSubscription rows (left behind by a Team soft-delete path that skipped callbacks — TeamSubscription itself is not acts_as_paranoid) are excluded from every metric above: .joins(:team) is an inner join, so it inherits Team's paranoia default scope.

How to verify it ran

• PostHog: Activity / Events explorer (project 368400) — filter by event team_metrics_snapshot. One event per day, timestamped ~03:45 UTC.
• Logs: the job only logs on failure — [Teams::SnapshotNorthStarMetricsJob] Failed: <message> in Rails logs / Sentry. There is no success log line (Crono jobs are silent by default); a fresh PostHog event plus the absence of a failure log is the "it ran" signal.
• A PostHog delivery failure alone does not raise or alert. PosthogNotifier.capture rescues its own StandardError, logs [posthog] capture failed (team_metrics_snapshot): ..., and returns — the job's rescue only fires if Teams::ComputeNorthStarMetrics.call itself raises (a real Postgres/data problem), not a PostHog-side outage. If the dashboard goes stale with no Sentry error, check Rails logs around 03:45 for that [posthog] capture failed line specifically.

How to disable it

Remove the Crono.perform(Teams::SnapshotNorthStarMetricsJob).every 1.day, at: '03:45' line from rails_api/config/cronotab.rb and deploy. Blast radius: the North Star dashboard goes stale — nothing else breaks. The job is read-only against Postgres (no writes beyond the PostHog capture itself), so disabling it has zero effect on billing, checkout, or any other Teams surface.

Crono footgun

Never add at: to a sub-day period (e.g. every 1.hour, at: '...'). Crono::Period#parse_at raises at boot in that case and silently kills every cron job in the app, not just the offending one (cronotab.rb:18-21 documents this in-repo). every 1.day, at: '03:45' — a full-day period — is safe; this job's own registration is the reference example to copy.

---

Last updated: 2026-07-15 — recorded the OBJ-1411/OBJ-1412 follow-up pass: invoice.paid + invoice.payment_failed are both webhook-driven now, TrialExpiredJob is a safety-net alarm not a writer, auto-cancellation calls Stripe first, and conversion verification/failure triage steps.