Objectuve Docs

Appearance

v4.43 — Email Delivery Canary ​

A daily synthetic email now proves transactional mail is reaching real inboxes — and proves it's still checking, not just staying quiet.

Summary ​

Before this milestone, Objectuve's only signal for transactional email health was raise_delivery_errors = true, which raises on SMTP-level refusal but says nothing about a message SMTP accepted and a recipient server later bounced. Josh's 2026-08-21 investigation found Mailtrap correctly configured — token valid, all six Cloud Run services wired, objectuve.com domain verified — while the delivery dashboard read Delivered 0 / Bounce 100% over 30 days. That reading was fully explained by staging smoke traffic to throwaway addresses that don't exist at Google Workspace, plus zero real production sends in two weeks. Nothing was actually broken. That was the problem: a revoked token, a bad DSN, a de-verified sending domain, or a blocklisted IP would have produced the identical dashboard, and nothing in the stack would have alerted on it.

Four phases, shipped in a single calendar day directly to master, close that gap. A Crono job sends one synthetic email per environment every morning; a second job resolves its real Mailtrap delivery status and pages Slack on a genuine bounce; and a new /health check reports whether the canary itself is still running, so "no alert fired" is no longer ambiguous between "everything is fine" and "the canary silently died." Phase 1's live read of the Mailtrap API reshaped the design twice before any of that shipped — the API turned out to be token-scoped rather than account-scoped, and a fresh send returns HTTP 404 for a second or two before resolving, which forced a seven-state result contract in place of the four states this milestone was originally specced with.

One half of the milestone's goal — a dead canary produces an alert — is fully proven: the alerting path was live-fire tested against staging, and a human visually confirmed the correct Slack post rather than trusting the job's return value. The other half — a real bounce produces an alert — is built, code-reviewed, and CI-tested, but has not yet executed against the live Mailtrap API end to end, because nobody in the delivery chain had staging or #alerts access to watch the first real 07:00 cycle. That gap is recorded honestly below rather than rounded up.

Goal ​

A transactional email that stops reaching real inboxes produces a Slack alert within 24 hours, and a canary that silently stops running produces one too.

Scope — What Shipped ​

  • Mailtrap::EmailLogs (rails_api/app/services/mailtrap/email_logs.rb) — a token-scoped client over Mailtrap's Email Logs API, resolving a message's live delivery status into a seven-state result (:delivered, :bounced, :pending, :not_found, :unknown, :error, :not_configured).
  • OpsMailer#delivery_canary and Email::DeliveryCanarySendJob — Crono, daily 07:00 in both staging and production, sends a synthetic email carrying a self-generated token and records the send to Rails.cache.
  • Email::DeliveryCanaryVerifyJob — Sidekiq, resolves the canary's real Mailtrap delivery status via find_message_id + status_for, writes a recent-success timestamp on confirmed delivery, and pages SlackNotifier on a bounce or an unresolved cannot-verify outcome.
  • bundle exec rake email_canary:live_fire — a repeatable rake task that forces the cannot-verify alert branch end to end, so the alerting path itself can be re-proven at any time, not just once at ship.
  • HealthController#check_email_canary — a new informational, fail-open /health check with three states: ok, stale, not_configured.
  • EMAIL_CANARY_RECIPIENT — a plain, non-secret env var wired across all six deploy manifests.
  • Docs: docs/architecture/email.md § Delivery Canary and docs/operations/alerting.md § Email Delivery Canary.

Phases ​

PhaseNameStatusPlansHighlights
1Mailtrap Email Logs API clientShipped1Read the live API first; found it token-scoped, not account-scoped, and collapsed the planned four-state contract into a safer seven states after observing a fresh send return HTTP 404 for ~0.5–2s.
2Canary send pathShipped1OpsMailer#delivery_canary + Email::DeliveryCanarySendJob (Crono, daily 07:00); EMAIL_CANARY_RECIPIENT wired non-secret across all six manifests.
3Delivery assertion + alertingShipped1Email::DeliveryCanaryVerifyJob + find_message_id; mandatory live-fire gate — a real failure was forced against staging and visually confirmed in #alerts.
4Green signal + docsShipped1Fail-open email_canary health check (ok/stale/not_configured); both architecture and alerting docs landed in-PR.

Key Decisions ​

  • Crono job, not GitHub Actions — deploy/crono.{production,staging}.yaml already inject MAILTRAP_API_TOKEN and SLACK_BUDGET_ALERT_WEBHOOK_URL, and Crono runs per-environment by construction (staging inherits production.rb, so it sends real mail and needed to be in scope). A GitHub Actions workflow would have needed new secrets and its own per-environment matrix to get the same coverage.
  • Slack (SlackNotifier), not Sentry, as the load-bearing alert channel — the sibling SENTRY_DSN investigation found objectuve-api had ingested zero Sentry events in 90 days. That call was retroactively justified mid-milestone: Phase 3's live-fire run found SENTRY_DSN_API had held an empty value since 2026-07-07, so Sentry's leg of the alert silently no-op'd during the very test meant to prove alerting worked.
  • A self-generated correlation token, not Mailtrap's message id — Mail::Message#deliver discards the SMTP response that would carry a sending_message_id, so the send job records its own token as the load-bearing key and treats the Mailtrap-side id as best-effort, resolved later by the verify job.
  • Seven result states, not four — the original four-state contract collapsed a clean 404, an unrecognized status, a non-2xx error, malformed JSON, and a transport timeout into one :not_found bucket. Two of those collapses were load-bearing: an in-flight send read identically to a vanished one, and "Mailtrap is unreachable" read identically to "the email never arrived." A canary that pages on every Mailtrap blip trains the team to ignore it — precisely the failure mode this milestone exists to prevent.
  • A mandatory live-fire gate for Phase 3, not a unit test alone — SLACK_BUDGET_ALERT_WEBHOOK_URL was already on record elsewhere as "wired but currently a no-op" for a related GitHub Actions use, so Phase 3 could not ship on the assumption that the same webhook worked for this purpose. The gate required forcing a real alert and reading the channel, not trusting the job's return value.
  • The health check is informational and fail-open — email_canary sits in HealthController::INFORMATIONAL_CHECKS alongside crono, so a stale canary never turns /health into an HTTP 503 or pages uptime monitors. The canary reports its own health; it does not gate the app's.

Requirements Coverage ​

3 / 4 acceptance criteria satisfied (see .planning/milestones/v4.43-email-delivery-canary-MILESTONE-AUDIT.md).

CriterionStatus
A bounced canary alerts within one cadence interval✅ Live-fire proven against staging
A successful canary is observable, distinguishing "no alert" from "canary died"✅ /health's email_canary check
The canary asserts delivered status from Mailtrap, not SMTP acceptance⚠️ Built and reviewed; the live resolution path has not yet executed end to end (see Tech Debt)
Documented under docs/architecture/email.md § Observability✅

Outcomes ​

Both staging and production now run a daily positive-delivery check with no manual step. A canary that stops running is visible in /health within one missed cadence window (default 26-hour staleness threshold) instead of silently disappearing. A genuine delivery failure — as opposed to a transient Mailtrap hiccup — is designed to distinguish itself from a resolution bug in the alert text itself, so whoever reads #alerts knows which one they're looking at without re-deriving it. The one thing not yet true in production is full end-to-end confidence in the happy path: that becomes true the first time someone with staging or #alerts access reads a 07:00 cycle and sees either silence or a CANNOT VERIFY message rather than nothing at all.

Tech Debt ​

  • (Phase 3/4) Route-1 confirmation is still open. Whether Mailtrap's ?message_id= filter resolves against the RFC Message-ID header has never been confirmed live — the live-fire test deliberately used a token with no cache record, so it short-circuited before reaching the Mailtrap lookup. If the assumption is wrong, email_canary:last_success:<env> is never written and /health reads stale indefinitely, with a daily cannot-verify alert naming the exact failure — contained, not silent, but not yet observed either way.
  • (Phase 2) EMAIL_CANARY_RECIPIENT: canary@objectuve.com — resolved 2026-08-29. The address was never a real mailbox: the first canary send (2026-08-27) genuinely reached Google's MX and hard-bounced (550-style, non-existent mailbox), and that one bounce put the address on Mailtrap's suppression list, which silently short-circuited the next two days' sends before any SMTP attempt — three not_delivered rows in the sending log, but one real bounce plus two local suppressions, not three independent failures (root-caused under OBJ-2967). Fixed by two required actions, both by Josh: canary@objectuve.com added as a Google Workspace alias on josh@objectuve.com (an alias, not a seat — nothing reads this mailbox via IMAP; DeliveryCanaryVerifyJob resolves delivery state via Mailtrap's API, not by reading mail), and the suppression-list entry removed via the Mailtrap UI. Production's own reporting path stays degraded until its next release, independent of this fix: production is running v4.7.0, which predates 1ca7e54af (OBJ-2951) and 8f9b58a98 (OBJ-2956), both already merged to master. See the suppression-list risk class in docs/architecture/email.md § Delivery Canary.
  • (Phase 3) Alert identity is borrowed. The canary posts through SLACK_BUDGET_ALERT_WEBHOOK_URL, so it appears in #alerts under the unrelated "AI Budget Webhooks" app. Functionally fine, semantically confusing — a dedicated webhook needs a new Slack app and a new secret across six manifests.
  • (Phase 3) Sentry's leg of the alert — resolved. SENTRY_DSN_API was found empty during the live-fire (OBJ-2821, unrelated pre-existing issue). OBJ-2821's blank-DSN fix shipped 2026-08-22, and OBJ-2971's live Sentry query since confirmed real production and staging events landing in objectuve/objectuve-api — the Sentry capture leg works, and OBJ-2971 additionally gave each alert kind/environment its own fingerprint so those events no longer fold into one issue.
  • (Phase 1) Spam-foldering is a known, accepted blind spot. Mailtrap's Email Logs API has no field for inbox-vs-spam placement, so a reputation regression that lands the canary in spam would still read :delivered. Closing this needs a seed-list/placement check — a different mechanism, explicitly out of scope for this milestone.
  • 14859e0a4 — Phase 1: Mailtrap Email Logs API client (PR #2578)
  • f43e9f0ad — Phase 2: Canary send path (PR #2591)
  • b03c0f893 — Phase 3: Delivery assertion + alerting, live-fire gate confirmed (PR #2592)
  • 966433030 — Phase 4: email_canary health check + docs (PR #2600)

Last updated: 2026-08-29 (Tech Debt: EMAIL_CANARY_RECIPIENT bullet closed out per OBJ-2972)

Loading…