Objectuve Docs

Appearance

AI Workforce — Business Process Runbooks ​

Human-facing process maps for operating the AI Workforce. The feature doc explains the architecture; this doc explains who does what, when, and how to recover when things go sideways.

Every node in every diagram traces to real code — see Cross-references at the bottom.

Process Inventory ​

#ProcessTriggerPrimary ActorCadenceInputsOutputsSuccess SignalEscalation
1Run LifecycleCron tick or admin "Trigger run"System (Rails + Agent Runner)Per-employee schedule (daily / weekly / biweekly)AiEmployee config, memory, skillsAiArtifact(pending) on the Ops BoardAiRun.status = succeeded, artifact appears on the board within 2 min of run completionRun stuck in running > 15 min → inspect Run Detail; Agent Runner unreachable → see Process 5
2Ops Board Review DecisionNew AiArtifact(pending) appearsOps operator (admin)Continuous; target ≤ 4 hr response SLA during business hoursArtifact payload, preview, delivery metadataApprove / Approve-with-edits / Reject + reasonArtifact approval_status transitions to approved or rejected; DeliverArtifactJob enqueued on approveReject count > 2 for same employee in 1 week → pause employee, review memory/skill config
3Delivery RoutingApproveArtifact success eventSystem (DeliverArtifactJob)Immediate (async) on approvalApproved artifact, artifact.kind, destination credsGitHub PR / Mailtrap draft / Slack message / docs commitdelivered_at timestamp set; external ID recorded (PR URL, draft ID, etc.)delivery_status = failed → check agent-runner-ops credentials, retry manually from Run Detail
4Activation Funnel WoWCron tick (daily at midnight UTC)System (Rails endpoint)Daily · 0 0 * * *7-day rolling-window user cohortsMetrics: signed_in, wizard_completed, first_goal, first_check_in, first_badge + WoW % changeResponse 200 + log entry shows current/previous counts and WoW %skip_run=true (zero cohort) → log and continue; 401/422 → log config error, investigate secret/server; 5xx → retry 3x with backoff
5Autonomy PromotionOperator clicks "Promote" in Employee DetailTeam lead (Josh / delegated)Quarterly review; ad-hoc when criteria metApproval-rate chart, weeks-active, recent-rejections, role typeAutonomy level change (shadow → semi → autonomous)DB validation promotion_requirements_met passes; level committed; employee card badge updatesBlocked by validation → fix criteria gap or acknowledge role lock (Ally is shadow-only)
6Failure & Budget CircuitAny error: agent runner down, HMAC mismatch, error_max_turns, budget exceededSystem → operatorOn exceptionError payload, run_log, tokens_used, month_spendRun transitions to failed with error_message; Slack budget alert fires for monthly capCircuit-breaker retries exhaust or budget alert fires → operator inspects, pauses or bumps capHMAC signature mismatch persisting → rotate RAILS_WEBHOOK_SECRET, coordinated redeploy (both services)
7Operator CadenceTime (daily / weekly / monthly / quarterly)Ops operator + team leadRecurringAdmin dashboard, Slack alert channelQueue drained, health recorded, promotions consideredZero pending artifacts > 24h; no paused employees without reason; budget utilization < 90%Cadence misses 2 cycles → raise in weekly lead review

Process 1 — Run Lifecycle (End-to-End) ​

How one execution flows from a scheduler tick to a delivered artifact.

flowchart LR
    subgraph Trigger
      A[Crono: every 5 min] --> B[ScheduleRunsJob]
      A2[Admin: Trigger Run button] --> C
      B --> C{TriggerRun<br/>budget gate}
    end

    C -->|month_spend ≥ cap| X1[Fail: monthly_budget_exceeded]
    C -->|concurrent run exists| X2[Fail: already_running]
    C -->|ok| D[AiRun queued]
    D --> E[ExecuteRunJob]
    E --> F[AgentRunnerClient<br/>POST /runs]

    subgraph AgentRunner[Agent Runner service]
      F --> G[Load skills + memory + MCP]
      G --> H[query loop<br/>maxTurns, maxBudgetUsd]
      H --> I{post_filter_skill<br/>configured?}
      I -->|yes| J[Brand-voice query<br/>auto_reject on fail]
      I -->|no| K[Skip post-filter]
      J --> L[webhook sender<br/>HMAC signed]
      K --> L
    end

    L --> M[Rails: /webhooks/ai-workforce]
    M --> N{HMAC valid?}
    N -->|no| X3[401 + log]
    N -->|yes| O[ProcessWebhookResult]
    O --> P[Upsert AiEmployeeMemory]
    O --> Q[Create AiArtifact pending]
    Q --> R[(Ops Board<br/>Needs review column)]
    R -.-> S[Process 2: Review]

    style X1 fill:#fee,stroke:#c33
    style X2 fill:#fee,stroke:#c33
    style X3 fill:#fee,stroke:#c33
    style R fill:#eef,stroke:#55a

Operator notes

  • Actor: None — fully automated until artifact lands on the Ops Board.
  • Trigger: Crono tick (scheduled) or admin clicks "Trigger run" on an employee card.
  • Latency: Typically 30s–3min end-to-end. Runs > 15 min in running state are stuck (Agent Runner lost the fire-and-forget call or webhook failed).
  • Success signal: AiArtifact appears on the Ops Board's Needs review column with approval_status = pending (or auto_rejected if post-filter blocked it).
  • Where to look: Admin → Run Detail for tool calls, cost, duration; Run Log JSON has the full SDK trace.
  • Escalation: Stuck run → check agent-runner-ops; repeated budget fails → Process 5.

Process 2 — Ops Board Review Decision Tree ​

What the operator does when they open a pending artifact's Focus Detail overlay.

flowchart TD
    A[Operator opens ticket<br/>Focus Detail overlay] --> B[Artifact card expands]
    B --> C{Artifact kind?}
    C -->|code_patch| D[Render diff2html<br/>read-only]
    C -->|email_template / slack_message / report / other| E[Editable textarea]

    D --> F{Quality check}
    E --> F

    F -->|Good as-is| G[Click Approve]
    F -->|Small edits needed| H[Edit payload inline]
    F -->|Off-brand or incorrect| I[Click Reject]

    H --> G2[Click Approve]
    G --> J[ApproveArtifact<br/>record reviewer + timestamp]
    G2 --> J2[ApproveArtifact<br/>record edit_distance Levenshtein]

    I --> K[Enter reason ≥ 10 chars]
    K --> L[RejectArtifact]

    J --> M[Publish event<br/>ai_workforce.artifact_approved]
    J2 --> M
    M --> N[Enqueue DeliverArtifactJob]
    N --> O[Process 3: Delivery]

    L --> P[Set approval_status rejected]
    P --> Q[Append reason to<br/>employee memory<br/>recent_rejections FIFO cap 5]
    Q --> R[Surfaces in next run<br/>via memory injection]

    style J fill:#efe,stroke:#393
    style J2 fill:#efe,stroke:#393
    style L fill:#fee,stroke:#c33
    style R fill:#eef,stroke:#55a

Operator notes

  • Actor: Ops operator (admin role).
  • Cadence: Continuous; target first-response within 4 hours during business hours. Queue polls every 30s.
  • Reject rule: Reason is required (min 10 chars) and becomes training data — write it as if teaching the employee ("too long, cut to 3 sentences" beats "bad").
  • Edit distance signal: Heavy edits on approval (Levenshtein ratio > 0.3) = artifact is borderline; watch for a pattern across runs.
  • Escalation: 2+ rejects in one week for the same employee → pause it (Process 6 weekly review) and inspect skill config / memory.

Process 3 — Delivery Routing ​

How an approved artifact reaches its destination.

flowchart LR
    A[Approved Artifact] --> B[DeliverArtifactJob]
    B --> C{delivered_at<br/>already set?}
    C -->|yes| Z[No-op idempotent]
    C -->|no| D{artifact.kind?}

    D -->|draft_post / code_patch| E[deliver_github_pr]
    D -->|email_template| F[deliver_mailtrap_sandbox]
    D -->|report| G[deliver_report_slack_and_docs]
    D -->|recommendation| H[deliver_recommendation_to_docs]

    E --> E1[Create branch via Octokit]
    E1 --> E2[Commit file]
    E2 --> E3[Open PR]
    E3 --> E4[Record PR URL in<br/>delivery_metadata]

    F --> F1[POST draft to<br/>Mailtrap sandbox inbox]
    F1 --> F2[Record draft_id<br/>never sends to recipients]

    G --> G1[Validate host = hooks.slack.com]
    G1 --> G2[POST to Slack webhook]
    G2 --> G3[Octokit commit to<br/>docs/reports/ path]

    H --> H1[Octokit commit to<br/>docs/recommendations/]

    E4 --> Y[Set delivered_at<br/>delivery_status = delivered]
    F2 --> Y
    G3 --> Y
    H1 --> Y

    Y --> Q[Visible in Run Detail<br/>expandable delivery panel]

    style Z fill:#eee,stroke:#888
    style Y fill:#efe,stroke:#393

Operator notes

  • Actor: System; no human intervention on the happy path.
  • Idempotency: delivered_at is the guard — re-enqueuing is safe.
  • Mailtrap is always sandbox. Lifecycle emails never send to real users from an AI run. Promotion to production mail sending would be a separate feature gate.
  • Slack host check is a security guard — the job refuses any webhook URL not under hooks.slack.com.
  • Failure recovery: delivery_status = failed surfaces in Run Detail. Fix the underlying cause (missing token, repo permission, revoked webhook) and retry manually.

Process 4 — Autonomy Promotion Ladder ​

When and how an employee moves up the trust ladder.

stateDiagram-v2
    [*] --> shadow: All new employees

    shadow --> semi_autonomous: approval_rate > 80%<br/>AND weeks_active ≥ 4<br/>AND zero rejections in last 2 weeks
    semi_autonomous --> autonomous: approval_rate > 95%<br/>AND weeks_active ≥ 8<br/>AND operator confirms modal<br/>(promotion_confirmed_at set)

    autonomous --> semi_autonomous: Demote button
    semi_autonomous --> shadow: Demote button

    state shadow {
        [*] --> All_artifacts_queue: Every artifact → Ops Board
    }
    state semi_autonomous {
        [*] --> Low_risk_auto: Low-risk kinds auto-approve<br/>High-risk → queue
    }
    state autonomous {
        [*] --> Direct_delivery: No review gate
    }

    note right of shadow
        customer_support_drafter (Ally)
        is permanently locked here
        by model validation
    end note

Operator notes

  • Actor: Team lead (Josh today; delegated later). Not an ops-operator action.
  • Cadence: Quarterly review cycle + ad-hoc when a thriving employee hits the metrics.
  • Gate enforcement: It's a database validation (promotion_requirements_met), not just a UI warning. A raw update from a rails console will still fail if criteria aren't met.
  • Autonomous tier requires a separate confirmation modal — this is intentional friction. Write a one-line justification in the commit when promoting.
  • Demote freely. Demotion has no criteria check and costs nothing — if behavior regresses, step the employee down immediately.
  • Ally (customer_support_drafter) is locked to shadow permanently by the model. Customer-facing copy is too high-risk for auto-approval.

Process 5 — Failure & Budget Circuit ​

What the system does — and what the operator does — when things break.

flowchart TD
    subgraph Failures
      A1[Agent Runner<br/>unreachable]
      A2[error_max_turns /<br/>SDK error]
      A3[HMAC signature<br/>mismatch]
      A4[Monthly budget<br/>exceeded at trigger]
    end

    A1 --> B1[ExecuteRunJob<br/>circuit breaker]
    B1 --> B2{Retries < 3?}
    B2 -->|yes| B3[Exponential backoff]
    B3 --> B1
    B2 -->|no| B4[AiRun failed<br/>error = unreachable]

    A2 --> C1[Agent Runner<br/>webhook fires]
    C1 --> C2[run.status = failed<br/>run_log has trace]

    A3 --> D1[Webhook controller<br/>returns 401]
    D1 --> D2[Rails log: signature_invalid]
    D2 --> D3[Run stays in running]
    D3 --> D4[Operator inspects<br/>after 15 min SLA]

    A4 --> E1[TriggerRun fails<br/>code: monthly_budget_exceeded]
    E1 --> E2[Slack budget alert<br/>via SLACK_WEBHOOK_URL]
    E2 --> E3{Operator action}
    E3 -->|bump cap| E4[Edit monthly_budget_cents<br/>in Employee Detail]
    E3 -->|wait for month reset| E5[Pause employee<br/>until month rollover]
    E3 -->|investigate overspend| E6[Run Detail → input/output<br/>token breakdown]

    B4 --> F[Operator inspects Run Detail]
    C2 --> F
    F --> G{Recoverable?}
    G -->|yes| H[Re-trigger manually]
    G -->|no| I[Pause employee<br/>open skill/config ticket]

    style A1 fill:#fee,stroke:#c33
    style A2 fill:#fee,stroke:#c33
    style A3 fill:#fee,stroke:#c33
    style A4 fill:#fee,stroke:#c33
    style E2 fill:#ffd,stroke:#c80

Operator notes

  • Agent Runner unreachable: Usually means Cloud Run cold start delay > client timeout, or the service is down. Check /health on AGENT_RUNNER_URL. The 3-retry circuit breaker with backoff gives it ~1 min of grace before failing the run.
  • HMAC mismatch: Almost always a secret-rotation gap between services. Verify RAILS_WEBHOOK_SECRET matches on both sides. Rotation = coordinated redeploy — both services in the same window, not sequential.
  • Budget exceeded: The TriggerRun interaction enforces the gate before dispatching. A rapidly-overspending employee almost certainly has skill prompts producing runaway tool calls — inspect the Run Detail tool-call timeline.
  • Ally special case: A failed run on the Ally employee is extra-sensitive; any failure pattern there warrants a same-day investigation because the role already can't auto-approve.
  • Never bypass HMAC verification to "unblock" a run — it exists to prevent spoofed webhooks. Fix the secret instead.

Process 6 — Operator Cadence (Daily / Weekly / Monthly / Quarterly) ​

The recurring human rituals that keep the workforce healthy.

flowchart LR
    subgraph Daily[Daily Ops ∼ 10 min]
      D1[Ops Board sweep<br/>drain pending artifacts]
      D2[Run Detail spot-check<br/>on any failed runs]
      D3[Confirm no stuck runs<br/>more than 15 min]
    end

    subgraph Weekly[Weekly Lead ∼ 20 min]
      W1[Approval-rate chart<br/>per employee]
      W2[Memory hygiene<br/>review recent_rejections]
      W3[Pause employees<br/>with 2+ rejects in week]
      W4[Review Slack budget<br/>alerts history]
    end

    subgraph Monthly[Monthly Lead ∼ 30 min]
      M1[Budget reconciliation<br/>actual vs cap per employee]
      M2[Cost-per-approved-artifact<br/>ratio trend]
      M3[Paused / stuck<br/>employees audit]
      M4[Skill performance review<br/>approval-rate by skill]
    end

    subgraph Quarterly[Quarterly Lead ∼ 60 min]
      Q1[Autonomy promotion<br/>review each employee]
      Q2[Consider new employees<br/>from skill catalog]
      Q3[Retire underperforming<br/>employees]
      Q4[Brand-voice post-filter<br/>sampling audit]
    end

    D1 --> D2 --> D3
    W1 --> W2 --> W3 --> W4
    M1 --> M2 --> M3 --> M4
    Q1 --> Q2 --> Q3 --> Q4

    Daily -.feeds.-> Weekly
    Weekly -.feeds.-> Monthly
    Monthly -.feeds.-> Quarterly

Operator notes

  • Daily is non-negotiable — the value prop is freshness. Artifacts going stale in the queue defeats the point.
  • Weekly is the leverage point — most quality issues surface here before they become monthly budget problems.
  • Monthly is financial hygiene — track trending, not just current month. A 15% MoM growth is fine; 40% warrants inspection.
  • Quarterly is the strategic review — promotions, retirements, new employees. Pair with the roadmap review.

Cross-references ​

Every diagram node maps to concrete code. Use these when updating or validating the runbooks.

Scheduling & dispatch ​

Agent execution ​

Rails result processing ​

Review + delivery ​

Autonomy ​

Operator surfaces ​

Last updated: 2026-07-20

Loading…