Agent Runner Architecture

Technical architecture of the AI Workforce Agent Runner (agent_runner/). For operational guide, see operations/agent-runner-ops.md. For setup, see agent_runner/README.md.

Overview

Standalone TypeScript Express service that executes AI employee runs using the Claude Agent SDK. Receives run requests from the Rails API, executes agents with skills and MCP tools, applies brand-voice filtering, and posts results back via HMAC-authenticated webhook.

Stack: Node.js 20 + Express 5 + @anthropic-ai/claude-agent-sdk 0.2.104

Execution Lifecycle

Rails API (ExecuteRunJob)
  │
  ├─ POST /runs { run_id, employee config, inputs? }
  │   │  inputs assembled by AiWorkforce::BuildRunInputs (Rails, no Agent Runner DB access)
  │   └─ Returns 202 Accepted (fire-and-forget)
  │
  ▼
Agent Runner
  │
  1. Build system prompt (skills + memory)
  2. Build user/task prompt — appends `inputs` as "Provided data" section (if present)
  3. Configure MCP servers + REST tools
  4. Start cancellation poller (500ms interval)
  5. Call query() with budget cap + turn limit
  │   ├─ Stream tool calls and results
  │   ├─ Check for is_error flag (max_turns, max_budget)
  │   └─ Collect final result
  6. Run brand-voice post-filter (if configured)
  │   ├─ Pass → artifact status: pending
  │   └─ Fail → artifact status: auto_rejected
  7. POST webhook to Rails with results
  │
  ▼
Rails API (WebhooksController)
  └─ ProcessWebhookResult interaction
      ├─ Update AiRun status + token usage
      ├─ Create AiArtifact records
      └─ Upsert AiEmployeeMemory

Prompt Construction

File: src/agent/promptBuilder.ts

System prompt (buildSystemPrompt)

Assembled from three sections:

1. Preamble: "You are {name}, {role}."
2. Skills: For each skillRef → load SKILL.md, strip YAML frontmatter
   "---\n## Skill: {skillName}\n\n{content}"
3. Memory: If present → "---\n## Memory (from previous runs)\n\n{markdown}"

Skill loading (src/agent/skillLoader.ts): Reads SKILLS_DIR/{name}/SKILL.md, strips YAML frontmatter (/^---[\s\S]*?---\n/). Returns empty string if skill not found (warn, don't fail). SKILLS_DIR is evaluated at call time (not module load) so tests can override via env var.

Skill cap: 3 skills per employee maximum. System prompt should not exceed ~6,000 tokens.

User/task prompt (buildUserPrompt)

The base task prompt comes from employee.taskPrompt. When inputs are present (non-empty), the serialized payload is appended as a fenced "Provided data" section:

{taskPrompt}
---
## Provided data

The data below was supplied for this run. Base your work on it; do not fabricate entries beyond what is given.

\`\`\`json
{JSON.stringify(inputs, null, 2)}
\`\`\`

Employees that need no dynamic input receive an empty inputs hash from Rails; buildUserPrompt returns their task prompt unchanged (backward compatible).

Per-Run Inputs Channel

The Agent Runner has no database access, so data-driven employees receive their working data at dispatch time via the inputs field on POST /runs.

Rails side (AiWorkforce::BuildRunInputs): Called inside AgentRunnerClient#build_payload immediately before dispatch. Routes by employee.role_key:

role_key	Data assembled
coach_quality_auditor	{ coach_interactions: [...] } — up to 25 CoachInteraction records, stratified per-feature (one query ranks each feature's rows independently via ROW_NUMBER() OVER (PARTITION BY feature ORDER BY created_at DESC), then round-robin-interleaved up to COACH_SAMPLE_SIZE) across the 13 audited coach-voice features in CoachInteraction::FEATURES (field subset: feature, persona, tone_brevity, tone_warmth, content truncated to 500 chars, created_at)
(any other)	{} (empty — task prompt unchanged)

Constants (build_run_inputs.rb):

• COACH_SAMPLE_SIZE = 25 — sample cap per run. Stratified per-feature, not a flat recency window — a flat .limit(N) let a single high-volume batch feature (daily_insight_pack, weekly_digest) crowd rarer features out of the sample entirely once its own row count exceeded the window (OBJ-1660). See ai-workforce.md — Vera for the audited/exempt feature partition this sample draws from.
• MAX_CONTENT_CHARS = 500 — defensive per-message cap to keep payload within prompt budget

Agent Runner side (PostRunsBody in types.ts): inputs is RunInputs | null (Record<string, unknown>). The runner passes it to buildUserPrompt; an absent or empty hash is a no-op.

MCP Tool Pipeline

File: src/tools/mcpConfig.ts

Tools are scoped per-employee based on the mcpServers array in the employee config:

MCP Servers (stdio subprocess)

Server	Command	Auth	Tool Pattern
GitHub	npx -y @modelcontextprotocol/server-github	GITHUB_PAT env var	mcp__github__*

REST API Clients (inline tool definitions)

Client	File	Tools	Auth
PostHog	posthogClient.ts	Analytics queries	POSTHOG_API_KEY
Sentry	sentryClient.ts	Issue querying	SENTRY_AUTH_TOKEN
Mailtrap	mailtrapClient.ts	Email template mgmt	MAILTRAP_API_TOKEN
Slack	slackClient.ts	Webhook posting	SLACK_WEBHOOK_URL

REST tools return error strings (not exceptions) on missing env vars — graceful degradation allows the agent to report misconfiguration.

Tool Scoping

buildMcpConfig(employee) gates tools by checking employee.mcpServers array. An employee without 'github' in their config cannot invoke GitHub tools. allowedTools whitelist enforces the boundary at the SDK level.

Brand-Voice Post-Filter

File: src/agent/postFilter.ts

A separate query() call (NOT part of the main run context) that validates user-facing artifacts against the tightening-brand-voice skill.

Artifact content
  │
  ├─ Load filter skill → if not found → pass (conservative)
  │
  ├─ query() with:
  │   ├─ prompt: "Evaluate for brand-voice compliance. Return JSON."
  │   ├─ systemPrompt: filter skill content
  │   ├─ maxTurns: 1
  │   ├─ tools: [] (no tools)
  │   └─ persistSession: false
  │
  ├─ Parse JSON from response: { pass: bool, reason: string }
  │   ├─ Pass → artifact approval_status: 'pending'
  │   └─ Fail → artifact approval_status: 'auto_rejected'
  │               rejection_reason: 'Brand voice: {reason}'
  │
  └─ On parse/execution error → pass (conservative — avoid false rejections)

Design decision: The filter is a separate query() call, not a loaded skill in the main run. This prevents context dilution — the main agent focuses on its task, and the filter evaluates the output independently.

Webhook Contract

File: src/webhook/sender.ts

Payload Shape

{
  run_id: string
  status: 'succeeded' | 'failed'
  token_usage: {
    prompt_tokens?: number
    completion_tokens?: number
    cost_cents?: number
  }
  artifacts?: [{
    kind: string              // e.g., 'draft_post', 'email_template'
    title: string             // "{employee.name} — {date}"
    payload: { content: string }
    approval_status?: string  // 'pending' | 'auto_rejected'
    rejection_reason?: string // set when auto_rejected by brand-voice filter
  }]
  error_message?: string      // set when status: 'failed'
  run_log?: {
    num_turns: number
    tool_calls?: [{ tool, timestamp, input?, output? }]
  }
}

Authentication

Mode	Header	Mechanism
secret (dev)	X-Workforce-Signature	HMAC-SHA256 of JSON body using RAILS_WEBHOOK_SECRET
oidc (prod)	Authorization: Bearer <token>	GCP metadata server identity token (55-min cache)

Retry Logic

• Max attempts: 3
• Backoff: Linear (1s, 2s, 3s)
• Timeout: 15 seconds per request
• On final failure: Throws error (run marked as failed by TimeoutStaleRunsJob)

Authentication Model

The Agent Runner authenticates to two downstream services using two independent environment variables:

Hop	Direction	Env var	Values	Dev default
Rails webhook	Agent Runner → Rails	AUTH_MODE	secret (HMAC) · oidc (OIDC bearer)	secret
LiteLLM proxy	Agent Runner → LiteLLM	LITELLM_AUTH_MODE	none (no extra auth) · oidc (OIDC bearer)	unset / none

AUTH_MODE and LITELLM_AUTH_MODE are fully independent. In production both are set to oidc. In local development AUTH_MODE=secret and LITELLM_AUTH_MODE is unset (equivalent to none).

Agent Runner
  │
  ├──[AUTH_MODE=secret → X-Workforce-Signature HMAC]──▶ Rails API
  │  [AUTH_MODE=oidc   → Authorization: Bearer <token>]
  │
  └──[LITELLM_AUTH_MODE=none → ANTHROPIC_API_KEY placeholder only]──▶ LiteLLM Cloud Run
     [LITELLM_AUTH_MODE=oidc → ANTHROPIC_AUTH_TOKEN = OIDC bearer → Authorization: Bearer eya…]

When LITELLM_AUTH_MODE=oidc, runner.ts calls fetchIdentityToken(anthropicBaseUrl) and injects the result as ANTHROPIC_AUTH_TOKEN in the env passed to query(). The Claude Agent SDK uses ANTHROPIC_AUTH_TOKEN as the Authorization: Bearer header on every request to the LiteLLM endpoint.

OIDC Token Fetch

File: src/auth/oidcToken.ts

fetchIdentityToken(audience) is called in two places — the Rails webhook sender (AUTH_MODE=oidc) and the LiteLLM auth path (LITELLM_AUTH_MODE=oidc). The same cache is used for both.

fetchIdentityToken(audience)
  │
  ├─ Check token cache → if valid (< 55 min old) → return cached
  │
  ├─ GET http://metadata.google.internal/computeMetadata/v1/instance/
  │     service-accounts/default/identity?audience={url}
  │   Header: Metadata-Flavor: Google
  │   Timeout: 5 seconds
  │
  ├─ Cache token with 55-min TTL (GCP tokens valid ~1 hour)
  │
  └─ Return JWT string

Only works on GCP Cloud Run (metadata server not available locally). Development uses AUTH_MODE=secret with HMAC and leaves LITELLM_AUTH_MODE unset.

State Management

File: src/state/runMap.ts

In-memory Map<string, RunState> tracking active runs:

interface RunState {
  status: 'running' | 'cancelled'
  cancelRequested: boolean
}

Lifecycle: Created on POST /runs, deleted after webhook delivery. Lost on restart — Rails AiRun table is the source of truth. TimeoutStaleRunsJob (Crono, every 15 min) marks runs stuck >15 min as failed.

Critical Implementation Notes

1. ANTHROPIC_BASE_URL must be set in BOTH process.env AND options.env passed to query() — the SDK requires both locations for LiteLLM routing
2. LITELLM_AUTH_MODE=oidc injects ANTHROPIC_AUTH_TOKEN into options.env so the SDK sends an OIDC bearer on every LiteLLM request — independent of AUTH_MODE (Rails hop). Both default to non-OIDC in dev; both must be oidc in production.
3. error_max_turns is NOT a thrown error — it's a result message with is_error=true flag
4. persistSession: false is required — Cloud Run is stateless, no session persistence
5. Do NOT call .return() on the stream generator after receiving the result message
6. Brand-voice filter returns pass: true on any error — conservative to avoid blocking legitimate content

---

Last updated: 2026-07-22