BYO keys & BYO harness

This is the integration contract for design partners — not optional enterprise fluff. ReasonRank is built so your LLM spend and your agent runtime stay yours. If you already have (or will build) an eval harness, CI job, or VPC agent, start here. ReasonRank scores and decides; it does not require you to rip out your runner.

Four key types (do not conflate)

Key	Where	Purpose
Provider keys (BYOK)	Settings → Providers	Offline evals, judges, online scoring — AES-256-GCM at rest
Ingestion keys (`rr_ingest_…`)	Settings → Ingestion	`POST /api/ingest` + OTLP
PATs (`rr_pat_…`)	Settings → API Keys	Public API, CLI, gateway recording, external cell submit
Platform env	Vercel	Stripe, Resend, encryption master key — never customer LLM keys

Connection defaults (Azure endpoint / api-version, Bedrock region, openai_compatible base URL) live on providerCredentials.config. Per-model overrides on Model Configs still win.

Agent execution modes

Set on the agent (Settings card on /tasks/[id] → Execution):

Mode	Who runs the agent	When to use
`provider` (default)	ReasonRank calls the model via BYOK	Prompt / model comparison suites
`http_endpoint`	ReasonRank POSTs each case to your HTTPS URL	Your agent is reachable from the internet
`external`	Your worker submits cell outputs via API	VPC / firewalled / CI-local harness

All three feed the same score → rank → recommend → verify loop.

Customer harness / agent
        │
        ├─ http_endpoint ◄── signed POST from ReasonRank
        │
        └─ external ──────── POST /api/v1/runs/:id/cells (PAT)
                │
                ▼
         scoreOutput → metrics → NI gate → recommendations

Mode A — HTTP endpoint (push)

Agent → Execution mode → HTTP endpoint
Set HTTPS URL (SSRF-blocked: no localhost / private / metadata IPs)
Optional hostname allowlist
Generate signing secret (shown once) — store in your agent
Implement POST handler:

Request (JSON):

{
  "runId": "…",
  "caseId": "…",
  "attempt": 1,
  "modelConfigId": "…",
  "modelName": "gpt-4o-mini",
  "provider": "openai",
  "input": "last user turn",
  "turns": [{ "role": "user", "content": "…" }],
  "systemPrompt": "…"
}

Headers:

X-ReasonRank-Timestamp — unix ms
X-ReasonRank-Signature — HMAC-SHA256(secret, "${timestamp}.${rawBody}") hex

Response (JSON):

{
  "output": "agent final answer",
  "tokensInput": 120,
  "tokensOutput": 40,
  "costUsd": 0.002,
  "latencyMs": 800
}

output is required; tokens/cost/latency optional but needed for honest $/success.

Verify signature with ±5 minutes skew. Reject unsigned requests.

Mode B — External cells (pull / BYO harness)

Agent → Execution mode → External
Start a run (UI, POST /api/v1/runs, or reasonrank run)
Run status becomes awaiting_external; every grid cell is parked
Your harness POSTs results:

curl -X POST "$APP/api/v1/runs/$RUN_ID/cells" \
  -H "Authorization: Bearer rr_pat_…" \
  -H "Content-Type: application/json" \
  -d '{
    "cells": [{
      "modelConfigId": "…",
      "testCaseId": "…",
      "attempt": 1,
      "output": "…",
      "tokensInput": 100,
      "tokensOutput": 50,
      "costUsd": 0.001,
      "latencyMs": 400
    }]
  }'

Idempotent on (modelConfigId, testCaseId, attempt). When every cell is terminal, the run completes and recommendations can regenerate.

This is the “bring your own harness” path — Braintrust-style local runners, CI jobs, internal eval frameworks all submit here.

Shortcut: reasonrank run --local is a first-party harness over this same API — it starts a run with a per-run executionMode: "external" override (no agent reconfiguration needed), executes the grid on your machine with your own env provider keys, and submits cells here. The server does all scoring, so local, CI, and cloud runs share one stats → non-inferiority path. Recorded tool-call episodes replay via reasonrank replay (design-partner preview).

Skills as versioned artifacts

Skills → Register skill (name should match OTel toolName / skill span)
Attach an agent suite (eval cases)
Register versions (semver + optional checksum)
Detail page shows 30d span rollups (invocations, error rate, spend)
Run the attached suite with any execution mode above

Skills do not yet have a separate apply/recommend path — production model switches stay on the agent. The skill object makes marketing claims truthful: versioned artifact + suite + production telemetry.

CI gating (same math as Savings)

export REASONRANK_API_KEY=rr_pat_…
pnpm --filter @reasonrank/cli exec tsx src/cli.ts run \
  --agent "$AGENT_ID" --models "$BASELINE,$CANDIDATE" --wait

pnpm --filter @reasonrank/cli exec tsx src/cli.ts assert \
  --run "$RUN_ID" --baseline "$BASELINE"

Fails if there are fewer than 5 shared cases or the paired-cluster 95% CI lower bound on quality delta is below -0.02. See /docs/ci-gating and the .github/actions/reasonrank-gate composite action.

Security guarantees

BYOK ciphertext is AES-256-GCM, key-versioned (v1.…); rotate via CREDENTIALS_ENCRYPTION_KEY_V1
Endpoint invoke: HTTPS-only in prod, SSRF IP blocklist, optional host allowlist, HMAC
External cells: PAT scope runs:write, org-scoped; cross-tenant IDs 404
Gateway is self-hosted single-tenant only — never put customer provider keys through a shared ReasonRank host
Ingest defaults to metadata-only; payloads are opt-in + retention-swept

Minimal “first dollar of value” path

BYOK → Settings → Providers
Create agent + 5+ test cases
Ingest real traffic (SDK / OTel) or set monthly volume override
Run evaluation across 2 models → apply recommendation
Confirm verified savings after switch detection (≥20 post-switch traces)