Cost–quality decisions for agents · private beta
Cut your LLM bill without
cutting quality — verified, not estimated.
For engineering teams who own the model bill. ReasonRank sits beside your stack — your provider keys, your harness, your guardrails — and owns the decision: cheapest model that holds quality, CI-gated, then verified dollars on production traffic. And it stays on watch between decisions: live quality drift, judge health, dataset freshness.
BYOK · BYO harness · no token markup · metadata-only ingest by default · judges calibrated to your labels
The problem
Teams ship agents fast, then overpay to run them forever.
Model sprawl
Every agent and skill was pinned to whatever model felt right that week. Nobody re-checks whether a cheaper one now does the job just as well.
Silent overspend
Token bills climb as traffic grows, but you can't see which agent is expensive, or how much a switch would actually save at your real volume.
Quality you can’t vouch for
Swapping models is scary because you have no repeatable, scored proof that quality holds. So you overpay for headroom you may not need.
Candidates for this skill
quality vs. cost per call| Model | Quality | Cost / call | Value |
|---|---|---|---|
| GPT-5.6 Lunarecommended | 0.91 | $0.0045/call | |
| Claude Haiku 4.5 | 0.89 | $0.0040/call | |
| Gemini 3.6 Flash | 0.90 | $0.0060/call | |
| GPT-5.6 Sol (in prod) | 0.94 | $0.0210/call |
Verified savings
$2,890/mo
support-triage-agent · GPT-5.6 Sol → GPT-5.6 Luna, live in production
95% CI on quality delta
−0.6% … +2.3% · clears −2% tolerance
paired over 24 test cases · production baseline: 1,204 pre-switch calls · switch detected in live traffic · prices synced daily
Illustrative figures. Your models, cases, and traffic produce your own numbers — that’s the point.
The closed loop: connect →
recommend → gate → verify.
How it works
Connect traffic or your harness
Stream metadata traces (OTLP / ingest) so volume and spend are real — or keep agents in your VPC and submit cell outputs from your own runner. Optional self-hosted gateway if you want zero app-code change.
Evaluate candidates
Score outputs and whole agent trajectories across models — deterministic checks plus judges that first have to agree with your team’s own labels. Pre-flight cost estimate on every run, on your keys.
Get a recommendation
ReasonRank finds the cheapest model that holds quality (cost per success, with a statistical gate) and projects dollar savings against your actual traffic.
Gate, apply & verify
Fail a PR when quality slips (`reasonrank assert`) — with the full diff posted as a PR comment — apply the switch in your stack or ours, then verify dollars on live traffic with rollback if it regresses.
Your keys · your harness
We don't replace your agent runtime.
We sit next to it.
Traces show spend. ReasonRank decides the model and proves the savings — including when your harness runs the agent. Bring your own provider keys; keep LangGraph, CI jobs, and VPC runners.
Provider (BYOK)
ReasonRank calls models with your encrypted keys — ideal for prompt/model comparison suites without standing up a runner.
HTTP endpoint
We POST each case to your HTTPS agent with signed requests. Your runtime, tools, and guardrails stay in place.
External cells
Your CI or VPC harness submits outputs via API and flows through the same score → recommend → verify loop. Your runner stays yours; the decision math stays consistent.
Works with your stack
Vercel AI SDK · OpenAI Agents SDK · LangGraph · Pydantic-AI · Claude Code · anything OpenTelemetry
One-call adapters for the frameworks above — or point any OTel exporter at us and skip the adapter entirely. Five minutes to first trace; your framework, runtime, and tools stay exactly where they are.
Integration contract for design partners: BYO keys & harness.
The decision unit
Cost per successful task —
then verified $/mo in production.
Other tools tell you which model is smartest. ReasonRank tells you which is smart enough for the job at the lowest defensible cost, gates the change in CI, and proves the dollars after you ship — not in a spreadsheet.
Supporting view: efficiency ∝ task quality / (tokens · cost · latency) — useful for ranking, never a substitute for verified savings.
Capabilities
Everything you need to right-size an agent — without ripping out your stack.
Verified savings recommendations
01The lever, not just the chart: move this agent to a cheaper model — quality holds within tolerance, save $X/mo at your volume. Projected first; verified after the switch is live.
Bring your own keys
02Evals and judges run on your provider accounts. Keys are encrypted at rest, never resold or marked up. Your OpenAI/Anthropic/Google bill stays yours.
Bring your own harness
03HTTPS invoke or external cell submit — keep LangGraph, internal runners, and CI jobs. We score and decide; we don’t force you onto our prompt runner.
CI quality gate with PR receipts
04Same non-inferiority math as Savings. `reasonrank assert` fails the build when a candidate regresses — and posts the full diff (score deltas, confidence bounds, cost impact) as a PR comment. Pin a frozen baseline run so “quality held” means held against the same bar.
Production volume, not synthetic guesswork
05Recommendations reflect your real monthly volume and spend from ingested traces (metadata-only by default). Payloads are opt-in.
Spend guardrails
06Pre-flight estimates, per-run and monthly budget caps, an output-token ceiling, live running cost, and a kill switch. Measuring waste never becomes it.
Scoring your staff eng can audit
07A library of deterministic scorers — structure, numbers, groundedness, retrieval, semantics — plus trajectory scoring for agents: right tools, no wasted steps, task actually done. Every score ships with the evidence behind it.
Judges that earn trust
08LLM judges aren’t trusted by default. They’re measured against your team’s own labels before their scores count, re-checked on a schedule, and flagged the moment they stop agreeing with humans. Scorers that turn noisy get benched automatically — visibly, never silently.
Versioned golden datasets
09Every run pins an immutable, versioned snapshot of your eval set — with lineage. “Which cases did that number come from” is always answerable, and dataset diffs separate suite drift from model drift.
Replay real episodes
10Promote recorded production episodes into replayable fixtures and re-run candidates against them — same inputs, same tool results, no live side effects. Compare agent behavior on what actually happened, on your laptop or in CI, with the same statistics either way.
Why our numbers hold up
Statistics a staff engineer can audit.
Every recommendation ships with its method, interval, and sample — and every claim below is visible on the evidence page of a real recommendation.
Paired cluster bootstrap
Two models scored on the same test case are paired observations, and repetitions within a case are correlated. We bootstrap over test cases — clusters — never over individual results, so correlated repeats can't launder themselves into fake sample size.
Non-inferiority, not vibes
A cheaper model is recommended only when the 95% confidence interval on the quality delta clears a −2% tolerance. Too few shared cases? The recommendation is flagged unproven and excluded from every headline dollar — we tell you exactly how many cases to add.
Production baselines
Realized savings compare production cost-per-call before the switch to production cost-per-call after it — never eval-suite numbers — and are withheld until the new model is actually observed in your traffic with enough calls to judge.
Built to be trusted
Ready for the way serious teams operate.
Security, isolation, and spend governance aren't features we added later — they're what the product is built on. ReasonRank was designed from day one for teams that answer to a security review.
Enterprise-ready: single sign-on, encryption at rest, strict workspace isolation, full audit trails, and the compliance paperwork your security team will ask for — available on request. Overview on /trust.
Bring your own keys
Evaluations run against your own provider accounts. We never resell tokens or mark them up — your provider bill stays yours.
Bring your own harness
Call your agent over HTTPS or push results from CI/VPC. Your runtime and guardrails stay yours; we own the cost–quality decision.
Spend governance
Org-level budgets, per-run caps, and token ceilings turn “hope it’s fine” into enforced limits — with alerts at 50/80/100%.
Encrypted & isolated
Provider keys are encrypted at rest, and everything you create is isolated to your workspace — access is scoped on every query, not policed after the fact.
Pricing
Packaging coming soon.
Plans will scale with the spend we manage and stay aligned with the savings we deliver. Bring your own keys and harness — never a token markup.
Stop guessing what your agents
should cost.
We're onboarding a small group of design partners. Bring your keys, your harness, your agents, and your rubrics — we'll help you ship a cheaper model you can defend.