ReasonRank

Blog

Creating a good agent: suites, baselines, and why the dropdowns are empty

ReasonRank Engineering · 2026-07-17

The most common “bug report” we see from new ReasonRank users isn’t a bug: the Production model and Candidate model dropdowns on an agent page only say “Not set” / “Select a candidate…” and offer nothing else.

That’s expected. Those lists are filled from Model Configs — labeled provider + model + parameter sets you register for the org — not from the agent create form. Until you add a production baseline and cheaper alternatives, there is nothing to select.

Here is how to create an agent that actually supports honest cost/quality decisions.

One workload, not “the chatbot”

An agent in ReasonRank is one production job you want to right-size: support triage, contract extraction, doc QA. If two teams share an agent, you can’t tell whose quality bar failed when a candidate looks worse.

Name it after the real workload. Write a short description and a default judge rubric so humans and scorers agree on pass/fail before you trust a savings number.

Suites before savings

You need shared test cases — preferably five or more for a gated recommendation. Prefer exact match, regex, or JSON checks when the output is constrained; use LLM-as-judge for free-form answers with a clear rubric.

Mine candidates from real traffic when you can, then approve them into the suite. Gold outputs beat vibe checks.

Register model configs next

Before you expect those dropdowns to work:

  1. Add provider keys under Settings (BYOK), or wire a harness.
  2. Open Model Configs and create today’s expensive baseline (label it clearly — e.g. gpt-4o @0 prod).
  3. Create two to four cheaper candidates (haiku @0 candidate, etc.).
  4. Return to the agent and set Production model.

Shadow tests and canaries need at least two configs: production and a different candidate. If you only create one config and set it as production, the candidate list will still look empty — again, by design.

Prefer temperature 0 when you’re gating a cost decision. Creative sampling adds noise that looks like quality regression.

Set volume so dollars are real

Production model alone is not enough for a finance-ready card. Volume comes from ingested traces or a monthly override. Without it, “save $Y/mo” is a story, not a measurement.

How the agent runs cases

| Mode | When | | --- | --- | | Provider (BYOK) | Prompt/model compare on your keys | | HTTP endpoint | ReasonRank POSTs cases to your HTTPS agent | | External | Your CI/VPC harness submits cell outputs |

Keep keys and runtime in your control when that matters; ReasonRank’s job is the comparison gate and the savings trail.

Common traps

  • Skipping Model Configs — empty dropdowns, no comparisons.
  • One config only — no candidate left after production is set.
  • Agent too broad — “all LLM traffic” can’t share one quality bar.
  • Zero cases — you can create the agent, but you can’t defend a switch.

What “good” looks like

An agent named after the workload, five-plus shared cases, production plus cheaper candidates registered, production model and volume set — ready for an offline run and a recommendation you can take to finance.

In the product: Creating a good agent and Model configs. Next up in the ladder: Testing agents 101.