Release Benchmarks · The gate

Know exactly what broke, before you ship it.

A private suite of frozen cases, re-graded by hand every release. Regressions surface before they reach users, not after, as an incident. So a prompt tweak the night before a customer demo can’t silently break the flow that closes the deal.

Book a call, grade your next release
30-min call, no prep · scope in 48h · first $200 free
Release candidateyour-agentsample
v2.4 vs v2.3, at the gate
✗ held from release
the frozen suite, re-graded by hand, release over release
fetch_record()
v2.1v2.2v2.3v2.4
Held on every release, all the way since v2.1.
policy answers
v2.1v2.2v2.3v2.4
Failed twice, fixed in v2.3, holding ever since.
guardrails
v2.1v2.2v2.3v2.4
Red for three releases, finally green in v2.4.
execute_action()
v2.1v2.2v2.3v2.4
Fired before the user confirmed, down vs v2.3.
The regression surfaced here, not in front of a customer.
Why it matters

Every prompt tweak is a fresh chance to quietly break what worked.So, we re-grade the same frozen cases, by hand, on every release.

The demo passes, the autograder scores an eight, and the regression ships.

What the judge reads

The final answer, nothing else. Every reply scores high.

Record lookup✓ 9.4/10
Status check✓ 9.1/10
Config change✓ 8.8/10
Irreversible action✓ 9.0/10
Policy question✓ 9.0/10
every reply scored 8+ship it
What a person finds

The same runs, walked step by step. The path underneath.

Right answer, wrong path✗ update_record(id: guessed) · turn 3
Passes the demo, loops in prod✗ retry loop · 27 calls
A silent regression⚠ tool selection · 3 flows broken
Acts before it confirms✗ execute_action() before confirm
Cites what doesn't exist⚠ “policy §4.2” · invented
5 would have shipped✗ hold it
Why private

Not a leaderboard. Your leaderboard.

Public benchmarks measure the model. They say nothing about your agent, on your flows, under your policy. This is a private suite built around your product: the only scoreboard that predicts whether your users have a good day.

Public benchmark · everyone’s scoreboard
1.frontier-model-a94.1
2.frontier-model-b93.8
3.frontier-model-c92.6
4.open-weights-d90.2
Measures the model. Says nothing about your agent, on your flows, under your policy.
Your private suite · v2.4
your critical flows, under your policy
your tool calls, with real ids
your handoffs, mid-conversation
your guardrails, under hostile input
The scoreboard that predicts whether your users have a good day.

The same frozen cases, on every release.

01Built and frozen

A locked, versioned set of cases built around your flows and your policy. It grows, but never silently.

02Re-run every release

The same cases against every new version. Domain graders walk every trajectory by hand. No autograder.

03The scored report lands

A release-over-release scoreboard: every regression flagged, every verdict explained, ship-readiness lanes.

Version scoreboard · your-agentsample
fetch_record()execute_action()update_record()error_handler()policy answersguardrailsmulti-issue askhostile userhandoff → humanmemory, 8 turns
v2.1v2.4
v2.49/10 passing1 regressed2 improved
passfailingimprovedregressed
@jadehuman grader
execute_action() fired before the user confirmed. Regressed vs v2.3, hold it.

Five failures that scored 8+ and still would have shipped.

Right answer, wrong path
The final message reads fine, but the agent called the wrong tool and invented an argument two steps earlier.
✓ judge: 9.4/10✗ caught: update_record(id: guessed) · turn 3
Passes the demo, loops in prod
Clean on the happy path, then 27 retries against a dead endpoint before the reply went out.
✓ judge: 9.1/10✗ caught: retry loop · 27 calls
A silent regression
A prompt tweak fixed one flow and silently broke tool selection on three others. Every answer still read clean.
✓ judge: 8.8/10⚠ caught: tool selection · 3 flows broken
Acts before it confirms
An irreversible action fired before the user confirmed. Invisible to a scorer that only reads the text.
✓ judge: 9.0/10✗ caught: execute_action() before confirm
Cites what doesn't exist
A policy section the agent made up, delivered with total confidence. The prose is flawless, so the judge scored it a nine.
✓ judge: 9.0/10⚠ caught: “policy §4.2” · invented

Every release lands as a graded report.

Ship-readiness lanes: ready, mediocre, low
A written reason on every verdict
Regressions vs your last release, path by path
A frozen, versioned suite that grows but never silently
Release report · your-agent v2.4sample
execute_action()regression vs v2.3
“Of course! I’ve gone ahead and applied the change for you.” → called execute_action
@jadehuman grader
execute_action() fired before the user confirmed. Regressed vs v2.3, hold it.
handoff → sub-agent
guardrails · pasted email
policy · stale limits
hostile user · tone
memory · 8-turn thread
multi-issue ask
fetch_record()
update_record()
status check · happy path

Who this is for

Best for you if
You ship on a weekly or biweekly cadence
Your team is too lean to hand-check every release
A regression in front of a customer is a cost you can't take
You learn about regressions from user reports, not release checks
Not for you if
You haven’t set the bar yet. Start with Gold Standard Eval.

Benefits of Release Benchmarks

Regressions surface as line items before you ship, not as incidents in front of customers.
Ship decisions take minutes: ready, mediocre, low, with a written reason on every verdict.
Prompt and model changes stop being a gamble. The same frozen cases tell you exactly what moved.
Scores stay comparable release over release, so you can prove quality is climbing, not guess.
No engineer burns release week hand-checking flows. The suite runs and your team keeps shipping.
Every report deepens the suite: new failures become frozen cases the next release is graded on.

See what we catch, on us.

Your next release is graded as a pilot. The first $200 of hand-graded evaluation is free: credited toward your retainer if you continue, yours to keep if you don't.

First $200 freeCredited if you continueThe report is yours either wayNo commitment
Book a call, grade your next release
30-min call, no prep · scope in 48h

What happens after you book

01A 30-minute call

An engineer scopes your agent with you. No deck, no prep.

02A written scope in 48 hours

What we'd test, how we'd grade it, and what it costs.

03Your first report within the week

Your first graded release report lands within the week. First $200 on us.

No codebase, no integration sprint. Your engineers stay on the roadmap.

Questions we get asked frequently

Isn't this just unit testing?
Unit tests assert exact outputs. Agents don't have exact outputs. We grade whole trajectories against your standard, and every run is read by a person, not an autograder.
How is this different from Gold Standard Eval?
Gold Standard Eval is the foundation, built once: the rubric and golden dataset that define good. Release Benchmarks are the recurring gate that holds that bar on every release.
Why not build it ourselves?
You can, but re-grading every release by hand is a standing cost that lands on an engineer you hired to ship product. We keep the suite and the graders; your engineers keep shipping.
Does the suite change over time?
It grows, but never silently. Cases are frozen and versioned, so scores stay comparable release over release. New cases land in the next suite version.
Is there a commitment?
None. Most teams run per-release on a retainer once they've seen the first report, but nothing obligates you, and there's no lock-in.

View other services

Set the bar
Gold Standard Evals

A custom rubric and a hand-graded golden dataset built around your agent: the fixed reference standard every eval, benchmark, and release measures against.

Explore →
Map the agents
Multi-Agent Audit

A hand-graded map of how your agents work together: routing, tool calls, retries, and agent-to-agent handoffs, scored privately and reproducibly, so you see what holds up and what quietly breaks.

Explore →
Tune the prompt
Prompt Tune-Up

Your agent's prompts benchmarked on real cases, rewritten for tool use and handoffs, and proven side by side: accuracy up, tokens down, edge cases fixed before your users find them.

Explore →
View all services →
Your benchmark partners

Catch the regression before your users do.

You build the agent. We hold the bar: the same frozen cases, re-graded by hand on every release, every regression flagged before it ships.

Book a call with an engineer