Authored by: Scott Weber, MegazoneCloud CTO, and Reynold Nathaniel Tanton, MegazoneCloud Data & AI Architect

Part 2 of 2: In Part 1, we built the governance layer — budgets enforced before spend, every dollar attributed. Part 2 answers the question governance can't: which model should you actually use? We built a benchmarking harness to find out, and the expensive model did not do well.

One of the models in our benchmark cost 13× more per successful task than another — and passed the exact same number of tests.

If your team is defaulting to the biggest model "to be safe," then your organization is wasting money. We built a system that can measure which is the best model to use.  Here is how we measured, what we found, and the two places our own method needed fixing along the way — because a benchmark you cannot audit is just an opinion with charts.
 

Price per token is the wrong metric

Model pricing pages give you dollars per million tokens. That number is almost useless for deciding which model to run, because it prices the attempt, not the outcome.

Consider a task your cheap model fails 40% of the time. Every failure gets retried, escalated to a bigger model, or — worst of all — sent out wrong and fixed by a human later. The metric that captures this is:

A model that costs half as much per token but succeeds half as often is not cheaper. Cost per successful task makes that visible; price per token hides it. In our experience, this one change in framing is what moves a cost conversation forward instead of letting it stall on "but what about quality?"
 

The harness: fan out, grade, rank

Benchmarking flow: golden dataset fans out through Step Functions to a runner Lambda per configuration; each answer is graded by an LLM judge; an aggregator ranks the cost-success matrix

The benchmarking harness (we call it the Power Tuner) is a Step Functions state machine with three moving parts:

1.     Fan out. A Map state launches one branch per candidate configuration — a configuration being a model ID plus its settings (max tokens, prompt compression ratio, system prompt) and its prices.

2.    Replay and grade. Each branch replays every case in a golden dataset through its model, then sends each answer to an LLM judge together with the case's rubric.

3.    Aggregate. A final Lambda ranks all configurations by cost per successful task and publishes the matrix.

The golden dataset for testing can be customized to the needs of your organization. It needs to be thorough enough to test the different aspects of the LLM and the Agent.  In our case the golden dataset is 40 cases across four enterpriuse task familiar, 10 each:

• Cloud/MSP support — ticket triage, IAM policy writing, HA architecture design

• General enterprise tasks — extraction, formatting, arithmetic word problems, instruction-following

• FinOps analysis — explaining bill line items, rightsizing, chargeback math

Each case has two fields: a realistic prompt, and a rubric — a plain-English statement of what a passing answer must contain.
 

How to make the judge reliable: forced tool calls

LLM-as-judge is often seen as unreliable, and most of that comes from one implementation detail: asking the judge to "respond with only JSON" and then parsing the reply. Models add extra text around the JSON. Parsers break. Broken parses get counted as candidate failures, and your headline metric is quietly wrong.

The fix: don't parse — force a tool call. The Bedrock Converse API lets you require that the model respond by invoking a tool with a JSON-schema-validated payload:

JUDGE_TOOL_CONFIG = {
    "tools": [{
        "toolSpec": {
            "name": "record_verdict",
            "inputSchema": {"json": {
                "type": "object",
                "properties": {
                    "success": {"type": "boolean"},
                   "score":  {"type": "integer", "minimum": 0, "maximum": 10},
                   "reason": {"type": "string"},
                },
                "required": ["success", "score", "reason"],
            }},
        }
    }],
    "toolChoice": {"tool": {"name": "record_verdict"}},   # no prose allowed
}

The judge cannot reply with free text — the verdict is machine-readable by construction. Add one retry for temporary throttling, and grading becomes stable and predictable: across the 120 gradings in this run (3 models × 40 cases), we had zero unusable verdicts.
 

Judge verdict flow: a forced record_verdict tool call, one retry, and an auditable failure label — never a silent miscount

The reason field matters as much as the boolean. Every verdict arrives with a one-sentence explanation, which turns the benchmark from a scoreboard into an auditable record — you'll see why that matters shortly.
 

The results

Cost per successful task: Nova Micro $0.000026 at 85% success, Nova Lite $0.000036 at 92.5%, Nova Pro $0.000469 at 92.5%
 

Nova Pro and Nova Lite passed the same number of cases. Pro cost 13× more per success. And Pro even failed a yes/no logic case ("are backups retained for more than a month?" given "retained for 35 days") that the cheaper models got right. "Use the biggest model to be safe" bought nothing here except a larger bill.

The category breakdown is where the real routing signal is:

Success by category: Micro 6/10 on MSP support but 10/10 on enterprise tasks and RAG; Lite and Pro 10/10 on MSP
 

• Micro is perfect on the task types that make up most production volume — 10/10 on extraction and formatting tasks, and 10/10 on RAG faithfulness, including correctly refusing to answer when the context did not contain the answer.

• Reasoning is what separates the tiers. Micro dropped 4 of 10 cloud-support cases; Lite and Pro passed all 10. That is the line where you need a bigger model — found with data, not guesses.

• The routing policy follows directly: default to Micro, send reasoning-heavy requests to Lite, and demand evidence before paying Pro prices for anything.
   

The failure autopsy (or: why you should read your judge's reasons)

Aggregate numbers justify decisions; individual verdicts justify trust. Here is what the judge actually said about three of Micro's six failures:

And here is the honest part: two of our rubrics were wrong. One case initially failed on all three models, which is a warning sign — when everyone fails, suspect the test. Inspection showed the judge (correctly) enforcing a rubric that demanded the exact phrase "on-demand" when a model had written "usage-based pricing." That is a rubric bug, not a model bug. We loosened the wording, kept the genuinely-failed case as-is, and re-ran.

That loop — read the reasons, fix the rubric, keep the dataset versioned — is the actual method. A golden dataset is a living asset you keep calibrating, not a file you write once. The judge's written explanations are what make that calibration possible.
 

The calibration loop: when every model fails the same case, suspect the rubric; real gaps feed the routing policy
 

What a benchmark like this costs

Under one dollar per full run — 3 models × 40 cases, with roughly 95% of the cost being the judge model (we used a top-tier model as judge; grade with a strong model, even when benchmarking cheap ones). The candidate models themselves cost fractions of a cent at small-model prices, and the harness is serverless, so it costs nothing between runs.

At that price, benchmarking stops being a project and becomes a habit: re-run weekly on a schedule, re-run when a provider updates a model, re-run before you commit to a routing change. Models do change quietly over time, and a regression test that costs less than a dollar is the cheapest insurance against it.

Takeaways

1.     Rank models by cost per successful task. Price per token prices the attempt; you care about the outcome.

2.    Don't parse judge prose — force a structured tool call. Zero unusable verdicts across the run, and every verdict carries an auditable reason.

3.    Break results down by task category. The overall number hides the routing policy; the breakdown is the routing policy.

4.    Suspect the test when everyone fails it. Read the judge's reasons; fix the rubrics; version the dataset.

5.    Make it cheap enough to re-run without a meeting. Ours costs less than a coffee, so it runs whenever we're curious.

The expensive model was 13× the price for identical results — on our tasks. Yours will be different, and that is exactly the point: stop relying on other people's benchmarks and measure your own. The harness to do it is a weekend of serverless work, and it pays for itself the first time it stops you from defaulting to the premium model.

This is Part 2 of 2. Part 1 covers the governance side: the LLM Gateway architecture, atomic reserve-and-settle budget enforcement, and per-tenant cost attribution with tagged Bedrock inference profiles.