- Recommendation and candidate cards
- Best overall (when you compared two or more models)
- Model Comparison
- Historical Spend Breakdown and Spend Forecast
- Cost Breakdown and TCO Breakdown
- Assumptions & Formulas
- Agent Audit
- Detailed Scenario Results
Recommendation
The headline depends on the run:
The winner ranks first on judged pass rate, then on mean judged quality.
For a single model, the summary reads, for example: “Model passed 27/30 scenarios at 88% average accuracy, for 210/mo at 5K users.” with “Add another model to see a cost/speed comparison.”
For several models, it summarizes the winner’s judged passes and how it compares with the runner-up on cost, speed and accuracy.
Candidate cards
One card per model tested, each with a badge:
Hover the badge for “Passed N/M scenarios (judge-graded quality ≥ 0.70)”. Each card shows Judged quality, Cost / 1K tok (cost per thousand tokens, so models are compared on price rather than on how much work they happened to do) and Latency.
Headline tiles
Share Results copies a plain-text summary to your clipboard: the project, the recommendation, pass count, accuracy, latency and cost, the full ranking, the monthly projection, the simulation ID, and a link that reopens this run (
?run=<id>). “Results summary + link copied”.
Where latency comes from. A scenario’s latency is the sum of the durations of its spans: every model call and simulated tool call it made. It is not a separately stored timer. When a run has no span timings, latency tiles show ”—” rather than zero.
Best overall
When you compared two or more models, a second card weighs the measured figures the way a person would, not by cost alone:- AI pick: a small Claude Haiku 4.5 call weighs accuracy, pass rate, latency and projected cost for up to eight candidates and writes a one-line rationale. It is funded from your organization’s Platform Demo Key allowance, not your BYOK keys, and costs a fraction of a cent.
- Scored pick: if that call is unavailable (no allowance, timeout, or an unusable answer), a deterministic score is used instead: accuracy 45%, pass rate 20%, cost 20% and speed 15%, with cost and speed scored relative to the best candidate. “The recommendation model was unavailable, so this is the deterministic best-overall pick: accuracy and pass rate weighed against cost and latency.”
Edit scale
Every projected dollar on the page is a measured rate multiplied by a monthly volume. Edit scale opens the dialog that sets it.
The dialog shows “Projected monthly cost at scale” and, when relevant, “Projected monthly saving vs. costliest model” (several models) or “Projected monthly gap vs. Step 1 estimate” (one model). The Advanced section spells out the chain, for example “5,000 users × 3 chats = 15,000 requests/mo × 6.00/mo”.
Apply keeps the change, Cancel puts everything back. The volume and chat rate are remembered for this project in your browser, and the Overview page’s Total Cost Analyzed tile uses the same figure.
Model Comparison
Two tabs. Model Performance: a radar chart with four axes, “Accuracy (Answer correctness)”, “Token Efficiency”, “Overall Cost” and “Latency”, a legend, and a table of each model’s Judged quality, Cost/scn, Latency and Pass rate, with Win on the winner. Scenario Performance: every scenario, with one row per model, showing Accuracy, Cost/1k and Latency (click a header to sort). A model that failed the scenario is marked Error. Click a row to open the scenario. Long lists show the first scenarios with See N more scenarios. Category chips (happy path, edge case, multi turn, tool failure, refusal) filter the table. A category the run didn’t generate is disabled: “This run generated no scenarios in this category.”Historical Spend Breakdown
Shown once the project has measured run spend. Every bar is the measured spend of Cost Calculator runs, by run date, starting at the first recorded run.- Scope: Company (all projects in the organization) or Project.
- Granularity: daily, weekly, monthly, yearly. A granularity is disabled until there is enough history to fill it.
- Stat boxes: Measured spend, Budget and Variance. No budget can be set today, so Budget reads “Not set” and Variance ”—”.
- Applications (Company scope): the largest four projects by all-time measured spend, each with a “Next 3 mo” projection tagged trend (fitted to its own monthly history) or flat (held at its latest month).
Spend Forecast
Monthly spend projected forward from the current month, for the Company or Project scope. Stat boxes: 3 mo from now, This month, Starts. Where there isn’t enough history to fit a trend, the forecast assumes 8% month-over-month growth, and says so in the card description.Cost Breakdown
How each scenario’s cost splits between the agent’s reasoning and each tool. Scopes: Company (the whole run rolled into one split), Use Case (the dominant scenario category) and Project (per scenario). Tool spans carry no token cost of their own, so their share is allocated by measured call count and latency.TCO Breakdown
“Everything this project costs to run for a month at scale: measured model tokens, plus any cost lines you add.” Add a line with a cost type, an optional Custom name, and a $ amount per month, then click add. Cost types: Human-in-the-Loop (HITL), Cloud Infrastructure, Licensing & API Fees, Engineering & Maintenance, Data Pipeline & ETL, Monitoring & Observability, Security & Compliance, Training & Fine-tuning, Support & Operations, Vendor & Third-party SaaS. The donut shows Total TCO per month. Each line is tagged measured (model tokens from this run) or custom (your lines), with its share. Custom lines can be edited or removed. Nothing is assumed: if you add no lines, TCO equals measured model cost.Custom TCO lines live in the page while you have it open. They are not saved with the run.
Assumptions & Formulas
A collapsed panel under the TCO card with every parameter behind it and the arithmetic. Volume basis switches between Per user and Per workflow: per workflow lets you describe workflows (a primary and further ones) with a trigger frequency, instead of a user count. The panel shows Estimated token usage, Total API cost and Total TCO, and has a reset to the defaults. Changing the volume here changes every projection on the page.Agent Audit
Agent Audit — Failure Analysis reads the run and proposes ranked system-prompt changes.- A chart of pass and fail by category, with a toggle between This run and All runs (every run in the project).
- Run Agent Audit (when the run has failures) sends the failing transcripts and the error spans behind them to the model you tested and returns ranked fixes, most impactful first. When every scenario passed, the button is Suggest improvements and it proposes hardening for an already-working prompt instead.
- Each fix shows the change as a diff and the scenarios behind it (“Failing cases behind this” or “Passing cases behind this”). Tick the fixes you want.
- Apply changes & rerun (with the count of ticked fixes) applies the ticked fixes to the system prompt and returns you to Step 1, set up to re-test the same scenarios, so any change in the results comes from the prompt. A fix whose original text no longer appears in the prompt is left out and counted.
- Runs per case (1 to 10) with a re-run button plays the cases again to accumulate failure data over several passes.
Detailed Scenario Results
A collapsible list of every scenario, with a count (or “N measured · N projected” when synthetic rows exist). Each scenario shows:- Name, category badge, difficulty badge
- Pass, Partial or Fail with the quality score. Hover for “Judged by GPT-4.1 (independent — never the model under test) against the scenario’s goal and success criteria. Pass ≥ 0.70 quality; partial 0.50–0.69; critical failures fail outright.”
- The failure reason, and the judge’s summary
- Two bars: Task + correctness and Instruction following (or Response match and Constraint compliance when the judge was unavailable)
- Measured cost
- View full scenario → to open the conversation plan
- For multi-model runs, one result line per model instead
The Simulation ID for the batch is shown with a copy button. Use it to find the batch’s traces, which are tagged with it.