Who writes the scenarios
Scenarios are written by one fixed, independent generator model: Claude Sonnet 5.5, on CorePillar’s own key. The screen says so: “Scenarios generated by Claude Sonnet 5.5 (independent) — never by a model under test, so no model is graded on test cases it wrote itself.”- Generation never uses your API keys and is not billed to you.
- The generator reads your system prompt, tone, skills, memory and tools, and writes scenarios specific to them: a goal, success criteria, a difficulty, and a turn-by-turn plan of what the synthetic user says and which tools get called.
- Scenarios are requested one category at a time, in parallel, so each category gets its own budget and the expensive multi-turn scenarios come back at full length.
Scenario categories
The mix is weighted towards normal traffic because it is meant to forecast cost. With Include multi-turn scenarios on, every batch has at least one multi-turn scenario. With it off, multi-turn is dropped, its share goes to the other categories, and every scenario is a single user turn.
Each scenario also gets a difficulty: Easy, Medium or Hard. The generator states it, or AgentHub derives it from the category and turn count (tool failure is always hard; a multi-turn scenario of 10 or more user turns is hard).
While scenarios generate
You move to Step 2 immediately. Scenarios stream in as each generation call finishes, so the list fills in rather than appearing all at once.- Before the first scenario arrives, a ticker cycles through stages (“Reading agent system prompt & tone…”, “Mapping tools & connectors…”, “Drafting synthetic user personas…”, and so on) above placeholder cards. Up to 10 placeholders are drawn, with ”+ N more scenarios generating…” for the rest.
- The heading reads “Generating N/M scenarios…” and then “N scenarios generated”.
- A coverage counter shows “N/5 categories” (or /4 with multi-turn off), with a chip per category that lights up once it has a scenario.
- Each scenario is a row: “Scenario N of M”, its name, category badge, the opening user message, and “N turns · N tool calls (simulated)” or “no tool calls”. Click it to open the scenario drawer.
The scenario drawer
Opening a scenario shows:- Category badge and turn count
- Conversation plan: every turn, labelled by role. User turns are blue, simulated tool turns are dashed monospace, drafted agent replies are grey.
- Score profile: a radar chart of “Target profile”, what a strong result looks like for this category. After a run, “Actual (measured)” is drawn over it.
- Monitored parameters (when opened from Simulations): deterministic scores from 0 to 1 for this scenario’s measured output.
Failed slots
Occasionally the generator can’t fill a slot. A slot that fails is retried once; a multi-turn slot that still fails is filled with an offline plan rather than shipped short. If a slot still has nothing, you see:- An amber banner: “N scenarios failed to generate — scenario 4 of 30, …, marked in red below.” with Show me (scrolls to the first one) and Retry / Retry all N.
- A red card per failed slot, “Scenario generation failed”, with Retry this scenario (reads Regenerating…).
- Run scenarios is disabled, with “N scenarios have no plan — retry them below, or edit the setup and generate again.”
Run scenarios
Two buttons sit at the top of the list: Edit setup (back to Step 1) and Run scenarios. When you click it:- A toast says “Running N models…”.
- For a multi-model run, AgentHub copies the scenario batch once per extra model, so every model plays exactly the same scenarios. Each model’s copy is its own run (it shows as its own row in Simulations), grouped under one simulation ID.
- Scenarios run in parallel, up to 32 at a time across the batch, shared equally between models with at least 4 each. Turns inside a scenario run in order, because each turn’s prompt includes the conversation so far. Rate-limit errors are retried with backoff.
- For each turn, any tool calls the plan places there run first. Their simulated results are written into the conversation, so the model reasons over them and is billed for them. Tool failure scenarios inject an error into one call, then continue so you can see whether the agent recovers.
- Each model is told that its connected tools are executed by the platform and to treat tool results as real data, so it doesn’t refuse with “I don’t have access to that system”.
Live progress
While the run executes:- A progress bar: “N/M scenario runs finished”, with a failed count when any failed, and “$X measured so far”.
- For several models, a per-model counter, for example “gpt-4.1: 12/30”. A model whose key can’t be resolved shows “no key” and is skipped, without stopping the others.
- Each scenario row gets a result line per model: Running, then a difficulty badge, a Pass / Partial / Fail verdict with its quality score, and “$cost · latency · tokens”. Failed rows show the judge’s reason underneath. Expand a row to see each success criterion marked “Met” or “Missed”.
How each scenario is graded
Every executed scenario, for every model, is graded by one fixed, independent judge: GPT-4.1 at temperature 0, on CorePillar’s own key. It is never the model under test, and is a different model family from the generator. The judge reads the whole transcript against the scenario’s goal and success criteria and scores three axes from 0 to 1:
A critical failure is a hard gate: complying with an out-of-policy request, inventing data to get past a missing tool result, or leaking something the agent shouldn’t. Tool-call correctness is not part of the verdict, because the harness scripts every tool call; how well the agent used the results is graded under correctness.
If the judge is unavailable for a scenario (for example it hits a rate limit or times out), that scenario falls back to offline keyword heuristics and is labelled “Heuristic score (judge unavailable)”. Treat those verdicts as rough.