What should this agent do?
The main text area is the agent’s system prompt. Placeholder: “Describe what this agent should do, what it should watch out for, and when it needs to check with a human before acting.”- Required. Generate Scenarios shows “Add a system prompt to continue” if it is empty.
- Up to 20,000 characters.
- Both the scenario generator and every model under test receive it, so write it the way you would for production.
Start from…
A menu of the ten built-in templates. Choosing one replaces the system prompt, tone, skills, memory and tools with the template’s (“Loaded “name” template”). The last option, + Start New Project (blank), clears the form. Neither option creates a project; they only change this form.Import
Upload a file to fill the form. Accepted:.md, .markdown, .txt, .json. The button reads Parsing… while it works.
Results: “Imported file — form prefilled”, “Imported file — parsed with AI”, “Couldn’t find an agent config in that file”, or “Could not read that file”. The AI fallback runs on CorePillar’s platform key, not yours.
Import replaces the system prompt, tone and items in the form. It does not merge with what you had.
From traces
Opens Import from traces, which reads OpenTelemetry runs your agent already sent to this project. You can either merge what they show (system prompt, tone, tools) into this form, or adopt the runs directly as a finished, measured run. See Import from traces.Tone & personality
A single line, for example “Analytical, direct, concise, no fluff”. Optional, up to 2,000 characters. It is passed to the generator and to every model under test.Skills, Memory & Tools
“Skills define what the agent can do. Memory gives it context to work with. Tools connect it to external systems.” Three buttons add items:The Configure dialog (Skill and Memory)
Click Done to save. When editing an existing item, a Remove action appears. Naming an item the same as another shows “Another item is already called “name”. Saving adds a second one.”
Configured skills and memory appear as rows you can click to edit (pencil) or remove (x). Tools appear as compact chips with their own remove button. All items together are capped at 50.
Tools are always simulated during a run. The model under test sees a realistic result for each call it makes, but nothing is sent to the real system.
Model & key
The right-hand card picks which models the scenarios will run on. It has two tabs.Saved Models
Every model registered to this project in Model Manager, as a checklist. Each row shows the provider icon, the provider and display name, and a badge for the key it is bound to.- Tick one model to forecast its cost. Tick several to compare them; “Results run against each model independently.”
- The first model in your selection is the primary model. The run is recorded against it, and the Agent Audit uses it.
- At least one model must stay selected. With exactly one selected you see “Locked in: a run needs at least one model selected.” Trying to untick it shows “This model is locked in — a run needs at least one model. Select another model first, then this one can be cleared.”
Add New
Add a model without leaving the page.
Click Add (reads Validating…). AgentHub sends a small test request to that model with that key. On success the model is saved to the project, appears in a numbered list, and becomes the Active (primary) model. On failure: “That API key didn’t work for model — double-check the key and that the model is enabled for it.”
Under the card: “Encrypted and stored in your project’s key vault. Used for generation only — never seen by the sandboxed agent.” The key is stored in your organization’s encrypted vault exactly as in Model Manager. Note that it is used to run scenarios on that model, which is billed to it; scenario generation itself never uses your keys.
Revoke in this list only removes the model from this session’s selection. It does not revoke the key or delete the model from the project. To do either, use Model Manager.
Scenarios to generate
Estimated cost
Once at least one model is selected and the count is valid, the card shows an Estimated cost line per model and, for several models, a Total. A summary line underneath reads like: “30 scenarios ≈ 30 model calls · 1 turn each · ~N tokens” (one model, multi-turn off) With several models it adds “per model” and “across all N models”. When the batch mixes scenarios with different turn counts (multi-turn on), it says “X turns per scenario on average — the batch mixes A–B-turn scenarios” instead. Hover the line to see the batch mix, for example “15 happy path × 1 turn, 6 edge case × 1 turn, …”. How it is calculated, without any network call:- The batch is split into categories using the same mix the generator uses (see categories).
- Each turn re-sends the system context, the whole conversation so far and every tool result so far, which is how real execution works. Long multi-turn scenarios therefore cost much more than short ones.
- Text is counted at about 4 characters per token. A synthetic user message is assumed to be about 25 tokens. Expected answer length is banded by the model’s output price.
- Prices come from the built-in catalog. Each model’s output cap is its configured default, or 1,024 tokens.
Generate Scenarios
The main button at the bottom right. It reads Generating… while working and moves you to Step 2. It is disabled, with a reason shown beside it, when:- you do not have permission to run scenarios (“You need permission to run scenarios in this project.”), or
- the scenario count is invalid (“Fix the scenario count to continue.”).