Evaluation runs

curl -X POST "https://pillarforge.corepillar.ai/api/v1/projects/$PROJECT_ID/traces/$TRACE_ID/evaluate" \
  -H "Content-Type: application/json" \
  -b "$SESSION_COOKIES" \
  -d '{"collection_id":"<collection-uuid>","model_config_id":"<model-uuid>"}'

{
  "data": {
    "runId": "<run-uuid>",
    "averageScore": 0.78,
    "passRate": 0.85,
    "results": [
      {
        "evaluator_name": "Answer Correctness",
        "eval_type": "online_llm_judge",
        "score": 0.9,
        "passed": true,
        "reasoning": "…"
      }
    ]
  }
}

curl -X POST "https://pillarforge.corepillar.ai/api/v1/projects/$PROJECT_ID/traces/$TRACE_ID/evaluate" \
  -H "Content-Type: application/json" \
  -b "$SESSION_COOKIES" \
  -d '{"collection_id":"<collection-uuid>","model_config_id":"<model-uuid>"}'

{
  "data": {
    "runId": "<run-uuid>",
    "averageScore": 0.78,
    "passRate": 0.85,
    "results": [
      {
        "evaluator_name": "Answer Correctness",
        "eval_type": "online_llm_judge",
        "score": 0.9,
        "passed": true,
        "reasoning": "…"
      }
    ]
  }
}

AgentHub has two kinds of evaluation run. They share the same evaluators and the same judge engine, but they score different things and only one is saved.

	Your Evaluations run	Trace evaluation run
Where you start it	Evaluation → Your Evaluations → Run	The evaluate API (no button in the UI today)
What it scores	A fresh answer from one or two of your models, generated from a test input you provide	A stored trace: its recorded input, output and spans
Evaluators	Any you tick, up to 25	The rules of one collection
Offline evaluators	Graded by the judge like any other	Run as offline checks over the spans, free
Judge	Required	Required only if the collection has an LLM judge
Saved	No. Kept on screen for the current session	Yes. Results page, trace detail, history, reports

Your Evaluations runs

Open Evaluation and switch to Your Evaluations (“Run pairwise or pointwise evaluations against your models.”). You need at least one model in Model Manager; without one the tab shows “Your Evaluations needs at least one model” and Go to Model Manager.

1. Choose a mode and models

The bar at the top of the runner:

Control	What it does
Single Model	Score one model’s answer (pointwise).
Model vs Model (LLM-as-Judge)	Score two models’ answers side by side and pick a winner (pairwise). This is the default.
First model picker	The model under test. Defaults to your project’s first model.
vs second picker	Only in Model vs Model. Defaults to your second model. You may pick the same model twice; the sides are then labelled “(A)” and “(B)”.
Judge	The model that grades the answers. Any model in the project. Anthropic models are listed first and marked ”· Recommended”; the first one is preselected.
Clock button	Shows or hides Evaluation History.

Every model listed is a real model configured in your project, called with your own provider key.

2. Write the test input

The large Test input box is what the models under test receive as the user message. It starts filled with a ready-to-run example.

Use template offers Test input scaffold (a structured example task) and LLM-judge prompt.
The paperclip attaches a text file, or you can drop one on the box. Up to 256 KB; accepted: .txt, .md, .markdown, .json, .jsonl, .csv, .tsv, .yaml, .yml, .log, .xml, .html. Binary files such as PDFs or Word documents are refused.
Ctrl + Enter (or ⌘ + Enter) runs.
If your text contains {{input}}, {{output}} or {{trace}}, a note reminds you those placeholders are only filled inside evaluator prompt templates. The box is sent to the models exactly as written.
The status line shows how many evaluators are selected and whether a system instruction is set.

When you arrive from a trace’s Evaluate this trace, the box is prefilled with that trace’s recorded input and answer and a Context from trace note links back to the trace. If the trace couldn’t be read, the note says Nothing prefilled.

3. Add context

The right-hand Evaluation context panel:

Section	Behavior
System Instruction	Optional system prompt sent to the models under test.
Knowledge Base	Not available yet (marked Soon).
Evaluators	Tick the evaluators to grade with. Shows “N/M selected”, Select all and Clear. Your organization’s evaluators are listed first under Your evaluators (N), then defaults by category, with Show more… for the rest. Up to four are preselected: your newest custom evaluators first, then Quality defaults.
Skills (Tools / MCP)	+ Function and + MCP are disabled (“Coming soon”).
Model Thinking	Output Tokens for the models under test (default 1,024; the field accepts 128 to 8,192). Thinking Level: Off (0), Low (~1k), Medium (~4k), High (~8k) or Custom.

4. Run

Press Run. AgentHub calls the model(s) under test with your system instruction and test input, then asks the judge to grade each answer once per selected evaluator. The page shows “Running evaluation — calling the models under test, then the judge…” while it works. Judge calls run five evaluators at a time. You can’t run when:

Message	Cause
”Enter a test input to run.”	The box is empty.
”Select a second model for a pairwise run.”	Model vs Model with no second model.
”Select a judge model.”	No judge chosen.
”Select at least one evaluator in Add Context.”	Nothing ticked.
”Select at most 25 evaluators for one run.”	More than 25 ticked.
”Viewer role — you cannot run evaluations.”	Your role lacks the run permission.

Inputs longer than 100,000 characters are cut to that length.

5. Read the results

Section	Contents
Model cards	One per model: latency in ms, input / output / total tokens, an estimated cost (`≈ $…`) for that model’s answer, and the answer text. The winner’s card is highlighted with a Winner badge. If a model call failed, its card shows the error instead.
Evaluation Judge	The judge model’s name, the overall result (”… wins” or “Tie”), and a table of every evaluator with each model’s score and, in pairwise mode, the per-evaluator winner.
Pass rate	A donut per model: how many evaluators passed.
Score by category	A radar chart of average score per evaluator category, one shape per model.
Per-evaluator scores	A bar per evaluator per model. In pairwise mode each row shows the gap and Why the gap? (see The explain feature).

The overall winner is the model with the higher sum of evaluator scores; equal sums are a tie.

The cost shown is for the models under test only. Judge calls are billed to your provider too (one per evaluator per model) but are not added to the figure.

Your Evaluations results are not saved. They are listed in Evaluation History with a This session badge (the last 20) and can be expanded with View, but they disappear when you reload or leave the page, and they don’t appear in Reports.

Evaluation History

Below the results, Evaluation History (“Trace-evaluation runs and pointwise/pairwise comparisons”) merges two lists, newest first:

Saved trace evaluation runs for the project (the 20 most recent): collection name, time in UTC, status, average score and pass rate, and View → to open the run page.
This session entries for Your Evaluations runs, labelled Single Model or Model vs Model, expandable inline.

With nothing yet: “No evaluation runs yet. Run a collection against a trace from Traces, or run a pointwise/pairwise comparison above.”

Trace evaluation runs

A trace evaluation run applies one collection to one stored trace and saves the result. It is the run type that uses offline checks for real, appears on the trace page, and feeds evaluation reports.

AgentHub has no button that starts one today. The Evaluate this trace button on a trace opens the Your Evaluations runner instead (see above). To create a saved trace run, call the evaluate API while signed in.

Start a run

string

required

Your project’s ID (a UUID).

string

required

The trace’s trace_id, URL-encoded.

string

required

The collection to run, shown as Collection ID on its card in My Collections. It must belong to your organization.

string | null

The judge model: the ID of one of this project’s models. Required if the collection contains any LLM-judge evaluator. Can be omitted for a collection of offline checks only.

curl -X POST "https://pillarforge.corepillar.ai/api/v1/projects/$PROJECT_ID/traces/$TRACE_ID/evaluate" \
  -H "Content-Type: application/json" \
  -b "$SESSION_COOKIES" \
  -d '{"collection_id":"<collection-uuid>","model_config_id":"<model-uuid>"}'

{
  "data": {
    "runId": "<run-uuid>",
    "averageScore": 0.78,
    "passRate": 0.85,
    "results": [
      {
        "evaluator_name": "Answer Correctness",
        "eval_type": "online_llm_judge",
        "score": 0.9,
        "passed": true,
        "reasoning": "…"
      }
    ]
  }
}

The request waits until every evaluator has been scored, then returns 201. The endpoint uses your signed-in session (the same cookies as the app), not a tracing key, and needs the run_agents or manage_agents permission. GET on the same path returns every saved run for the trace, newest first, with its results.

What happens during a run

The collection’s rules are loaded in order. If the collection has an LLM judge, the judge model must be given and must exist in the project.
A run is recorded with status running.
Each evaluator is scored, six at a time:
- Offline checks (tagged offline or deterministic) are computed from the trace’s spans. No model call.
- LLM judges get their prompt template with {{input}}, {{output}} and {{trace}} filled from the trace, and the judge replies in JSON. Each judge call allows up to 1,500 output tokens.
All results are saved together, the run’s average score (mean of scores) and pass rate (share passed) are computed, and the status becomes completed.

Choosing a judge model

Any model configured in the project can judge. A capable model gives steadier verdicts on the longer default rubrics; the Your Evaluations picker recommends Anthropic models.
If the chosen judge fails with a rate limit, an outage, a retired model, or an unusable API key, AgentHub retries that evaluator on the project’s other models, chosen judge first, then the rest ordered from the most to the least expensive tier. Requests that would fail on any model (prompt too long, content filtered, malformed) are not retried.
The run records the judge you chose. If a fallback model actually answered, the run still shows the chosen judge’s identifier.

The run page

Open a saved run from Evaluation History (View →). It lives at /{org}/{project}/evaluations/runs/{runId}.

Part	Contents
Header	The collection name, a status badge (`completed`, `failed` or `running`), and a line with the time (UTC), `trace <id>`, and `judge <model>` (or ”—” for an offline-only run).
Stat chips	Average score, Pass rate, Rules (how many evaluators the collection had).
Error banner	For a failed run, the error message.
Pass rate	Donut of passed versus failed evaluators.
Score by category	Radar of average score per evaluator category.
Per-evaluator scores	One bar per evaluator.
Reasoning	Every evaluator with its score, Passed or Failed, a severity badge when the judge flagged a `low`, `medium`, `high` or `critical` issue, and the judge’s (or offline check’s) written reasoning.

If a run has no results: “No results recorded”, usually because it failed before scoring. If the link points to a run in another project or one that doesn’t exist: Run not found with Back to Evaluations. The same run also appears under Evaluation results on the trace detail page, with each evaluator’s Type (offline_deterministic or online_llm_judge), and as an Evaluation subject in Reports.

Score, pass and reasoning

Score: 0 to 1, shown as a percentage. Judges answering on 0 to 10 or 0 to 100 in whole numbers are rescaled.
Pass: the judge’s own passed. If it gave none, the evaluator passes at 0.5 or above unless it flagged a critical issue. Offline checks follow the same rule.
Reasoning: the judge’s prose. Sub-scores, issues and evidence are stored with it and drive the severity badges and charts.

Cost

Offline checks are free. Each LLM-judge evaluator is one call to your judge model on your provider key, plus a possible JSON-repair call (see below). Saved runs don’t record or display a cost, so check spend on your provider’s own usage dashboard.

Failure modes

What happened	Result
Collection has an LLM judge but no judge was given	”Select a judge model for LLM evaluators in this collection.” No run is recorded.
Collection or judge model not found, or the collection is empty	”Collection not found.”, “Judge model not found.” or “Collection has no evaluators.” No run is recorded.
Judge rate-limited, down, retired, or its key unusable	That evaluator is retried on the project’s other models. If one answers, the run continues normally.
Every model failed for some evaluator	The whole run is marked `failed` with the error, and no per-evaluator results are saved, including the ones that succeeded. The API returns `500` with “An unexpected error occurred.”; read the real error on the run page.
Judge reply isn’t valid JSON	AgentHub first tries to read the JSON itself (including inside a code fence), then asks a small CorePillar-provided model to repair it. The repair call sees only the judge’s reply.
Judge reply was cut off mid-verdict	AgentHub recovers what it can: the last headline score, or the average of completed sub-scores, with ”[…judge reply was cut off]” added to the reasoning.
Nothing readable could be recovered	The evaluator is saved with score 0, failed, and reasoning beginning “Not evaluated — the judge model did not return readable JSON” (or “…ran out of output budget before finishing its verdict”), followed by the start of the raw reply. Treat these as missing, not as real failing grades.
A run stays `running`	The request was interrupted before it finished. Start a new run.

In Your Evaluations runs the same fallback applies per evaluator, but failures stay soft: a model under test that fails shows its error on its card; an evaluator whose judge can’t be reached on any model scores 0 with “Judge model unavailable: …” instead of stopping the run.

If many evaluators come back as “Not evaluated” or the run fails on rate limits, run a smaller collection, choose a judge with higher rate limits, or add a second model to the project so fallback has somewhere to go.

Get started

Platform

Evaluate and observe

Administration

Your Evaluations runs

1. Choose a mode and models

2. Write the test input

3. Add context

4. Run

5. Read the results

Evaluation History

Trace evaluation runs

Start a run

What happens during a run

Choosing a judge model

The run page

Score, pass and reasoning

Cost

Failure modes

​Your Evaluations runs

​1. Choose a mode and models

​2. Write the test input

​3. Add context

​4. Run

​5. Read the results

​Evaluation History

​Trace evaluation runs

​Start a run

​What happens during a run

​Choosing a judge model

​The run page

​Score, pass and reasoning

​Cost

​Failure modes

Your Evaluations runs

1. Choose a mode and models

2. Write the test input

3. Add context

4. Run

5. Read the results

Evaluation History

Trace evaluation runs

Start a run

What happens during a run

Choosing a judge model

The run page

Score, pass and reasoning

Cost

Failure modes