> ## Documentation Index
> Fetch the complete documentation index at: https://docs.corepillar.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation runs

> Run evaluations in the Your Evaluations tab, start saved trace evaluation runs, choose a judge model, read the results page, and understand what happens when a judge fails.

AgentHub has two kinds of evaluation run. They share the same evaluators and the same judge engine, but they score different things and only one is saved.

| | Your Evaluations run | Trace evaluation run |
| - | - | - |
| Where you start it | **Evaluation → Your Evaluations → Run** | The evaluate API (no button in the UI today) |
| What it scores | A fresh answer from one or two of your models, generated from a test input you provide | A stored [trace](/agenthub/traces): its recorded input, output and spans |
| Evaluators | Any you tick, up to 25 | The rules of one [collection](/agenthub/evaluation/collections) |
| Offline evaluators | Graded by the judge like any other | Run as offline checks over the spans, free |
| Judge | Required | Required only if the collection has an LLM judge |
| Saved | **No.** Kept on screen for the current session | **Yes.** Results page, trace detail, history, reports |

## Your Evaluations runs

Open **Evaluation** and switch to **Your Evaluations** ("Run pairwise or pointwise evaluations against your models."). You need at least one model in [Model Manager](/agenthub/model-manager); without one the tab shows "Your Evaluations needs at least one model" and **Go to Model Manager**.

### 1. Choose a mode and models

The bar at the top of the runner:

| Control | What it does |
| - | - |
| **Single Model** | Score one model's answer (pointwise). |
| **Model vs Model (LLM-as-Judge)** | Score two models' answers side by side and pick a winner (pairwise). This is the default. |
| First model picker | The model under test. Defaults to your project's first model. |
| **vs** second picker | Only in Model vs Model. Defaults to your second model. You may pick the same model twice; the sides are then labelled "(A)" and "(B)". |
| **Judge** | The model that grades the answers. Any model in the project. Anthropic models are listed first and marked "· Recommended"; the first one is preselected. |
| Clock button | Shows or hides **Evaluation History**. |

Every model listed is a real model configured in your project, called with your own provider key.

### 2. Write the test input

The large **Test input** box is what the models under test receive as the user message. It starts filled with a ready-to-run example.

* **Use template** offers **Test input scaffold** (a structured example task) and **LLM-judge prompt**.
* The paperclip attaches a text file, or you can drop one on the box. Up to 256 KB; accepted: `.txt`, `.md`, `.markdown`, `.json`, `.jsonl`, `.csv`, `.tsv`, `.yaml`, `.yml`, `.log`, `.xml`, `.html`. Binary files such as PDFs or Word documents are refused.
* **Ctrl + Enter** (or **⌘ + Enter**) runs.
* If your text contains `{{input}}`, `{{output}}` or `{{trace}}`, a note reminds you those placeholders are only filled inside evaluator prompt templates. The box is sent to the models exactly as written.
* The status line shows how many evaluators are selected and whether a system instruction is set.

When you arrive from a trace's **Evaluate this trace**, the box is prefilled with that trace's recorded input and answer and a **Context from trace** note links back to the trace. If the trace couldn't be read, the note says **Nothing prefilled**.

### 3. Add context

The right-hand **Evaluation context** panel:

| Section | Behavior |
| - | - |
| **System Instruction** | Optional system prompt sent to the models under test. |
| **Knowledge Base** | Not available yet (marked **Soon**). |
| **Evaluators** | Tick the evaluators to grade with. Shows "N/M selected", **Select all** and **Clear**. Your organization's evaluators are listed first under **Your evaluators (N)**, then defaults by category, with **Show more...** for the rest. Up to four are preselected: your newest custom evaluators first, then Quality defaults. |
| **Skills (Tools / MCP)** | **+ Function** and **+ MCP** are disabled ("Coming soon"). |
| **Model Thinking** | **Output Tokens** for the models under test (default 1,024; the field accepts 128 to 8,192). **Thinking Level**: Off (0), Low (\~1k), Medium (\~4k), High (\~8k) or Custom. |

### 4. Run

Press **Run**. AgentHub calls the model(s) under test with your system instruction and test input, then asks the judge to grade each answer once per selected evaluator. The page shows "Running evaluation — calling the models under test, then the judge…" while it works. Judge calls run five evaluators at a time.

You can't run when:

| Message | Cause |
| - | - |
| "Enter a test input to run." | The box is empty. |
| "Select a second model for a pairwise run." | Model vs Model with no second model. |
| "Select a judge model." | No judge chosen. |
| "Select at least one evaluator in Add Context." | Nothing ticked. |
| "Select at most 25 evaluators for one run." | More than 25 ticked. |
| "Viewer role — you cannot run evaluations." | Your role lacks the run permission. |

Inputs longer than 100,000 characters are cut to that length.

### 5. Read the results

| Section | Contents |
| - | - |
| Model cards | One per model: latency in ms, input / output / total tokens, an estimated cost (`≈ $…`) for that model's answer, and the answer text. The winner's card is highlighted with a **Winner** badge. If a model call failed, its card shows the error instead. |
| **Evaluation Judge** | The judge model's name, the overall result ("… wins" or "Tie"), and a table of every evaluator with each model's score and, in pairwise mode, the per-evaluator winner. |
| **Pass rate** | A donut per model: how many evaluators passed. |
| **Score by category** | A radar chart of average score per evaluator category, one shape per model. |
| **Per-evaluator scores** | A bar per evaluator per model. In pairwise mode each row shows the gap and **Why the gap?** (see [The explain feature](/agenthub/evaluation/overview#the-explain-feature)). |

The overall winner is the model with the higher **sum** of evaluator scores; equal sums are a tie.

<Note>
  The cost shown is for the models under test only. Judge calls are billed to your provider too (one per evaluator per model) but are not added to the figure.
</Note>

<Warning>
  Your Evaluations results are not saved. They are listed in **Evaluation History** with a **This session** badge (the last 20) and can be expanded with **View**, but they disappear when you reload or leave the page, and they don't appear in Reports.
</Warning>

### Evaluation History

Below the results, **Evaluation History** ("Trace-evaluation runs and pointwise/pairwise comparisons") merges two lists, newest first:

* **Saved trace evaluation runs** for the project (the 20 most recent): collection name, time in UTC, status, average score and pass rate, and **View →** to open the run page.
* **This session** entries for Your Evaluations runs, labelled **Single Model** or **Model vs Model**, expandable inline.

With nothing yet: "No evaluation runs yet. Run a collection against a trace from Traces, or run a pointwise/pairwise comparison above."

## Trace evaluation runs

A trace evaluation run applies one collection to one stored trace and saves the result. It is the run type that uses offline checks for real, appears on the trace page, and feeds evaluation [reports](/agenthub/reports).

<Warning>
  AgentHub has no button that starts one today. The **Evaluate this trace** button on a trace opens the Your Evaluations runner instead (see above). To create a saved trace run, call the evaluate API while signed in.
</Warning>

### Start a run

<ParamField path="projectId" type="string" required>
  Your project's ID (a UUID).
</ParamField>

<ParamField path="traceId" type="string" required>
  The trace's `trace_id`, URL-encoded.
</ParamField>

<ParamField body="collection_id" type="string" required>
  The collection to run, shown as **Collection ID** on its card in **My Collections**. It must belong to your organization.
</ParamField>

<ParamField body="model_config_id" type="string | null">
  The judge model: the ID of one of this project's models. Required if the collection contains any LLM-judge evaluator. Can be omitted for a collection of offline checks only.
</ParamField>

<RequestExample>
  ```bash POST theme={null}
  curl -X POST "https://pillarforge.corepillar.ai/api/v1/projects/$PROJECT_ID/traces/$TRACE_ID/evaluate" \
    -H "Content-Type: application/json" \
    -b "$SESSION_COOKIES" \
    -d '{"collection_id":"<collection-uuid>","model_config_id":"<model-uuid>"}'
  ```
</RequestExample>

<ResponseExample>
  ```json 201 theme={null}
  {
    "data": {
      "runId": "<run-uuid>",
      "averageScore": 0.78,
      "passRate": 0.85,
      "results": [
        {
          "evaluator_name": "Answer Correctness",
          "eval_type": "online_llm_judge",
          "score": 0.9,
          "passed": true,
          "reasoning": "…"
        }
      ]
    }
  }
  ```
</ResponseExample>

The request waits until every evaluator has been scored, then returns `201`. The endpoint uses your signed-in session (the same cookies as the app), not a tracing key, and needs the `run_agents` or `manage_agents` permission. `GET` on the same path returns every saved run for the trace, newest first, with its results.

### What happens during a run

1. The collection's rules are loaded in order. If the collection has an LLM judge, the judge model must be given and must exist in the project.
2. A run is recorded with status `running`.
3. Each evaluator is scored, six at a time:
   * **Offline checks** (tagged `offline` or `deterministic`) are computed from the trace's spans. No model call.
   * **LLM judges** get their prompt template with `{{input}}`, `{{output}}` and `{{trace}}` filled from the trace, and the judge replies in JSON. Each judge call allows up to 1,500 output tokens.
4. All results are saved together, the run's **average score** (mean of scores) and **pass rate** (share passed) are computed, and the status becomes `completed`.

### Choosing a judge model

* Any model configured in the project can judge. A capable model gives steadier verdicts on the longer default rubrics; the Your Evaluations picker recommends Anthropic models.
* If the chosen judge fails with a rate limit, an outage, a retired model, or an unusable API key, AgentHub retries that evaluator on the project's **other** models, chosen judge first, then the rest ordered from the most to the least expensive tier. Requests that would fail on any model (prompt too long, content filtered, malformed) are not retried.
* The run records the judge you chose. If a fallback model actually answered, the run still shows the chosen judge's identifier.

## The run page

Open a saved run from **Evaluation History** (**View →**). It lives at `/{org}/{project}/evaluations/runs/{runId}`.

| Part | Contents |
| - | - |
| Header | The collection name, a status badge (`completed`, `failed` or `running`), and a line with the time (UTC), `trace <id>`, and `judge <model>` (or "—" for an offline-only run). |
| Stat chips | **Average score**, **Pass rate**, **Rules** (how many evaluators the collection had). |
| Error banner | For a failed run, the error message. |
| **Pass rate** | Donut of passed versus failed evaluators. |
| **Score by category** | Radar of average score per evaluator category. |
| **Per-evaluator scores** | One bar per evaluator. |
| **Reasoning** | Every evaluator with its score, **Passed** or **Failed**, a severity badge when the judge flagged a `low`, `medium`, `high` or `critical` issue, and the judge's (or offline check's) written reasoning. |

If a run has no results: "No results recorded", usually because it failed before scoring. If the link points to a run in another project or one that doesn't exist: **Run not found** with **Back to Evaluations**.

The same run also appears under **Evaluation results** on the [trace detail page](/agenthub/traces#evaluation-results), with each evaluator's **Type** (`offline_deterministic` or `online_llm_judge`), and as an **Evaluation** subject in [Reports](/agenthub/reports).

### Score, pass and reasoning

* **Score**: 0 to 1, shown as a percentage. Judges answering on 0 to 10 or 0 to 100 in whole numbers are rescaled.
* **Pass**: the judge's own `passed`. If it gave none, the evaluator passes at 0.5 or above unless it flagged a `critical` issue. Offline checks follow the same rule.
* **Reasoning**: the judge's prose. Sub-scores, issues and evidence are stored with it and drive the severity badges and charts.

### Cost

Offline checks are free. Each LLM-judge evaluator is one call to your judge model on your provider key, plus a possible JSON-repair call (see below). Saved runs don't record or display a cost, so check spend on your provider's own usage dashboard.

## Failure modes

| What happened | Result |
| - | - |
| Collection has an LLM judge but no judge was given | "Select a judge model for LLM evaluators in this collection." No run is recorded. |
| Collection or judge model not found, or the collection is empty | "Collection not found.", "Judge model not found." or "Collection has no evaluators." No run is recorded. |
| Judge rate-limited, down, retired, or its key unusable | That evaluator is retried on the project's other models. If one answers, the run continues normally. |
| Every model failed for some evaluator | The whole run is marked `failed` with the error, and **no** per-evaluator results are saved, including the ones that succeeded. The API returns `500` with "An unexpected error occurred."; read the real error on the run page. |
| Judge reply isn't valid JSON | AgentHub first tries to read the JSON itself (including inside a code fence), then asks a small CorePillar-provided model to repair it. The repair call sees only the judge's reply. |
| Judge reply was cut off mid-verdict | AgentHub recovers what it can: the last headline score, or the average of completed sub-scores, with "\[…judge reply was cut off]" added to the reasoning. |
| Nothing readable could be recovered | The evaluator is saved with score 0, failed, and reasoning beginning "Not evaluated — the judge model did not return readable JSON" (or "…ran out of output budget before finishing its verdict"), followed by the start of the raw reply. Treat these as missing, not as real failing grades. |
| A run stays `running` | The request was interrupted before it finished. Start a new run. |

In **Your Evaluations** runs the same fallback applies per evaluator, but failures stay soft: a model under test that fails shows its error on its card; an evaluator whose judge can't be reached on any model scores 0 with "Judge model unavailable: …" instead of stopping the run.

<Tip>
  If many evaluators come back as "Not evaluated" or the run fails on rate limits, run a smaller collection, choose a judge with higher rate limits, or add a second model to the project so fallback has somewhere to go.
</Tip>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.