> ## Documentation Index > Fetch the complete documentation index at: https://docs.corepillar.ai/llms.txt > Use this file to discover all available pages before exploring further. # How evaluation works > Evaluators, collections, offline checks versus LLM judges, and the two ways AgentHub turns them into scores and pass rates. Evaluation in AgentHub answers one question: did this output hold up? You pick **evaluators** (individual metrics such as Answer Correctness or Latency Budget), optionally bundle them into a **collection**, and run them. Every evaluator returns a **score** from 0 to 1, a **pass/fail**, and written **reasoning**. AgentHub then rolls those up into an **average score** and a **pass rate**. Open it from **Evaluation** in the sidebar. The page has two tabs: | Tab | What it's for | | - | - | | **Evaluators** | Browse the evaluator library (heading: "Parameters for Evaluations"), create and edit your own, and manage collections under **My Collections**. See [Evaluators](/agenthub/evaluation/evaluators) and [Collections](/agenthub/evaluation/collections). | | **Your Evaluations** | Run an evaluation: one model scored on its own, or two models compared by a judge. Also shows your evaluation history. See [Evaluation runs](/agenthub/evaluation/runs). | A five-step bar across the top of the page tracks where you are: **Choose Evaluators**, **Pick Models**, **Add Context**, **Run Evaluation**, **See Results**. Steps tick off as you do them; clicking a step switches to the matching tab. ## The building blocks | Concept | What it is | | - | - | | **Evaluator** | One metric. It has a name, a category, tags, and a prompt template. Its tags decide whether it runs as an offline check or an LLM judge. | | **Default evaluator** | One of the system evaluators that ship with AgentHub. Every organization sees the same set. Read-only; editing one saves a copy. | | **Custom evaluator** | An evaluator your organization owns. You can edit and delete it. | | **Collection** | An ordered list of evaluators (its **rules**), saved under a name so you can run the same bundle again. | | **Judge model** | One of your project's models from [Model Manager](/agenthub/model-manager), used to grade outputs for LLM-judge evaluators. Calls go through your own provider keys. | | **Run** | One application of a set of evaluators to one output. | ## Offline checks versus LLM judges An evaluator tagged `offline` or `deterministic` is an **offline check**. Everything else is an **LLM judge**. | | Offline check | LLM judge | | - | - | - | | How it scores | Fixed rules over the trace's spans (counts, durations, tokens, statuses). | A judge model reads the evaluator's prompt with the input, output and trace filled in, and answers in JSON. | | Model call | None. | One call per evaluator per output, on your keys. | | Cost | Free. | Billed by your provider. | | Repeatable | Yes, same trace gives the same score. | Mostly; a model can vary. | | What it can see | Only what the spans recorded: kinds, statuses, latency, tokens, tool calls. It cannot judge meaning. | Whatever the prompt template includes: the input, the output and a trace summary. | | Result type label | `offline_deterministic` | `online_llm_judge` | How each offline check is computed is listed in [Evaluators](/agenthub/evaluation/evaluators#offline-evaluators). Offline checks only run as offline checks when a **collection is run against a stored trace**. In the **Your Evaluations** runner, every selected evaluator, offline-tagged ones included, is graded by the judge model, because there is no trace with spans to measure, only a fresh model answer. ## Two ways to run an evaluation ```mermaid theme={null} flowchart TD subgraph Library["Evaluator library"] D["Default evaluators"] C["Custom evaluators"] end Library --> Col["Collection
(ordered rules)"] Library --> Pick["Evaluators picked
in Your Evaluations"] T["Stored trace
(spans)"] --> TR["Trace evaluation run
(evaluate API)"] Col --> TR TR --> Split{"Evaluator tagged
offline or deterministic?"} Split -- yes --> Off["Offline check
no model call"] Split -- no --> Judge["LLM judge
on your judge model"] Off --> Saved["Saved run
score, pass, reasoning per evaluator
average score, pass rate"] Judge --> Saved Saved --> RunPage["Run page and
trace detail"] Saved --> Rep["Evaluation report"] Input["Test input
(typed, or carried from a trace)"] --> AH["Your Evaluations run
(Single Model or Model vs Model)"] Pick --> AH AH --> Gen["Model(s) under test
generate an answer"] Gen --> Judge2["LLM judge grades
each answer per evaluator"] Judge2 --> Session["Results on screen
kept for this session only"] ``` ### 1. Your Evaluations (in the app) The runner on the **Your Evaluations** tab. You type or paste a test input (or carry one over from a trace), pick one model (**Single Model**) or two (**Model vs Model (LLM-as-Judge)**), pick a judge, choose evaluators, and press **Run**. AgentHub calls the model(s) under test with your input, then asks the judge to grade each answer with each evaluator. With two models it picks a winner per evaluator and overall. These results appear on screen and in the history list as **This session**. They are **not saved**: reload the page and they are gone. ### 2. Trace evaluation runs (API) A **collection** applied to a **stored trace**. Offline checks run against the trace's spans; LLM judges grade the trace's recorded input and output. The run, every per-evaluator result, the average score and the pass rate are **saved**. Saved runs show up: * under **Evaluation results** on the [trace detail page](/agenthub/traces#evaluation-results), * in **Evaluation History** on the Your Evaluations tab, linking to a run page, * as **Evaluation** subjects in [Reports](/agenthub/reports). There is currently no button in AgentHub that starts a trace evaluation run. **Evaluate this trace** on a trace opens the Your Evaluations runner with the trace's text prefilled (method 1), and its results are not saved. Saved trace runs are created through the evaluate API. See [Evaluation runs](/agenthub/evaluation/runs#trace-evaluation-runs). ## What goes into the judge For a trace evaluation run, AgentHub builds three values from the trace and substitutes them into the evaluator's prompt template: | Placeholder | Filled with | | - | - | | `{{input}}` | The text of the root span's input. If the root span has no readable text (common for OpenTelemetry wrapper spans), the last model-call span's input. When the input has separate system and user parts, both are included. | | `{{output}}` | The root span's output text, or the last model-call span's output when the root has none. | | `{{trace}}` | A JSON summary of the trace: ID, root name, status, duration, span count, and for each span its ID, name, kind, status, latency and model. Span payloads are not included. | In the Your Evaluations runner, `{{input}}` is your test input, `{{output}}` is the answer of the model under test, and `{{trace}}` is empty. The judge is told to reason first and reply with a single JSON object. It may also return sub-scores per criterion, issues with a severity (`none`, `low`, `medium`, `high`, `critical`), short evidence quotes, and a confidence. AgentHub keeps these alongside the reasoning, and they feed the charts and the **Why the gap?** explanations. ## Scores, passing and roll-ups * **Score** is a number from 0 to 1, shown as a percentage. If a judge answers on a 0 to 10 or 0 to 100 scale in whole numbers, AgentHub rescales it. * **Passed** is whatever the judge returned. If it returned no pass/fail, the evaluator passes when its score is at least 0.5 and the worst issue is not `critical`. Offline checks use the same rule. * **Average score** for a run is the mean of every evaluator's score. * **Pass rate** is the share of evaluators that passed. It is a rate over evaluators, not over scenarios or traces. If a judge reply can't be read even after AgentHub asks a model to repair the JSON, the evaluator is recorded with score 0, failed, and reasoning that starts "Not evaluated —" so it is not mistaken for a real failing grade. See [Failure modes](/agenthub/evaluation/runs#failure-modes). ## The explain feature In a **Model vs Model** run, each evaluator row in **Per-evaluator scores** shows the gap ("Model A +12 pts", or "Tied at 80%") and a **Why the gap?** (or **Why the tie?**) button. Opening it asks a small model to read the two verdicts the judge already produced for that evaluator (scores, pass/fail, reasoning, sub-scores, issues and evidence) and explain in one or two sentences what separated them. It does not see the answers themselves and does not re-judge them. * The explanation runs on a CorePillar-provided model, not your provider keys. * Each explanation is fetched once per evaluator per run and then cached in the page. * If the explanation model can't be reached or your organization's platform allowance is used up, AgentHub states the score difference from the numbers instead, labelled "Written from the scores". ## Who can do what AgentHub is only open to Owners, Admins and Editors (Viewers use ModelHub). All three roles can browse, create, edit and delete custom evaluators and collections, and run evaluations. The permission checks behind each action are listed on the individual pages. ## Next steps The full default library, custom evaluators, and prompt templates. Bundle evaluators into a reusable suite. Run an evaluation and read the results. The traces that trace evaluation runs score. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.