Skip to main content
Evaluation in AgentHub answers one question: did this output hold up? You pick evaluators (individual metrics such as Answer Correctness or Latency Budget), optionally bundle them into a collection, and run them. Every evaluator returns a score from 0 to 1, a pass/fail, and written reasoning. AgentHub then rolls those up into an average score and a pass rate. Open it from Evaluation in the sidebar. The page has two tabs: A five-step bar across the top of the page tracks where you are: Choose Evaluators, Pick Models, Add Context, Run Evaluation, See Results. Steps tick off as you do them; clicking a step switches to the matching tab.

​
The building blocks

​
Offline checks versus LLM judges

An evaluator tagged offline or deterministic is an offline check. Everything else is an LLM judge. How each offline check is computed is listed in Evaluators.
Offline checks only run as offline checks when a collection is run against a stored trace. In the Your Evaluations runner, every selected evaluator, offline-tagged ones included, is graded by the judge model, because there is no trace with spans to measure, only a fresh model answer.

​
Two ways to run an evaluation

​
1. Your Evaluations (in the app)

The runner on the Your Evaluations tab. You type or paste a test input (or carry one over from a trace), pick one model (Single Model) or two (Model vs Model (LLM-as-Judge)), pick a judge, choose evaluators, and press Run. AgentHub calls the model(s) under test with your input, then asks the judge to grade each answer with each evaluator. With two models it picks a winner per evaluator and overall. These results appear on screen and in the history list as This session. They are not saved: reload the page and they are gone.

​
2. Trace evaluation runs (API)

A collection applied to a stored trace. Offline checks run against the trace’s spans; LLM judges grade the trace’s recorded input and output. The run, every per-evaluator result, the average score and the pass rate are saved. Saved runs show up:
  • under Evaluation results on the trace detail page,
  • in Evaluation History on the Your Evaluations tab, linking to a run page,
  • as Evaluation subjects in Reports.
There is currently no button in AgentHub that starts a trace evaluation run. Evaluate this trace on a trace opens the Your Evaluations runner with the trace’s text prefilled (method 1), and its results are not saved. Saved trace runs are created through the evaluate API. See Evaluation runs.

​
What goes into the judge

For a trace evaluation run, AgentHub builds three values from the trace and substitutes them into the evaluator’s prompt template: In the Your Evaluations runner, {{input}} is your test input, {{output}} is the answer of the model under test, and {{trace}} is empty. The judge is told to reason first and reply with a single JSON object. It may also return sub-scores per criterion, issues with a severity (none, low, medium, high, critical), short evidence quotes, and a confidence. AgentHub keeps these alongside the reasoning, and they feed the charts and the Why the gap? explanations.

​
Scores, passing and roll-ups

  • Score is a number from 0 to 1, shown as a percentage. If a judge answers on a 0 to 10 or 0 to 100 scale in whole numbers, AgentHub rescales it.
  • Passed is whatever the judge returned. If it returned no pass/fail, the evaluator passes when its score is at least 0.5 and the worst issue is not critical. Offline checks use the same rule.
  • Average score for a run is the mean of every evaluator’s score.
  • Pass rate is the share of evaluators that passed. It is a rate over evaluators, not over scenarios or traces.
If a judge reply can’t be read even after AgentHub asks a model to repair the JSON, the evaluator is recorded with score 0, failed, and reasoning that starts “Not evaluated —” so it is not mistaken for a real failing grade. See Failure modes.

​
The explain feature

In a Model vs Model run, each evaluator row in Per-evaluator scores shows the gap (“Model A +12 pts”, or “Tied at 80%”) and a Why the gap? (or Why the tie?) button. Opening it asks a small model to read the two verdicts the judge already produced for that evaluator (scores, pass/fail, reasoning, sub-scores, issues and evidence) and explain in one or two sentences what separated them. It does not see the answers themselves and does not re-judge them.
  • The explanation runs on a CorePillar-provided model, not your provider keys.
  • Each explanation is fetched once per evaluator per run and then cached in the page.
  • If the explanation model can’t be reached or your organization’s platform allowance is used up, AgentHub states the score difference from the numbers instead, labelled “Written from the scores”.

​
Who can do what

AgentHub is only open to Owners, Admins and Editors (Viewers use ModelHub). All three roles can browse, create, edit and delete custom evaluators and collections, and run evaluations. The permission checks behind each action are listed on the individual pages.

​
Next steps

Evaluators

The full default library, custom evaluators, and prompt templates.

Collections

Bundle evaluators into a reusable suite.

Evaluation runs

Run an evaluation and read the results.

Traces

The traces that trace evaluation runs score.