Evaluators

An evaluator is one metric: Answer Correctness, PII Leakage, Latency Budget and so on. Each one returns a score from 0 to 1, a pass/fail, and reasoning. You find them on Evaluation → Evaluators (the tab heading reads “Parameters for Evaluations”). There are two kinds by ownership:

Default evaluators ship with AgentHub and are shared by every organization. They are read-only. Editing one saves an editable copy owned by your organization.
Custom evaluators belong to your organization. Anyone with access to AgentHub can edit or delete them.

And two kinds by how they score: offline checks (tagged offline or deterministic, computed from the trace’s spans with no model call) and LLM judges (everything else, graded by a judge model). See How evaluation works.

Browsing the library

Control	What it does
All (N) / Default (N) / Custom (N)	Scope pills. The counts are what is loaded for your organization.
My Collections	Switches the panel to your collections.
Search… + Apply (or Enter)	Matches evaluator name and description.
Filter	Opens tag chips. Selecting several shows only evaluators that carry all of them.
Category tabs	All plus one tab per category present. With All selected, evaluators are grouped under category headings with counts.
+ Create New Parameter	Opens the create form.

Each evaluator card shows the name, Default or Custom, the description, its tags, and a round badge such as Q1 or A14 (category letter plus its position in the visible grid; it is cosmetic and not a stable ID). Cards have Edit and, for custom evaluators only, Delete. If the Custom scope is empty: “No custom evaluators yet”. If filters hide everything: “No evaluators match your filters” with Clear filters.

Default evaluators

The current library has 75 default evaluators across 11 categories. Every LLM-judge default carries a full rubric in its prompt: a definition, what is out of scope, sub-criteria, an anchored scoring scale, an issue vocabulary, and worked examples. You can read the whole prompt by opening Edit on the card.

The library is loaded into each deployment by an administrator. If your Default count is lower than 75, your deployment has an older copy of the library.

Tags in the tables: llm-judge means graded by a judge model; offline / deterministic means computed from spans with no model call.

Quality (11)

Evaluator	What it checks	Tags
Answer Correctness	Whether the answer is factually and procedurally right for what was actually asked.	single-turn, llm-judge, quality
Coherence	Logical flow, internal consistency and structural sense across the answer.	single-turn, llm-judge, quality
Tonality	Whether the response tone matches the requested style, audience and situation.	single-turn, llm-judge, quality
Conciseness	Penalizes padding and repetition while protecting content the reader actually needs.	single-turn, llm-judge, quality
Completeness	That every part of a multi-part request was actually addressed.	single-turn, multi-turn, llm-judge, quality
Helpfulness	Whether the response actually moves the user forward on their real goal.	single-turn, llm-judge, quality
Format Adherence	That the output matches the requested structure, schema or format exactly.	single-turn, llm-judge, quality
Readability	Whether the writing is easy to parse for the intended reader.	single-turn, llm-judge, quality
Language Consistency	That the response is in the language and locale the user expects.	single-turn, llm-judge, quality
Refusal Appropriateness	Whether refusing, or not refusing, was the right call for this request.	single-turn, llm-judge, quality, safety
Sycophancy Resistance	Agreement, flattery or capitulation not warranted by the evidence.	single-turn, multi-turn, llm-judge, quality

Agent (19)

Evaluator	What it checks	Tags
Tool Use Appropriateness	Tools were chosen correctly, invoked when needed, and skipped when not.	agent, tool-use, llm-judge
Goal Completion	Whether the agent actually achieved the objective it was given.	agent, multi-turn, llm-judge
Plan Quality	The plan the agent formed, judged only on what it knew when it formed it.	agent, llm-judge
Plan Adherence	How closely execution followed the plan the agent committed to.	agent, multi-turn, llm-judge
Step Efficiency	Penalizes redundant work, loops and detours.	agent, tool-use, llm-judge
Error Recovery	How well the agent responded to tool failures, errors and dead ends.	agent, tool-use, llm-judge
Context Retention	Facts and constraints established earlier are still honoured later.	agent, multi-turn, llm-judge
Multi-turn Consistency	Contradictions between what the agent said earlier and later.	agent, multi-turn, llm-judge
Delegation Quality	How work was handed to sub-agents, including briefing and result integration.	agent, multi-turn, llm-judge
Reasoning Quality	The soundness of the intermediate reasoning visible in the trace.	agent, llm-judge
Action Justification	Each significant action was warranted by user intent and available context.	agent, tool-use, llm-judge
Stop Condition	The agent stopped at the right point, neither early nor long past done.	agent, llm-judge
Handoff Quality	Transitions between tools, stages or workflows for lost context.	agent, tool-use, llm-judge
Memory Usage	Whether the agent read from and wrote to memory at the right moments.	agent, memory, llm-judge
Reflection Quality	Whether self-checks and self-corrections were genuine and productive.	agent, llm-judge
Autonomy Balance	The balance between acting independently and checking in with the user.	agent, multi-turn, llm-judge
Tool Argument Correctness	Tool calls had valid, correctly typed and complete arguments.	agent, tool-use, llm-judge
Tool Result Handling	What the agent did with what the tools returned.	agent, tool-use, llm-judge
User Friction	Whether the interaction generated friction: correction, retry or frustration.	agent, multi-turn, llm-judge

Safety (11)

Evaluator	What it checks	Tags
Toxicity	Toxic, abusive, demeaning or harassing language.	safety, llm-judge
Misuse Prevention	Clearly harmful or disallowed requests were declined or safely redirected.	safety, llm-judge
PII Leakage	Personal data exposed to a recipient who should not receive it.	safety, privacy, llm-judge
Harmful Content	Content that could cause physical, financial, legal or psychological harm.	safety, llm-judge
Role Violation Detection	The assistant stepping outside its assigned role, scope or operating policy.	safety, agent, llm-judge
Jailbreak Resistance	Resistance to prompt-injection and instruction-override attempts.	safety, llm-judge
Bias Mitigation	Unfair or discriminatory treatment in how the response is constructed.	safety, bias, llm-judge
Safety Refusal	The craft of a refusal when declining was the correct decision.	safety, llm-judge
Prompt Injection Resistance	Instructions found inside retrieved or tool content were treated as data.	safety, agent, tool-use, llm-judge
Excessive Agency	The agent exceeding the authority, permissions or autonomy it was granted.	safety, agent, llm-judge
Over-Refusal	Benign requests wrongly declined or watered down out of excess caution.	safety, quality, llm-judge

MCP (5)

Evaluator	What it checks	Tags
MCP Tool Selection	The right MCP server and tool were chosen from those connected.	mcp, tool-use, llm-judge
MCP Protocol Compliance	MCP interactions followed the protocol’s request and response conventions.	mcp, llm-judge
MCP Resource Access	MCP resources were accessed appropriately and minimally.	mcp, llm-judge
MCP Error Handling	How MCP protocol and tool errors were interpreted and recovered from.	mcp, tool-use, llm-judge
MCP Schema Adherence	Tool arguments against the JSON Schema the MCP server advertised.	mcp, tool-use, llm-judge

Memory (5)

Evaluator	What it checks	Tags
Memory Recall Accuracy	Facts retrieved from memory match what was actually stored.	memory, rag, llm-judge
Memory Update Correctness	What was written to memory is correct, confirmed and well scoped.	memory, agent, llm-judge
Memory Relevance	Penalizes irrelevant stored context injected into the current task.	memory, rag, llm-judge
Memory Conflict Resolution	How contradictions between stored facts and new information were resolved.	memory, agent, llm-judge
Memory Staleness	Reliance on stored facts that have expired or are no longer plausible.	memory, llm-judge

Hallucination (5)

Evaluator	What it checks	Tags
Factual Accuracy	Claims unsupported by the evidence available in the run.	hallucination, llm-judge
Citation Faithfulness	Citations point at real sources that actually support the cited claim.	hallucination, rag, faithfulness, llm-judge
Groundedness	Whether the response stays inside the evidence it was given.	hallucination, rag, faithfulness, llm-judge
Fabricated Action Detection	Claims that work was done when the trace shows it was not.	hallucination, agent, llm-judge
Numeric Consistency	Figures agree with their sources and with each other.	hallucination, rag, llm-judge

Bias (3)

Evaluator	What it checks	Tags
Demographic Fairness	Outcomes that differ across demographic groups in comparable situations.	bias, safety, llm-judge
Stereotype Avoidance	Stereotypical assumptions and group generalizations.	bias, llm-judge
Political Neutrality	Contested political questions handled without partisan advocacy.	bias, safety, llm-judge

Privacy (4)

Evaluator	What it checks	Tags
Data Minimization	Only the personal data the task requires is requested or handled.	privacy, llm-judge
Consent Awareness	Handling of data uses that require the subject’s consent or notice.	privacy, safety, llm-judge
PII Redaction Quality	Personal data that had to be masked was masked completely.	privacy, safety, llm-judge
Purpose Limitation	Data is used only for the purpose it was provided for.	privacy, llm-judge

Faithfulness (4)

Evaluator	What it checks	Tags
Instruction Following	Adherence to the explicit instructions in the system and user prompts.	faithfulness, llm-judge
Source Adherence	The answer stays faithful to the retrieved or tool-provided sources.	faithfulness, rag, llm-judge
Persona Consistency	A configured persona’s voice, knowledge and limits held throughout.	faithfulness, agent, llm-judge
Constraint Satisfaction	Hard operational constraints (budgets, limits, policies) were respected.	faithfulness, agent, llm-judge

Relevance (5)

Evaluator	What it checks	Tags
Query Relevance	How directly the response addresses the question that was asked.	relevance, single-turn, llm-judge
Response Focus	Penalizes tangents and material outside the scope of what was asked.	relevance, llm-judge
Context Relevance	Whether the retrieved context was actually useful for answering the query.	relevance, rag, llm-judge
Topic Adherence	The agent stayed within its intended subject domain across the interaction.	relevance, agent, multi-turn, llm-judge
Retrieval Utilization	How much of the useful retrieved context actually reached the answer.	relevance, rag, llm-judge

Performance (3)

Evaluator	What it checks	Tags
Latency Budget	End-to-end and per-span latency against expected budgets.	offline, deterministic, performance, agent
Token Efficiency	Token spend relative to the work the run accomplished.	offline, deterministic, performance, agent
Tool Call Economy	Number of tool calls relative to the work required.	offline, deterministic, performance, tool-use, agent

Canonical custom evaluators

An administrator can also load a set of 15 canonical evaluators into your organization. These are seeded as custom evaluators (owned by your organization, editable, deletable) and carry an extra canonical tag. When one shares a name with a default, it uses that default’s rubric.

Evaluator	Category	Type	What it measures
Tool Selection Accuracy	agent	offline (`sdk`)	Whether the agent chose the correct tools for the task.
Tool Execution Precision	agent	offline (`sdk`)	Whether invoked tools were executed correctly.
Response Similarity	quality	offline (`sdk`)	Alignment between the response and the expected answer.
Requirement Compliance	quality	offline (`sdk`)	Whether required terms, formats or constraints were satisfied.
Context Retrieval Precision	memory	offline (`sdk`)	Fraction of retrieved context that is relevant.
Context Retrieval Recall	memory	offline (`sdk`)	Fraction of required context that was retrieved.
Tonality	quality	LLM judge	Response tone matches requested style and audience.
Coherence	quality	LLM judge	Logical flow and internal consistency of the answer.
Answer Correctness	quality	LLM judge	Factual and task correctness relative to the request.
Tool Use Appropriateness	agent	LLM judge	Tools were chosen correctly and invoked when needed.
Faithfulness	faithfulness	LLM judge	Response stays faithful to retrieved sources and instructions.
Role Violation Detection	safety	LLM judge	Breaks in assigned role or policy.
Toxicity	safety	LLM judge	Toxic, abusive or harassing content.
Misuse Prevention	safety	LLM judge	Appropriate refusal or handling of harmful requests.
Safety Compliance	safety	LLM judge	Overall safe and policy-compliant behavior.

The six offline canonical evaluators are spans-only heuristics in AgentHub (see Offline evaluators below). Response Similarity and Requirement Compliance, for example, check that a substantive output exists; they do not compare against a reference answer. For a semantic comparison, use an LLM judge.

Loading the canonical set also creates the production-quality collection. See Collections.

Creating a custom evaluator

Open the form

On Evaluation → Evaluators, click + Create New Parameter. The Create Evaluator dialog opens: “Custom evaluators are owned by your organization and can be edited or deleted later.”

Fill in the fields

See the field reference below. The prompt template starts with a working example you can edit.

Save

Click Create Evaluator. The new evaluator appears under Custom and is immediately selectable in Your Evaluations and in new collections.

Field reference

Field	Required	Rules
Name	Yes	1 to 128 characters. Placeholder “e.g. Tone Checker”.
Category	No	One of Quality, Agent, Safety, MCP, Memory, Hallucination, Bias, Privacy, Faithfulness, Relevance, Performance, Custom. Defaults to Custom. Decides which tab and heading the evaluator appears under and which axis it joins in the “Score by category” chart.
Organization	Shown	Read-only. Your organization’s slug; the evaluator belongs to it.
Description	No	Up to 2,000 characters. Shown on the card and searched by Search….
Tags	No	Comma-separated, for example `single-turn, llm-judge, rag`. Up to 20 tags of 1 to 64 characters each. Tags drive filtering, and `offline` or `deterministic` change how the evaluator runs (see below).
Prompt Template	Yes	Up to 24,000 characters. Must contain at least one placeholder. See next section.

Adding the tag offline or deterministic makes the evaluator run as an offline check in trace evaluation runs. Its prompt template is then ignored there, and it is scored by a built-in heuristic chosen from its name. Unless your evaluator’s name matches one of the built-in checks below, that is the generic span-error heuristic. Only add these tags if that is what you want.

Prompt templates

A prompt template is the instruction the judge model receives. Before each judge call, AgentHub replaces these placeholders:

Placeholder	Replaced with
`{{input}}`	What the agent was asked: the trace’s input text, or your test input in Your Evaluations.
`{{output}}`	What the agent answered: the trace’s output text, or the answer of the model under test.
`{{trace}}`	A JSON summary of the trace (IDs, root name, status, duration, and each span’s name, kind, status, latency and model). Empty in Your Evaluations runs.

Details of how each value is built are in What goes into the judge. Validation rules

The template can’t be empty: “Prompt template is required.”
It must contain at least one of {{trace}}, {{input}} or {{output}}: “Include at least one placeholder: {{trace}}, {{input}}, or {{output}}.” These are checked in the form before you save and again on the server.
Placeholders are matched exactly, lowercase, with double braces and no spaces. {{ input }} or {{Input}} are not replaced.
Maximum 24,000 characters.

Expected reply. AgentHub tells the judge to reason first and reply with one JSON object. The minimum it needs is a score (0 to 1). It also reads passed (true or false) and reason or reasoning. Optionally the judge can return criteria (named sub-scores with notes), issues (each with a category, a severity of none, low, medium, high or critical, and a detail), evidence (short quotes), confidence and severity. If score is missing but criteria are present, the score is the average of the criteria. If passed is missing, the evaluator passes at a score of 0.5 or more unless the severity is critical. The template the create form starts with:

You are an expert LLM judge. Score the assistant response.

Input:
{{input}}

Output:
{{output}}

Trace context:
{{trace}}

Return JSON only: {"score": number between 0 and 1, "passed": boolean, "reason": string}

Editing and deleting

Edit on a card opens the same form, prefilled.

Custom evaluator: title “Edit evaluator”. Save changes updates it in place. Collections that include it keep pointing at it and pick up the change on their next run.
Default evaluator: title “Edit default evaluator”, with the notice “This is a system default and is read-only. Saving creates an editable copy owned by your organization; the original stays untouched.” The name is prefilled as <name> (Copy). Save as custom copy creates a new custom evaluator; the default is unchanged.

Delete appears only on custom evaluators and takes effect immediately, with no confirmation step. Deleting an evaluator also removes it from every collection that contained it. Default evaluators can’t be deleted (“Default evaluators cannot be deleted. Duplicate it to customize.”).

An evaluator that already has saved results in a trace evaluation run can’t be deleted; the delete is refused by the database to protect those results.

If a save takes longer than 20 seconds, the form says “No response after 20s. The save may not have gone through — retry, or close and check the list.” and offers Retry.

Offline evaluators

Offline checks never call a model. In a trace evaluation run, AgentHub picks the check by looking for words in the evaluator’s name (case-insensitive), in this order:

Name contains	Check	How it is scored
`latency`	Latency	Sums span latencies. Duration score is 1 at or under 3 s and falls to 0 at 20 s. Distribution score is 1 when the slowest span is at most 50% of total time, falling to 0 at 95%. Score = 0.6 × duration + 0.4 × distribution. Flags `over_budget` and `dominant_span` (slowest span over 60%). No timed spans: 0.5.
`token`	Token efficiency	Total tokens: 1 at or under 8,000, 0 at 80,000. Input tokens per model call: 1 at or under 4,000, 0 at 30,000. Score = average of the two. Flags `excessive_total` and `context_resent`. No token data: 0.5.
`tool call economy`	Tool call economy	Call count (1 at or under 8 calls, 0 at 30), share of identical repeat calls (same name and input), and failure rate. Score = 0.3 × count + 0.45 × no-duplicates + 0.25 × no-failures. No tool spans: 0.5.
`tool selection`	Tool selection	With tool spans: 1.0 if none failed, 0.5 if any failed. With no tool spans: 0.6 if no span errored, 0.2 otherwise.
`tool execution`	Tool execution	0.75 × success rate + 0.25 × share of non-duplicate calls. No tool spans: 0.5.
`response similarity` or `requirement compliance`	Output presence	No extractable output: 0.15 with a `critical` `empty_output` issue (so it fails). Output present and no span errors: 0.85 if at least 40 characters, 0.725 if shorter. Output present with span errors: 0.5.
`retrieval precision` or `retrieval recall`	Retrieval health	Looks at `db`, `http` and `tool` spans. None: 0.4. Otherwise 0.5 + 0.4 × share that succeeded.
(anything else)	Generic	0.6 × share of spans without errors + 0.4 if output text was found.

Every offline check passes at a score of 0.5 or more, unless it raised a critical issue. Each also records sub-scores, issues and a confidence, so its results chart the same way as an LLM judge’s.

Offline checks see only what the spans record. They can tell you a tool failed or a run was slow; they cannot tell you whether the answer was right. Pair them with LLM judges in a collection.

API

Evaluators are also available over the organization API, authenticated with your signed-in session:

Method	Path	Notes
`GET`	`/api/v1/orgs/{orgId}/evaluators`	Lists defaults and your organization’s evaluators.
`POST`	`/api/v1/orgs/{orgId}/evaluators`	Creates a custom evaluator. Same field rules as the form; `tags` is an array.
`PATCH`	`/api/v1/orgs/{orgId}/evaluators/{evaluatorId}`	Updates a custom evaluator. Defaults return `403`.
`DELETE`	`/api/v1/orgs/{orgId}/evaluators/{evaluatorId}`	Deletes a custom evaluator. Defaults return `403`.

Writes need the manage_agents or manage_org_settings permission (Owner, Admin and Editor have one of them).

Get started

Platform

Evaluate and observe

Administration

Browsing the library

Default evaluators

Quality (11)

Agent (19)

Safety (11)

MCP (5)

Memory (5)

Hallucination (5)

Bias (3)

Privacy (4)

Faithfulness (4)

Relevance (5)

Performance (3)

Canonical custom evaluators

Creating a custom evaluator

Field reference

Prompt templates

Editing and deleting

Offline evaluators

API

​Browsing the library

​Default evaluators

​Quality (11)

​Agent (19)

​Safety (11)

​MCP (5)

​Memory (5)

​Hallucination (5)

​Bias (3)

​Privacy (4)

​Faithfulness (4)

​Relevance (5)

​Performance (3)

​Canonical custom evaluators

​Creating a custom evaluator

​Field reference

​Prompt templates

​Editing and deleting

​Offline evaluators

​API

Browsing the library

Default evaluators

Quality (11)

Agent (19)

Safety (11)

MCP (5)

Memory (5)

Hallucination (5)

Bias (3)

Privacy (4)

Faithfulness (4)

Relevance (5)

Performance (3)

Canonical custom evaluators

Creating a custom evaluator

Field reference

Prompt templates

Editing and deleting

Offline evaluators

API