> ## Documentation Index
> Fetch the complete documentation index at: https://docs.corepillar.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluators
> The default evaluator library, the canonical custom set, how to create and edit your own evaluator, prompt-template rules, and how each offline check is scored.
An **evaluator** is one metric: Answer Correctness, PII Leakage, Latency Budget and so on. Each one returns a score from 0 to 1, a pass/fail, and reasoning. You find them on **Evaluation → Evaluators** (the tab heading reads "Parameters for Evaluations").
There are two kinds by ownership:
* **Default** evaluators ship with AgentHub and are shared by every organization. They are read-only. Editing one saves an editable copy owned by your organization.
* **Custom** evaluators belong to your organization. Anyone with access to AgentHub can edit or delete them.
And two kinds by how they score: **offline checks** (tagged `offline` or `deterministic`, computed from the trace's spans with no model call) and **LLM judges** (everything else, graded by a judge model). See [How evaluation works](/agenthub/evaluation/overview#offline-checks-versus-llm-judges).
## Browsing the library
| Control | What it does |
| - | - |
| **All (N)** / **Default (N)** / **Custom (N)** | Scope pills. The counts are what is loaded for your organization. |
| **My Collections** | Switches the panel to your [collections](/agenthub/evaluation/collections). |
| **Search...** + **Apply** (or Enter) | Matches evaluator name and description. |
| **Filter** | Opens tag chips. Selecting several shows only evaluators that carry **all** of them. |
| Category tabs | **All** plus one tab per category present. With **All** selected, evaluators are grouped under category headings with counts. |
| **+ Create New Parameter** | Opens the create form. |
Each evaluator card shows the name, **Default** or **Custom**, the description, its tags, and a round badge such as `Q1` or `A14` (category letter plus its position in the visible grid; it is cosmetic and not a stable ID). Cards have **Edit** and, for custom evaluators only, **Delete**.
If the Custom scope is empty: "No custom evaluators yet". If filters hide everything: "No evaluators match your filters" with **Clear filters**.
## Default evaluators
The current library has **75** default evaluators across 11 categories. Every LLM-judge default carries a full rubric in its prompt: a definition, what is out of scope, sub-criteria, an anchored scoring scale, an issue vocabulary, and worked examples. You can read the whole prompt by opening **Edit** on the card.
The library is loaded into each deployment by an administrator. If your **Default** count is lower than 75, your deployment has an older copy of the library.
Tags in the tables: `llm-judge` means graded by a judge model; `offline` / `deterministic` means computed from spans with no model call.
### Quality (11)
| Evaluator | What it checks | Tags |
| - | - | - |
| Answer Correctness | Whether the answer is factually and procedurally right for what was actually asked. | single-turn, llm-judge, quality |
| Coherence | Logical flow, internal consistency and structural sense across the answer. | single-turn, llm-judge, quality |
| Tonality | Whether the response tone matches the requested style, audience and situation. | single-turn, llm-judge, quality |
| Conciseness | Penalizes padding and repetition while protecting content the reader actually needs. | single-turn, llm-judge, quality |
| Completeness | That every part of a multi-part request was actually addressed. | single-turn, multi-turn, llm-judge, quality |
| Helpfulness | Whether the response actually moves the user forward on their real goal. | single-turn, llm-judge, quality |
| Format Adherence | That the output matches the requested structure, schema or format exactly. | single-turn, llm-judge, quality |
| Readability | Whether the writing is easy to parse for the intended reader. | single-turn, llm-judge, quality |
| Language Consistency | That the response is in the language and locale the user expects. | single-turn, llm-judge, quality |
| Refusal Appropriateness | Whether refusing, or not refusing, was the right call for this request. | single-turn, llm-judge, quality, safety |
| Sycophancy Resistance | Agreement, flattery or capitulation not warranted by the evidence. | single-turn, multi-turn, llm-judge, quality |
### Agent (19)
| Evaluator | What it checks | Tags |
| - | - | - |
| Tool Use Appropriateness | Tools were chosen correctly, invoked when needed, and skipped when not. | agent, tool-use, llm-judge |
| Goal Completion | Whether the agent actually achieved the objective it was given. | agent, multi-turn, llm-judge |
| Plan Quality | The plan the agent formed, judged only on what it knew when it formed it. | agent, llm-judge |
| Plan Adherence | How closely execution followed the plan the agent committed to. | agent, multi-turn, llm-judge |
| Step Efficiency | Penalizes redundant work, loops and detours. | agent, tool-use, llm-judge |
| Error Recovery | How well the agent responded to tool failures, errors and dead ends. | agent, tool-use, llm-judge |
| Context Retention | Facts and constraints established earlier are still honoured later. | agent, multi-turn, llm-judge |
| Multi-turn Consistency | Contradictions between what the agent said earlier and later. | agent, multi-turn, llm-judge |
| Delegation Quality | How work was handed to sub-agents, including briefing and result integration. | agent, multi-turn, llm-judge |
| Reasoning Quality | The soundness of the intermediate reasoning visible in the trace. | agent, llm-judge |
| Action Justification | Each significant action was warranted by user intent and available context. | agent, tool-use, llm-judge |
| Stop Condition | The agent stopped at the right point, neither early nor long past done. | agent, llm-judge |
| Handoff Quality | Transitions between tools, stages or workflows for lost context. | agent, tool-use, llm-judge |
| Memory Usage | Whether the agent read from and wrote to memory at the right moments. | agent, memory, llm-judge |
| Reflection Quality | Whether self-checks and self-corrections were genuine and productive. | agent, llm-judge |
| Autonomy Balance | The balance between acting independently and checking in with the user. | agent, multi-turn, llm-judge |
| Tool Argument Correctness | Tool calls had valid, correctly typed and complete arguments. | agent, tool-use, llm-judge |
| Tool Result Handling | What the agent did with what the tools returned. | agent, tool-use, llm-judge |
| User Friction | Whether the interaction generated friction: correction, retry or frustration. | agent, multi-turn, llm-judge |
### Safety (11)
| Evaluator | What it checks | Tags |
| - | - | - |
| Toxicity | Toxic, abusive, demeaning or harassing language. | safety, llm-judge |
| Misuse Prevention | Clearly harmful or disallowed requests were declined or safely redirected. | safety, llm-judge |
| PII Leakage | Personal data exposed to a recipient who should not receive it. | safety, privacy, llm-judge |
| Harmful Content | Content that could cause physical, financial, legal or psychological harm. | safety, llm-judge |
| Role Violation Detection | The assistant stepping outside its assigned role, scope or operating policy. | safety, agent, llm-judge |
| Jailbreak Resistance | Resistance to prompt-injection and instruction-override attempts. | safety, llm-judge |
| Bias Mitigation | Unfair or discriminatory treatment in how the response is constructed. | safety, bias, llm-judge |
| Safety Refusal | The craft of a refusal when declining was the correct decision. | safety, llm-judge |
| Prompt Injection Resistance | Instructions found inside retrieved or tool content were treated as data. | safety, agent, tool-use, llm-judge |
| Excessive Agency | The agent exceeding the authority, permissions or autonomy it was granted. | safety, agent, llm-judge |
| Over-Refusal | Benign requests wrongly declined or watered down out of excess caution. | safety, quality, llm-judge |
### MCP (5)
| Evaluator | What it checks | Tags |
| - | - | - |
| MCP Tool Selection | The right MCP server and tool were chosen from those connected. | mcp, tool-use, llm-judge |
| MCP Protocol Compliance | MCP interactions followed the protocol's request and response conventions. | mcp, llm-judge |
| MCP Resource Access | MCP resources were accessed appropriately and minimally. | mcp, llm-judge |
| MCP Error Handling | How MCP protocol and tool errors were interpreted and recovered from. | mcp, tool-use, llm-judge |
| MCP Schema Adherence | Tool arguments against the JSON Schema the MCP server advertised. | mcp, tool-use, llm-judge |
### Memory (5)
| Evaluator | What it checks | Tags |
| - | - | - |
| Memory Recall Accuracy | Facts retrieved from memory match what was actually stored. | memory, rag, llm-judge |
| Memory Update Correctness | What was written to memory is correct, confirmed and well scoped. | memory, agent, llm-judge |
| Memory Relevance | Penalizes irrelevant stored context injected into the current task. | memory, rag, llm-judge |
| Memory Conflict Resolution | How contradictions between stored facts and new information were resolved. | memory, agent, llm-judge |
| Memory Staleness | Reliance on stored facts that have expired or are no longer plausible. | memory, llm-judge |
### Hallucination (5)
| Evaluator | What it checks | Tags |
| - | - | - |
| Factual Accuracy | Claims unsupported by the evidence available in the run. | hallucination, llm-judge |
| Citation Faithfulness | Citations point at real sources that actually support the cited claim. | hallucination, rag, faithfulness, llm-judge |
| Groundedness | Whether the response stays inside the evidence it was given. | hallucination, rag, faithfulness, llm-judge |
| Fabricated Action Detection | Claims that work was done when the trace shows it was not. | hallucination, agent, llm-judge |
| Numeric Consistency | Figures agree with their sources and with each other. | hallucination, rag, llm-judge |
### Bias (3)
| Evaluator | What it checks | Tags |
| - | - | - |
| Demographic Fairness | Outcomes that differ across demographic groups in comparable situations. | bias, safety, llm-judge |
| Stereotype Avoidance | Stereotypical assumptions and group generalizations. | bias, llm-judge |
| Political Neutrality | Contested political questions handled without partisan advocacy. | bias, safety, llm-judge |
### Privacy (4)
| Evaluator | What it checks | Tags |
| - | - | - |
| Data Minimization | Only the personal data the task requires is requested or handled. | privacy, llm-judge |
| Consent Awareness | Handling of data uses that require the subject's consent or notice. | privacy, safety, llm-judge |
| PII Redaction Quality | Personal data that had to be masked was masked completely. | privacy, safety, llm-judge |
| Purpose Limitation | Data is used only for the purpose it was provided for. | privacy, llm-judge |
### Faithfulness (4)
| Evaluator | What it checks | Tags |
| - | - | - |
| Instruction Following | Adherence to the explicit instructions in the system and user prompts. | faithfulness, llm-judge |
| Source Adherence | The answer stays faithful to the retrieved or tool-provided sources. | faithfulness, rag, llm-judge |
| Persona Consistency | A configured persona's voice, knowledge and limits held throughout. | faithfulness, agent, llm-judge |
| Constraint Satisfaction | Hard operational constraints (budgets, limits, policies) were respected. | faithfulness, agent, llm-judge |
### Relevance (5)
| Evaluator | What it checks | Tags |
| - | - | - |
| Query Relevance | How directly the response addresses the question that was asked. | relevance, single-turn, llm-judge |
| Response Focus | Penalizes tangents and material outside the scope of what was asked. | relevance, llm-judge |
| Context Relevance | Whether the retrieved context was actually useful for answering the query. | relevance, rag, llm-judge |
| Topic Adherence | The agent stayed within its intended subject domain across the interaction. | relevance, agent, multi-turn, llm-judge |
| Retrieval Utilization | How much of the useful retrieved context actually reached the answer. | relevance, rag, llm-judge |
### Performance (3)
| Evaluator | What it checks | Tags |
| - | - | - |
| Latency Budget | End-to-end and per-span latency against expected budgets. | offline, deterministic, performance, agent |
| Token Efficiency | Token spend relative to the work the run accomplished. | offline, deterministic, performance, agent |
| Tool Call Economy | Number of tool calls relative to the work required. | offline, deterministic, performance, tool-use, agent |
## Canonical custom evaluators
An administrator can also load a set of **15 canonical evaluators** into your organization. These are seeded as **custom** evaluators (owned by your organization, editable, deletable) and carry an extra `canonical` tag. When one shares a name with a default, it uses that default's rubric.
| Evaluator | Category | Type | What it measures |
| - | - | - | - |
| Tool Selection Accuracy | agent | offline (`sdk`) | Whether the agent chose the correct tools for the task. |
| Tool Execution Precision | agent | offline (`sdk`) | Whether invoked tools were executed correctly. |
| Response Similarity | quality | offline (`sdk`) | Alignment between the response and the expected answer. |
| Requirement Compliance | quality | offline (`sdk`) | Whether required terms, formats or constraints were satisfied. |
| Context Retrieval Precision | memory | offline (`sdk`) | Fraction of retrieved context that is relevant. |
| Context Retrieval Recall | memory | offline (`sdk`) | Fraction of required context that was retrieved. |
| Tonality | quality | LLM judge | Response tone matches requested style and audience. |
| Coherence | quality | LLM judge | Logical flow and internal consistency of the answer. |
| Answer Correctness | quality | LLM judge | Factual and task correctness relative to the request. |
| Tool Use Appropriateness | agent | LLM judge | Tools were chosen correctly and invoked when needed. |
| Faithfulness | faithfulness | LLM judge | Response stays faithful to retrieved sources and instructions. |
| Role Violation Detection | safety | LLM judge | Breaks in assigned role or policy. |
| Toxicity | safety | LLM judge | Toxic, abusive or harassing content. |
| Misuse Prevention | safety | LLM judge | Appropriate refusal or handling of harmful requests. |
| Safety Compliance | safety | LLM judge | Overall safe and policy-compliant behavior. |
The six offline canonical evaluators are spans-only heuristics in AgentHub (see [Offline evaluators](#offline-evaluators) below). Response Similarity and Requirement Compliance, for example, check that a substantive output exists; they do not compare against a reference answer. For a semantic comparison, use an LLM judge.
Loading the canonical set also creates the **production-quality** collection. See [Collections](/agenthub/evaluation/collections#the-production-quality-collection).
## Creating a custom evaluator
On **Evaluation → Evaluators**, click **+ Create New Parameter**. The **Create Evaluator** dialog opens: "Custom evaluators are owned by your organization and can be edited or deleted later."
See the field reference below. The prompt template starts with a working example you can edit.
Click **Create Evaluator**. The new evaluator appears under **Custom** and is immediately selectable in Your Evaluations and in new collections.
### Field reference
| Field | Required | Rules |
| - | - | - |
| **Name** | Yes | 1 to 128 characters. Placeholder "e.g. Tone Checker". |
| **Category** | No | One of Quality, Agent, Safety, MCP, Memory, Hallucination, Bias, Privacy, Faithfulness, Relevance, Performance, Custom. Defaults to **Custom**. Decides which tab and heading the evaluator appears under and which axis it joins in the "Score by category" chart. |
| **Organization** | Shown | Read-only. Your organization's slug; the evaluator belongs to it. |
| **Description** | No | Up to 2,000 characters. Shown on the card and searched by **Search...**. |
| **Tags** | No | Comma-separated, for example `single-turn, llm-judge, rag`. Up to 20 tags of 1 to 64 characters each. Tags drive filtering, and `offline` or `deterministic` change how the evaluator runs (see below). |
| **Prompt Template** | Yes | Up to 24,000 characters. Must contain at least one placeholder. See next section. |
Adding the tag `offline` or `deterministic` makes the evaluator run as an **offline check** in trace evaluation runs. Its prompt template is then ignored there, and it is scored by a built-in heuristic chosen from its name. Unless your evaluator's name matches one of the built-in checks below, that is the generic span-error heuristic. Only add these tags if that is what you want.
### Prompt templates
A prompt template is the instruction the judge model receives. Before each judge call, AgentHub replaces these placeholders:
| Placeholder | Replaced with |
| - | - |
| `{{input}}` | What the agent was asked: the trace's input text, or your test input in Your Evaluations. |
| `{{output}}` | What the agent answered: the trace's output text, or the answer of the model under test. |
| `{{trace}}` | A JSON summary of the trace (IDs, root name, status, duration, and each span's name, kind, status, latency and model). Empty in Your Evaluations runs. |
Details of how each value is built are in [What goes into the judge](/agenthub/evaluation/overview#what-goes-into-the-judge).
**Validation rules**
* The template can't be empty: "Prompt template is required."
* It must contain at least one of `{{trace}}`, `{{input}}` or `{{output}}`: "Include at least one placeholder: `{{trace}}`, `{{input}}`, or `{{output}}`." These are checked in the form before you save and again on the server.
* Placeholders are matched exactly, lowercase, with double braces and no spaces. `{{ input }}` or `{{Input}}` are not replaced.
* Maximum 24,000 characters.
**Expected reply.** AgentHub tells the judge to reason first and reply with one JSON object. The minimum it needs is a `score` (0 to 1). It also reads `passed` (true or false) and `reason` or `reasoning`. Optionally the judge can return `criteria` (named sub-scores with notes), `issues` (each with a category, a severity of `none`, `low`, `medium`, `high` or `critical`, and a detail), `evidence` (short quotes), `confidence` and `severity`. If `score` is missing but criteria are present, the score is the average of the criteria. If `passed` is missing, the evaluator passes at a score of 0.5 or more unless the severity is `critical`.
The template the create form starts with:
```text theme={null}
You are an expert LLM judge. Score the assistant response.
Input:
{{input}}
Output:
{{output}}
Trace context:
{{trace}}
Return JSON only: {"score": number between 0 and 1, "passed": boolean, "reason": string}
```
## Editing and deleting
**Edit** on a card opens the same form, prefilled.
* **Custom evaluator:** title "Edit evaluator". **Save changes** updates it in place. Collections that include it keep pointing at it and pick up the change on their next run.
* **Default evaluator:** title "Edit default evaluator", with the notice "This is a system default and is read-only. Saving creates an editable copy owned by your organization; the original stays untouched." The name is prefilled as ` (Copy)`. **Save as custom copy** creates a new custom evaluator; the default is unchanged.
**Delete** appears only on custom evaluators and takes effect immediately, with no confirmation step. Deleting an evaluator also removes it from every collection that contained it. Default evaluators can't be deleted ("Default evaluators cannot be deleted. Duplicate it to customize.").
An evaluator that already has saved results in a trace evaluation run can't be deleted; the delete is refused by the database to protect those results.
If a save takes longer than 20 seconds, the form says "No response after 20s. The save may not have gone through — retry, or close and check the list." and offers **Retry**.
## Offline evaluators
Offline checks never call a model. In a trace evaluation run, AgentHub picks the check by looking for words in the evaluator's **name** (case-insensitive), in this order:
| Name contains | Check | How it is scored |
| - | - | - |
| `latency` | Latency | Sums span latencies. Duration score is 1 at or under 3 s and falls to 0 at 20 s. Distribution score is 1 when the slowest span is at most 50% of total time, falling to 0 at 95%. Score = 0.6 × duration + 0.4 × distribution. Flags `over_budget` and `dominant_span` (slowest span over 60%). No timed spans: 0.5. |
| `token` | Token efficiency | Total tokens: 1 at or under 8,000, 0 at 80,000. Input tokens per model call: 1 at or under 4,000, 0 at 30,000. Score = average of the two. Flags `excessive_total` and `context_resent`. No token data: 0.5. |
| `tool call economy` | Tool call economy | Call count (1 at or under 8 calls, 0 at 30), share of identical repeat calls (same name and input), and failure rate. Score = 0.3 × count + 0.45 × no-duplicates + 0.25 × no-failures. No tool spans: 0.5. |
| `tool selection` | Tool selection | With tool spans: 1.0 if none failed, 0.5 if any failed. With no tool spans: 0.6 if no span errored, 0.2 otherwise. |
| `tool execution` | Tool execution | 0.75 × success rate + 0.25 × share of non-duplicate calls. No tool spans: 0.5. |
| `response similarity` or `requirement compliance` | Output presence | No extractable output: 0.15 with a `critical` `empty_output` issue (so it fails). Output present and no span errors: 0.85 if at least 40 characters, 0.725 if shorter. Output present with span errors: 0.5. |
| `retrieval precision` or `retrieval recall` | Retrieval health | Looks at `db`, `http` and `tool` spans. None: 0.4. Otherwise 0.5 + 0.4 × share that succeeded. |
| *(anything else)* | Generic | 0.6 × share of spans without errors + 0.4 if output text was found. |
Every offline check passes at a score of 0.5 or more, unless it raised a `critical` issue. Each also records sub-scores, issues and a confidence, so its results chart the same way as an LLM judge's.
Offline checks see only what the spans record. They can tell you a tool failed or a run was slow; they cannot tell you whether the answer was right. Pair them with LLM judges in a collection.
## API
Evaluators are also available over the organization API, authenticated with your signed-in session:
| Method | Path | Notes |
| - | - | - |
| `GET` | `/api/v1/orgs/{orgId}/evaluators` | Lists defaults and your organization's evaluators. |
| `POST` | `/api/v1/orgs/{orgId}/evaluators` | Creates a custom evaluator. Same field rules as the form; `tags` is an array. |
| `PATCH` | `/api/v1/orgs/{orgId}/evaluators/{evaluatorId}` | Updates a custom evaluator. Defaults return `403`. |
| `DELETE` | `/api/v1/orgs/{orgId}/evaluators/{evaluatorId}` | Deletes a custom evaluator. Defaults return `403`. |
Writes need the `manage_agents` or `manage_org_settings` permission (Owner, Admin and Editor have one of them).
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.