> ## Documentation Index
> Fetch the complete documentation index at: https://docs.corepillar.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Collections

> Bundle evaluators into an ordered, reusable suite, see which collections a project can use, and what the production-quality collection contains.

A **collection** is a named, ordered list of evaluators. Each evaluator in it is a **rule**. You build a collection once and reuse it: a [trace evaluation run](/agenthub/evaluation/runs#trace-evaluation-runs) always applies one collection to one trace, and the trace page's **Evaluate this trace** uses a collection to preselect evaluators in the runner.

Collections live on **Evaluation → Evaluators → My Collections**.

## Viewing collections

Each collection is a card showing:

* the name,
* a **N rules** badge and a **N custom** badge (how many of its rules are custom evaluators),
* the description, if any,
* **Collection ID**, the identifier you pass to the evaluate API.

Collections are listed newest first, and the first one starts expanded. Click a card to expand or collapse it. Expanded, it lists its evaluators in rule order and, for anyone who can manage evaluators, a **Delete collection** button.

With none yet: "No collections yet. Create a collection to apply a bundle of evaluators across the product."

## Organization-level and project-level collections

A collection is either tied to one project or available to every project in the organization:

| Scope | Visible in | How it's created |
| - | - | - |
| **Project** | Only that project. | Every collection you create from the AgentHub UI is tied to the project you have open. |
| **Organization** | Every project in the organization. | Created through the API with no `project_id`, or by the seed that creates the production-quality collection. |

When you open a project, AgentHub shows its project collections plus all organization-level collections.

## Creating a collection

<Steps>
  <Step title="Open the dialog">
    In **My Collections**, click **+ New collection**. The **New collection** dialog opens: "Bundle evaluators into a reusable suite for Playground, traces, and simulations."
  </Step>

  <Step title="Name and describe it">
    **Name** is required, up to 128 characters (placeholder "Refund agent suite"). **Description** is optional, up to 2,000 characters.
  </Step>

  <Step title="Pick evaluators in order">
    The **Evaluators (N selected)** list shows every default and custom evaluator with its category and "Default" or "Custom". Use **Search evaluators** to narrow it by name or description. Tick evaluators **in the order you want them to run and be reported**: the order you tick them is the rule order.
  </Step>

  <Step title="Create">
    Click **Create collection (N)**. It is disabled until at least one evaluator is ticked. On success you see "Collection created — N rules."
  </Step>
</Steps>

| Limit | Value |
| - | - |
| Evaluators per collection | 1 to 100 |
| Name | 128 characters |
| Description | 2,000 characters |

If creating takes longer than 20 seconds, the dialog says the collection may not have been created and offers **Retry**. Check **My Collections** before retrying so you don't create it twice.

### Rule order

Rule order is the order evaluators are listed in a run's results, on the run page, and in reports. It does not change scores: every evaluator in a run is graded independently and in parallel.

<Note>
  There is no way to edit a collection after creating it in the UI: you can't rename it, add or remove rules, or reorder them. To change one, create a new collection with the evaluators you want and delete the old one. Editing a **custom evaluator** that a collection contains does carry through, since the collection points at the evaluator itself.
</Note>

## Deleting a collection

Expand the card and click **Delete collection**. It takes effect immediately, with no confirmation. Deleting a collection removes its rules, not the evaluators themselves.

<Warning>
  A collection that has been used by a saved trace evaluation run can't be deleted. The database keeps it so those runs and their results stay readable, and the delete returns an error.
</Warning>

Deleting an evaluator removes it from every collection that contained it.

## Collections in other places

* **Trace detail.** The **Evaluate trace** panel on a trace carries over the **most recently created** collection available to the project. In the Your Evaluations runner that opens, the collection's evaluators (up to 25, and only ones that still exist) are preselected. If no collection exists, the panel asks you to create one first.
* **Evaluate API.** A trace evaluation run takes exactly one `collection_id`. See [Evaluation runs](/agenthub/evaluation/runs#trace-evaluation-runs).
* **Reports.** An evaluation report is titled after the collection it ran. See [Reports](/agenthub/reports).

## The production-quality collection

When an administrator loads the canonical evaluator set for your organization, AgentHub also creates an organization-level collection named **production-quality**: "SDK canonical bundle (custom evaluators) — offline metrics + core LLM judges." It has 13 rules, in this order:

| # | Evaluator | Type |
| - | - | - |
| 1 | Tonality | LLM judge |
| 2 | Coherence | LLM judge |
| 3 | Answer Correctness | LLM judge |
| 4 | Role Violation Detection | LLM judge |
| 5 | Toxicity | LLM judge |
| 6 | Misuse Prevention | LLM judge |
| 7 | Safety Compliance | LLM judge |
| 8 | Tool Use Appropriateness | LLM judge |
| 9 | Faithfulness | LLM judge |
| 10 | Tool Selection Accuracy | Offline |
| 11 | Tool Execution Precision | Offline |
| 12 | Response Similarity | Offline |
| 13 | Requirement Compliance | Offline |

All 13 are the organization's **custom** copies of the canonical evaluators (see [Canonical custom evaluators](/agenthub/evaluation/evaluators#canonical-custom-evaluators)), so you can edit them. Because it contains LLM judges, running it needs a judge model. If the collection already exists, loading the canonical set again leaves it as it is.

## API

| Method | Path | Notes |
| - | - | - |
| `GET` | `/api/v1/orgs/{orgId}/evaluator-collections` | Add `?project_id=` to get that project's collections plus organization-level ones. Without it, every collection in the organization. Each collection includes its rules in order. |
| `POST` | `/api/v1/orgs/{orgId}/evaluator-collections` | Body: `name`, optional `description`, optional `project_id` (omit or `null` for organization-level), and `evaluator_ids` (1 to 100, in rule order). Needs `manage_agents` or `manage_org_settings`. |

Both use your signed-in session, not a tracing key.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.