Skip to main content
A collection is a named, ordered list of evaluators. Each evaluator in it is a rule. You build a collection once and reuse it: a trace evaluation run always applies one collection to one trace, and the trace page’s Evaluate this trace uses a collection to preselect evaluators in the runner. Collections live on Evaluation → Evaluators → My Collections.

​
Viewing collections

Each collection is a card showing:
  • the name,
  • a N rules badge and a N custom badge (how many of its rules are custom evaluators),
  • the description, if any,
  • Collection ID, the identifier you pass to the evaluate API.
Collections are listed newest first, and the first one starts expanded. Click a card to expand or collapse it. Expanded, it lists its evaluators in rule order and, for anyone who can manage evaluators, a Delete collection button. With none yet: “No collections yet. Create a collection to apply a bundle of evaluators across the product.”

​
Organization-level and project-level collections

A collection is either tied to one project or available to every project in the organization: When you open a project, AgentHub shows its project collections plus all organization-level collections.

​
Creating a collection

1

Open the dialog

In My Collections, click + New collection. The New collection dialog opens: “Bundle evaluators into a reusable suite for Playground, traces, and simulations.”
2

Name and describe it

Name is required, up to 128 characters (placeholder “Refund agent suite”). Description is optional, up to 2,000 characters.
3

Pick evaluators in order

The Evaluators (N selected) list shows every default and custom evaluator with its category and “Default” or “Custom”. Use Search evaluators to narrow it by name or description. Tick evaluators in the order you want them to run and be reported: the order you tick them is the rule order.
4

Create

Click Create collection (N). It is disabled until at least one evaluator is ticked. On success you see “Collection created — N rules.”
If creating takes longer than 20 seconds, the dialog says the collection may not have been created and offers Retry. Check My Collections before retrying so you don’t create it twice.

​
Rule order

Rule order is the order evaluators are listed in a run’s results, on the run page, and in reports. It does not change scores: every evaluator in a run is graded independently and in parallel.
There is no way to edit a collection after creating it in the UI: you can’t rename it, add or remove rules, or reorder them. To change one, create a new collection with the evaluators you want and delete the old one. Editing a custom evaluator that a collection contains does carry through, since the collection points at the evaluator itself.

​
Deleting a collection

Expand the card and click Delete collection. It takes effect immediately, with no confirmation. Deleting a collection removes its rules, not the evaluators themselves.
A collection that has been used by a saved trace evaluation run can’t be deleted. The database keeps it so those runs and their results stay readable, and the delete returns an error.
Deleting an evaluator removes it from every collection that contained it.

​
Collections in other places

  • Trace detail. The Evaluate trace panel on a trace carries over the most recently created collection available to the project. In the Your Evaluations runner that opens, the collection’s evaluators (up to 25, and only ones that still exist) are preselected. If no collection exists, the panel asks you to create one first.
  • Evaluate API. A trace evaluation run takes exactly one collection_id. See Evaluation runs.
  • Reports. An evaluation report is titled after the collection it ran. See Reports.

​
The production-quality collection

When an administrator loads the canonical evaluator set for your organization, AgentHub also creates an organization-level collection named production-quality: “SDK canonical bundle (custom evaluators) — offline metrics + core LLM judges.” It has 13 rules, in this order: All 13 are the organization’s custom copies of the canonical evaluators (see Canonical custom evaluators), so you can edit them. Because it contains LLM judges, running it needs a judge model. If the collection already exists, loading the canonical set again leaves it as it is.

​
API

Both use your signed-in session, not a tracing key.