> ## Documentation Index > Fetch the complete documentation index at: https://docs.corepillar.ai/llms.txt > Use this file to discover all available pages before exploring further. # Traces > Browse every run your agent sent to AgentHub, open one to see each step it took, and hand it to Evaluation or Reports. A **trace** is one run of your agent: one request, one conversation turn, one job. Each trace holds **spans**, one per step the agent took (a model call, a tool call, an HTTP request, a database query, or any function you chose to time). The **Traces** page in the AgentHub sidebar lists the traces for the project you have selected and lets you open any one of them. Traces arrive in two ways: * **Your own agent** sends them, either as JSON to the ingest endpoint or through an OpenTelemetry exporter. See the [tracing quickstart](/agenthub/developers/tracing-quickstart), the [trace format](/agenthub/developers/trace-format) and [OpenTelemetry (OTLP)](/agenthub/developers/opentelemetry). * **The [Cost Calculator](/agenthub/cost-calculator)** writes one trace per scenario it executes. These are marked as `simulation` traces. ## The traces list ### Columns | Column | What it shows | | - | - | | **Trace** | The trace's root name as a link (or its ID when it has no root name). Under it: the trace ID and up to three provenance badges (see below). | | **Status** | `OK` or `ERROR`. A trace is `ERROR` if you sent it as `ERROR` or if any of its spans is `ERROR`. | | **Duration** | Total run time of the trace. | | **Spans** | How many spans the trace has. | | **Tokens** | Input tokens / output tokens, summed across spans. | | **Cost** | Recorded cost in USD, summed across spans. | | **Started** | When the trace started, in your browser's local time. | | *(unlabelled)* | A **Report** button that opens the [report](/agenthub/reports) for this trace. | The list is sorted newest first. It loads 50 traces at a time; **Load more** at the bottom appends the next 50 without moving you back to the top. A counter under the table reads "Showing N traces". ### Provenance badges The small outline badges next to a trace ID tell you where it came from. They are read from the trace's `source`, `environment` and `tags`: | Badge | Shown when | | - | - | | `simulation` | The trace's source is `simulation` (written by the Cost Calculator). | | the SDK or library name | A tag of the form `sdk:` is present, for example from an OpenTelemetry instrumentation library. The badge shows ``. | | `otel` | The trace arrived through the OpenTelemetry receiver and no `sdk:` tag was set. | | the service name | A tag of the form `service:` is present. The badge shows ``. | | the environment | The trace's environment is anything other than `production` (the default). | At most three badges are shown. Hover a badge to see the trace's full tag list. ### Source and environment * **Source** is either `live` (the default for anything you send) or `simulation`. There is no filter for source in the list; use the badge to tell them apart. * **Environment** is a free-text label of up to 64 characters that you set when sending the trace (for OpenTelemetry, it is read from `deployment.environment.name` or `deployment.environment`). It defaults to `production`, which is why production traces show no environment badge. ### Search and filter Above the table: | Control | Behavior | | - | - | | **Search** | Matches the text you type against the trace name and the trace ID (case-insensitive, partial match). Placeholder: "Trace name or ID". | | **Status** | `All`, `OK` or `Error`. | | **Apply** | Applies both. Filters are kept in the page URL (`?q=` and `?status=`), so you can bookmark or share a filtered view. | ### Empty and error states | State | What you see | | - | - | | Project has no traces at all | **No traces yet**, with a ready-to-copy `curl` example for the ingest endpoint, a **Copy snippet** button, and **Get tracing key**, which opens the tracing keys page in Settings. | | Filters match nothing | **Nothing matches these filters**, naming the active search and status, with a **Clear filters** button. | | The list fails to load | An error panel with a retry button. If loading *more* rows fails, you get a toast and the rows already on screen stay. | Tracing keys are managed in Settings, and on AgentHub only your organization's super admins see the Tracing keys tab. If **Get tracing key** shows you nothing, ask one of them for a key. See [Tracing keys](/agenthub/settings/tracing-keys). ## Trace detail Click a trace name to open it. The page has four parts: the header, the stat cards, the **Run path**, and the evaluation section. ### Header * The root name (or trace ID) as the title, the trace ID under it, and the same provenance badges as the list. * **Run report** opens the [report](/agenthub/reports) for this trace. * A status badge, `OK` or `ERROR`. * **← Back to traces** returns to the list. ### Stat cards | Card | Value | | - | - | | **Duration** | The trace's duration. | | **Spans** | Span count. | | **Tokens** | `N in / N out`. | | **Cost** | Recorded cost in USD. | ### Run path "Every step the agent took, in the order it took them." A summary strip sits above three tabs. **Summary strip** | Stat | Meaning | | - | - | | **Wall clock** | First span start to last span end, measured from the spans themselves. | | **Steps** | Number of spans. | | **Recorded cost** | Sum of span costs. | | **Failed steps** | Number of spans with status `ERROR`. Only shown when there is at least one. | | **Time by step kind** | A stacked bar and percentages showing how the run's time split across span kinds. | **Expand all / Collapse all** opens or closes the detail panel on every step at once. When you open a failed trace, the first failing step is already expanded. A waterfall. Each row is one span, numbered in the order it ran, and indented under its parent. Bars are placed and sized on a shared clock, so a wide bar really did take longer. Gaps where nothing was running are marked with the idle time before the step, and the slowest step in the trace is flagged. A thin underline under each bar is that step's cost relative to the most expensive step in the trace. A legend lists each span kind with its count, a **Failed** entry when there are failures, and "Numbered rail = execution order". If the trace is marked `ERROR`, the waterfall shows where the run stopped, or notes when the trace is `ERROR` even though every recorded span is `OK`. Click any row to open its span detail (see below). The same steps read as a conversation: what the user asked, what the agent answered, and which connector (tool) it called in between. Steps with no recorded user message or completion say so ("No user message was recorded on this step"). A failed tool call reads as "The connector call failed", or as "Returned … — the agent got no data back" when an error reason was recorded. A flat table of every span: | Column | Value | | - | - | | **Span** | Span name, span ID, and model (when set). | | **Kind** | The span kind, color-coded. | | **Status** | `OK` or `ERROR`. | | **Latency** | Span duration. | | **Tokens** | Input / output. | | **Cost** | Span cost in USD. | #### Very large traces When a trace has more than 150 spans, the Timeline and Flow tabs first show only the key steps: the entry point, every failure, and the 20 slowest steps. A notice reads "Showing N of M steps" with a **Show all M steps** link. Payload previews are also hidden inside span details on these traces; the Table tab always lists every span. #### Span kinds | Kind | Label | Typical use | | - | - | - | | `llm` | LLM | A model call. | | `tool` | Tool | A tool or connector call made by the agent. | | `http` | HTTP | An outbound HTTP request. | | `db` | DB | A database or vector-store query. | | `function` | Function | Any other timed step. This is the default when no kind is sent. | | `evaluation` | Evaluation | An evaluation step recorded inside the trace, such as the acceptance check the Cost Calculator writes. | #### Span detail Opening a step shows: * **Started** (offset from the trace start), **Duration**, **Tokens** (in / out), **Cost**, **Status**, **Model** and **Span ID**. * A red **Failed:** line with the error reason, when the span recorded one in its output. * The payload, which depends on the kind: * **Model-call spans:** a collapsible **System prompt**, the **User** message, and the **Assistant** reply. If the model returned nothing, "The model returned an empty response." If nothing was recorded, "No prompt or completion recorded on this span." * **Tool spans:** the tool name, a `simulated` badge when the Cost Calculator simulated the tool, and the **Result**. * **Acceptance spans** (written by the Cost Calculator): a readable verdict with "Acceptance passed" or "Acceptance failed", the composite score, the scenario category, "Why this failed" reasons, each assertion with its measured value against its bar, and any other measured scores. * **Other input**: any input keys not already shown above. * **Output**: the output object, when it isn't already shown as a reply. * **Attributes**: every attribute you sent on the span. * **Raw JSON**: a collapsed section with the span's input, output and attributes exactly as stored. If a trace was ingested with no spans at all, the Run path shows **No spans recorded** instead. ### Evaluate this trace Below the run path is **Evaluate trace**. What it shows depends on your setup: | Situation | What you see | | - | - | | Your role can't run evaluations | "You need Editor access to run evaluations on this trace." | | No evaluator collections exist for this project | A prompt to create one, with **Go to Evaluation**. | | At least one collection exists | The trace ID, the most recently created collection and its rule count, and the most recently added model as the judge, then an **Evaluate this trace** button. | | The project has no models | An extra line: "Register at least one model under Models to run LLM-judge evaluators." | **Evaluate this trace** opens the **Your Evaluations** tab of [Evaluation](/agenthub/evaluation/overview) with the trace carried over: the test input box is prefilled with the trace's real first input and final answer, the judge is preselected, and the collection's evaluators are preselected (up to 25). You review the input, then press **Run**. That button does **not** run the collection against the stored trace. It starts a fresh model run seeded from the trace's text, scored by an LLM judge. Those results are not saved to the trace. Collection runs that score the stored trace itself (and appear under **Evaluation results** on this page) are created through the evaluate API. See [Evaluation runs](/agenthub/evaluation/runs) for the difference and how to start one. ### Evaluation results Under the panel, **Evaluation results** lists every saved collection run for this trace, newest first. Each card shows the collection name, the time, the judge model (or "offline only"), the run status, and the average score. A table lists each evaluator with its **Type** (`offline_deterministic` or `online_llm_judge`), **Score**, **Pass** and **Reason**. If a run failed, its error message is shown. With no saved runs, the section reads "No evaluations yet." ## Using traces elsewhere * **Reports.** Every trace has a report: **Report** in the list or **Run report** on the detail page. See [Reports](/agenthub/reports). * **Cost Calculator import.** In the Cost Calculator, **Import from traces** lets you pick up to 20 OpenTelemetry-collected traces at a time and analyze them as if they were scenarios, without re-running them through a model. Traces generated by CorePillar itself are not listed there. See [Cost Calculator](/agenthub/cost-calculator). * **Simulations.** Cost Calculator runs and their per-scenario traces are summarized on [Simulations](/agenthub/simulations). ## Limits | Item | Limit | | - | - | | Spans per ingest request | 1 to 500 | | Trace ID / span ID | 128 characters | | Root name / span name | 512 characters | | Environment | 64 characters | | Traces per page in the list | 50 | | Spans shown before key-steps mode | 150 | Traces are stored in the same database as the rest of the platform and ingestion is processed as it arrives, without a queue. It is sized for evaluation and debugging workloads, not for high-volume production telemetry. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.