> ## Documentation Index
> Fetch the complete documentation index at: https://docs.corepillar.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Trace format

> The exact JSON payload accepted by the trace ingest endpoint: every trace and span field, its type, default and limit, and how trace totals are computed.

This page is the field-by-field reference for the body of `POST /api/v1/traces/ingest`. One request carries **one trace** and between **1 and 500 spans** of it. If a trace has more than 500 spans, send it in several requests with the same `trace_id` (see [Sending a trace in several batches](#sending-a-trace-in-several-batches)).

Unknown fields are ignored and dropped, so you can't add your own top-level fields. Put custom data in a span's `attributes` object, or in `tags`.

## Trace object

<ParamField body="trace_id" type="string" required>
  Your identifier for this run. 1 to 128 characters. Unique per project: sending the same `trace_id` again adds spans to (or updates spans in) the existing trace rather than creating a second one. Any string works, such as a UUID, your request id, or an OpenTelemetry-style hex id. If you use characters like `/` or `#`, URL-encode the id when you put it in an API path.
</ParamField>

<ParamField body="root_name" type="string" default="&#x22;&#x22;">
  The label shown for the trace in the Traces list, such as `agent_run` or `support_bot.reply`. Up to 512 characters.

  <Warning>
    Every request for a trace overwrites its stored `root_name`. If you send a trace in several batches and leave `root_name` out of a later batch, the name is reset to an empty string. Send the same `root_name` on every batch.
  </Warning>
</ParamField>

<ParamField body="status" type="&#x22;OK&#x22; | &#x22;ERROR&#x22;">
  Optional. It only affects the `status` echoed in this request's response. The **stored** trace status is always derived from the spans: the trace is `ERROR` if any span ever stored for it has `status: "ERROR"`, otherwise `OK`. To mark a run as failed, set `status: "ERROR"` on the span that failed.
</ParamField>

<ParamField body="environment" type="string" default="&#x22;production&#x22;">
  Free text, up to 64 characters, such as `production`, `staging` or `development`. Overwritten by each batch for the same trace.
</ParamField>

<ParamField body="source" type="&#x22;live&#x22; | &#x22;simulation&#x22;" default="&#x22;live&#x22;">
  Where the trace came from. Use `live` for real traffic. `simulation` is what AgentHub's own Cost Calculator and simulation runs write; you'd normally leave this at the default. Overwritten by each batch.
</ParamField>

<ParamField body="started_at" type="string (ISO 8601, UTC)">
  Optional. When omitted, the earliest span `started_at` is used. Either way, once the spans are stored the trace start is restated as the earliest `started_at` of all its stored spans, so in practice the spans decide it.
</ParamField>

<ParamField body="tags" type="string[]" default="[]">
  Labels for filtering and grouping, each up to 128 characters. Overwritten by each batch for the same trace, so resend the full list every time.
</ParamField>

<ParamField body="spans" type="Span[]" required>
  The spans in this batch. At least 1, at most 500.
</ParamField>

## Span object

<ParamField body="span_id" type="string" required>
  Unique within the trace. 1 to 128 characters. Sending a span with a `span_id` that already exists in the trace **replaces** that span completely (every field below is overwritten), which is what makes retries safe.
</ParamField>

<ParamField body="parent_span_id" type="string | null">
  The `span_id` of the span this one ran inside. Up to 128 characters. Leave it out or set it to `null` for the root span. The parent doesn't have to be in the same request, or to have arrived yet. See [Parent and child spans](#parent-and-child-spans).
</ParamField>

<ParamField body="name" type="string" required>
  What the step was, such as `chat.completions.create`, `search_docs` or `GET /v1/orders`. 1 to 512 characters.
</ParamField>

<ParamField body="kind" type="string" default="&#x22;function&#x22;">
  One of the [span kinds](#span-kinds) below. The kind decides which trace totals the span counts towards.
</ParamField>

<ParamField body="model" type="string | null">
  The model identifier for `llm` spans, such as `gpt-4o-mini` or `claude-sonnet-4-5`. Up to 256 characters. Stored as you send it.
</ParamField>

<ParamField body="status" type="&#x22;OK&#x22; | &#x22;ERROR&#x22;" default="&#x22;OK&#x22;">
  Whether this step succeeded. Any `ERROR` span makes the whole trace `ERROR`, and that sticks: later batches of `OK` spans don't clear it.
</ParamField>

<ParamField body="tokens_in" type="integer" default="0">
  Input (prompt) tokens. Whole number, 0 or more.
</ParamField>

<ParamField body="tokens_out" type="integer" default="0">
  Output (completion) tokens. Whole number, 0 or more.
</ParamField>

<ParamField body="cost_usd" type="number" default="0">
  What this step cost, in US dollars. 0 or more. Stored with 6 decimal places, so anything below a millionth of a dollar is rounded away. AgentHub does **not** price JSON spans for you: if you leave this at 0, the span and the trace show zero cost. Compute it from your provider's pricing and send it.
</ParamField>

<ParamField body="latency_ms" type="integer">
  How long the step took, in milliseconds. Whole number, 0 or more. When omitted or 0, it's computed as `ended_at` minus `started_at`. When you do send it, it wins over the timestamps.
</ParamField>

<ParamField body="input" type="object" default="{}">
  What went into the step. Must be a JSON **object** (not a string, array or number). See [Input and output conventions](#input-and-output-conventions).
</ParamField>

<ParamField body="output" type="object" default="{}">
  What came out of the step. Must be a JSON object.
</ParamField>

<ParamField body="attributes" type="object" default="{}">
  Any extra key/value data you want to keep with the span, such as temperature, tool arguments, user id or retry count. Must be a JSON object.
</ParamField>

<ParamField body="started_at" type="string (ISO 8601, UTC)" required>
  When the step started.
</ParamField>

<ParamField body="ended_at" type="string (ISO 8601, UTC)" required>
  When the step finished.
</ParamField>

## Enums

### Span kinds

| `kind` | Use it for | Counts towards |
| - | - | - |
| `llm` | A call to a language model. | **LLM calls** and **LLM time** |
| `tool` | A tool or function the agent chose to call. | **Tool calls** |
| `http` | An outbound HTTP request. | **HTTP time** |
| `db` | A database or vector-store query. | Span count only |
| `function` | Any other unit of work, including the root span. This is the default. | Span count only |
| `evaluation` | An evaluation or scoring step. | Span count only |

Any other value is rejected with `400 VALIDATION_ERROR`.

### Status

`OK` or `ERROR`, uppercase, for both traces and spans.

### Source

`live` (default) or `simulation`.

### Environment

Not an enum. Any string up to 64 characters; `production` when omitted.

## Timestamps

`started_at` and `ended_at` (and the optional trace `started_at`) must be ISO 8601 date-times **in UTC with a trailing `Z`**. Any number of fractional-second digits is accepted.

| Value | Accepted |
| - | - |
| `2026-03-17T10:00:00Z` | Yes |
| `2026-03-17T10:00:00.123Z` | Yes |
| `2026-03-17T10:00:00.123456Z` | Yes |
| `2026-03-17T10:00:00+00:00` | No (offset instead of `Z`) |
| `2026-03-17T12:00:00+02:00` | No |
| `2026-03-17 10:00:00Z` | No (space instead of `T`) |
| `1773741600000` | No (epoch number) |

JavaScript's `Date.prototype.toISOString()` produces the right format. Python's `datetime.isoformat()` does not (it writes `+00:00`); format with `strftime("%Y-%m-%dT%H:%M:%S.%fZ")` on a UTC datetime instead.

## Parent and child spans

Spans form a tree through `parent_span_id`:

```mermaid theme={null}
graph TD
  A["root · agent_run · function"] --> B["llm-1 · chat.completions.create · llm"]
  A --> C["tool-1 · search_docs · tool"]
  C --> D["http-1 · GET /search · http"]
  A --> E["llm-2 · chat.completions.create · llm"]
```

* The root span has no `parent_span_id`. A trace normally has exactly one.
* A child's `parent_span_id` must equal its parent's `span_id` within the same trace.
* AgentHub doesn't check that the parent exists when you send a child. Children can arrive before their parent, in a different request, and they're linked when the trace is displayed. This is what lets you send spans as they finish rather than all at the end.
* Spans are listed in `started_at` order inside a trace.

## Input and output conventions

`input` and `output` accept any JSON object, but the trace view and the evaluators look for a few keys first. Using them makes traces easier to read and evaluate:

| Object | Key | Meaning |
| - | - | - |
| `input` | `system` | The system prompt. |
| `input` | `user` | The user message for this step. |
| `input` | `messages` | The full message list, as `[{ "role": "...", "content": "..." }]`. |
| `output` | `content` | The model's or step's text answer. |
| `output` | `finish_reason` | Why generation stopped, such as `stop` or `length`. |
| `output` | `error` | An error message to show for a failed span. |

These are the same keys AgentHub writes when it converts OpenTelemetry spans, so JSON and OTLP traces look the same in the UI.

## How trace totals are computed

You don't send trace totals. After every request, AgentHub recomputes them from **all spans stored for the trace**, not just the ones in the request. Totals are therefore correct whether you send a trace in one request or many, in order or out of order, and when you resend spans.

| Trace field | How it's computed |
| - | - |
| `started_at` | Earliest span `started_at`. |
| `duration_ms` | Wall-clock time from the earliest span start to the latest span end. A span's end is the later of its `ended_at` and `started_at + latency_ms`. Never less than the longest single span's `latency_ms`. |
| `llm_time_ms` | Wall-clock time covered by `llm` spans, with overlapping spans counted once. Capped at `duration_ms`, so two parallel 2-second model calls add 2 seconds, not 4. |
| `http_time_ms` | The same calculation over `http` spans. |
| `span_count` | Number of spans stored for the trace. |
| `llm_call_count` | Number of `llm` spans. |
| `tool_call_count` | Number of `tool` spans. |
| `tokens_in` / `tokens_out` | Sum over all spans. |
| `cost_usd` | Sum of span `cost_usd` over all spans, rounded to 6 decimals. |
| `status` | `ERROR` if any stored span is `ERROR`, otherwise `OK`. |

Token and cost sums include every span kind. If you record a model call's tokens on both an `llm` span and its parent `function` span, they're counted twice; put them only on the span that made the call.

## Sending a trace in several batches

You can split one trace across requests, for example to stay under the 500-span limit or to stream spans as steps finish:

* Use the same `trace_id` on every request.
* Each span only needs to be sent once, in any batch. Resending a `span_id` replaces that span.
* Resend `root_name`, `environment`, `source` and `tags` on every batch. Each request overwrites them on the trace.
* Use a **different** `Idempotency-Key` per batch (or none). Reusing one key for two different batches returns the first batch's cached response and silently drops the second. See [Idempotency and retries](/agenthub/developers/idempotency-and-retries).

## Validation errors

A payload that doesn't match the schema returns `400 VALIDATION_ERROR` with the problems in `error.details.issues`:

```json theme={null}
{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid trace payload.",
    "details": {
      "issues": {
        "formErrors": [],
        "fieldErrors": {
          "spans": ["Invalid datetime"]
        }
      }
    }
  }
}
```

`fieldErrors` is keyed by the top-level field name. A problem inside any span is reported under `spans` without the span's index, so check every span when you see it. Nothing from a rejected request is stored.

## Full annotated example

A support agent that looks up an order, calls a model twice, and has one failed tool call. Comments are for explanation only; strip them before sending, because JSON doesn't allow comments.

```jsonc theme={null}
{
  "trace_id": "support-7f3a9c21",          // your id, unique per project
  "root_name": "support_agent.reply",     // label in the Traces list; resend on every batch
  "environment": "production",            // any string, default "production"
  "source": "live",                       // "live" (default) or "simulation"
  "tags": ["support", "tier:pro"],        // replaces the trace's tags on every batch
  "spans": [
    {
      "span_id": "root",                  // no parent_span_id: this is the root
      "name": "support_agent.reply",
      "kind": "function",
      "input": { "user": "Where is order 1042?" },
      "output": { "content": "Order 1042 shipped on 12 March and arrives Friday." },
      "attributes": { "customer_tier": "pro" },
      "started_at": "2026-03-17T10:00:00.000Z",
      "ended_at": "2026-03-17T10:00:03.400Z"
    },
    {
      "span_id": "plan",
      "parent_span_id": "root",
      "name": "chat.completions.create",
      "kind": "llm",                      // counts as an LLM call and towards LLM time
      "model": "gpt-4o-mini",
      "tokens_in": 410,
      "tokens_out": 38,
      "cost_usd": 0.000084,               // you compute cost; AgentHub doesn't price JSON spans
      "latency_ms": 720,
      "input": {
        "system": "You are a helpful support agent.",
        "user": "Where is order 1042?"
      },
      "output": { "content": "Call lookup_order with id 1042.", "finish_reason": "tool_calls" },
      "attributes": { "temperature": 0.2 },
      "started_at": "2026-03-17T10:00:00.050Z",
      "ended_at": "2026-03-17T10:00:00.770Z"
    },
    {
      "span_id": "lookup-1",
      "parent_span_id": "root",
      "name": "lookup_order",
      "kind": "tool",                     // counts as a tool call
      "status": "ERROR",                  // makes the whole trace ERROR, permanently
      "input": { "order_id": "1042" },
      "output": { "error": "orders-api timed out after 1000 ms" },
      "started_at": "2026-03-17T10:00:00.800Z",
      "ended_at": "2026-03-17T10:00:01.800Z"   // latency_ms omitted: computed as 1000
    },
    {
      "span_id": "lookup-2",
      "parent_span_id": "root",
      "name": "lookup_order",
      "kind": "tool",
      "input": { "order_id": "1042", "attempt": 2 },
      "output": { "status": "shipped", "shipped_at": "2026-03-12" },
      "started_at": "2026-03-17T10:00:01.850Z",
      "ended_at": "2026-03-17T10:00:02.150Z"
    },
    {
      "span_id": "lookup-2-http",
      "parent_span_id": "lookup-2",       // grandchild of the root
      "name": "GET /orders/1042",
      "kind": "http",                     // counts towards HTTP time
      "attributes": { "http.status_code": 200 },
      "started_at": "2026-03-17T10:00:01.860Z",
      "ended_at": "2026-03-17T10:00:02.140Z"
    },
    {
      "span_id": "answer",
      "parent_span_id": "root",
      "name": "chat.completions.create",
      "kind": "llm",
      "model": "gpt-4o-mini",
      "tokens_in": 520,
      "tokens_out": 61,
      "cost_usd": 0.000115,
      "latency_ms": 1180,
      "input": { "user": "Where is order 1042?" },
      "output": {
        "content": "Order 1042 shipped on 12 March and arrives Friday.",
        "finish_reason": "stop"
      },
      "started_at": "2026-03-17T10:00:02.200Z",
      "ended_at": "2026-03-17T10:00:03.380Z"
    }
  ]
}
```

After this is ingested, the trace shows:

| Field | Value | Why |
| - | - | - |
| `status` | `ERROR` | `lookup-1` failed. |
| `duration_ms` | 3400 | First span start (10:00:00.000) to last span end (10:00:03.400). |
| `llm_time_ms` | 1900 | `plan` (720) + `answer` (1180), which don't overlap. |
| `http_time_ms` | 280 | `lookup-2-http`. |
| `span_count` | 6 | |
| `llm_call_count` | 2 | |
| `tool_call_count` | 2 | |
| `tokens_in` / `tokens_out` | 930 / 99 | Sum over all spans. |
| `cost_usd` | 0.000199 | Sum over all spans. |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.