> ## Documentation Index
> Fetch the complete documentation index at: https://docs.corepillar.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Idempotency and retries

> How the ingest endpoint deduplicates requests, tolerates out-of-order spans, and which errors you should retry.

Networks fail, workers restart, and queues redeliver. The ingest endpoint is built so that sending the same data twice, or sending it in the wrong order, never corrupts a trace. This page explains the guarantees and how to write a client that relies on them.

## The short version

* A `202 Accepted` response means the batch is stored. There's no background job to wait for.
* Resending a span with the same `trace_id` and `span_id` **replaces** it. It never creates a duplicate.
* Trace totals (duration, LLM time, counts, tokens, cost, status) are recomputed from the stored spans after every request, so retries and multi-batch traces never double-count.
* An optional `Idempotency-Key` header makes an exact retry return the original response without reprocessing anything.
* Retry network errors and `5xx` responses with backoff. Don't retry `400` or `401` until you've fixed the request.

## The Idempotency-Key header

```http theme={null}
POST /api/v1/traces/ingest HTTP/1.1
Host: pillarforge.corepillar.ai
Authorization: Bearer tk_live_...
Content-Type: application/json
Idempotency-Key: support-7f3a9c21-batch-1
```

| Behavior | Detail |
| - | - |
| Required | No. Leading and trailing whitespace is trimmed; an empty value is treated as no key. |
| Format | Any string. Use something you can regenerate on retry, such as `<trace_id>` for a single-batch trace or `<trace_id>-batch-<n>` for multi-batch traces. |
| Scope | Per project. The same key value used with keys for two different projects is two different keys. |
| When it's recorded | Only after a batch is stored successfully. A request that fails (bad payload, bad key, server error) leaves nothing behind, so you can retry it with the same key. |
| Replay | A later request with the same key returns the **first** request's response body with `202`, and does nothing else. The new body isn't read beyond validation, even if it's different. |
| Order of checks | The tracing key is authenticated and the body is validated before the key is looked up. A replay with a malformed body still gets `400`. |
| Expiry | Keys don't expire. A key stays bound to its first response for good. |

<Warning>
  Never reuse an idempotency key for different data. If you send batch 2 of a trace with batch 1's key, you get batch 1's response back and batch 2 is **silently dropped**. Give each distinct batch its own key.
</Warning>

### Do I need it?

Not for correctness. Because spans are stored by `(trace_id, span_id)`, a retry without a key rewrites the same rows with the same values and the totals come out identical. The key is still worth sending because:

* A replay is cheaper: nothing is reprocessed.
* It protects the trace-level fields. Without a key, a stale retry that arrives after a newer batch rewrites `root_name`, `environment`, `source` and `tags` with the stale values (spans and totals stay correct either way).

### Concurrent duplicates

The key is recorded after the batch is stored, so two copies of the same request that arrive at the same moment may both be processed. That's harmless for the reasons above: both write the same spans, and the totals are restated from storage.

## Response

Every successful ingest returns `202 Accepted` with the [standard envelope](/api-reference/introduction#response-envelope):

```json theme={null}
{
  "data": {
    "trace_id": "support-7f3a9c21",
    "project_id": "6f1c2a9e-3b0d-4c55-9a51-2f7d8e4b1c3a",
    "span_count": 6,
    "status": "ERROR"
  }
}
```

| Field | Meaning |
| - | - |
| `trace_id` | The trace the batch was written to. |
| `project_id` | The project, from your tracing key. |
| `span_count` | Spans in **this batch**, not the trace's running total. |
| `status` | `ERROR` if this batch's trace-level `status` was `ERROR` or any span in it was `ERROR`; otherwise `OK`. The stored trace status can differ: it's `ERROR` if any span ever stored for the trace is `ERROR`. |

Despite the `202`, ingestion is synchronous. The spans and the recomputed totals are written before the response is sent, so a `GET` of the trace right after returns the new data.

## Out-of-order and partial delivery

You don't have to send a trace in one piece or in order:

* **Children before parents.** A span's `parent_span_id` isn't checked on arrival. The tree is assembled from the stored spans when the trace is displayed, so a parent that arrives later is linked in.
* **Batches in any order.** The trace's `started_at` is always the earliest stored span start and `duration_ms` the full extent, whichever batch arrives first.
* **Errors are sticky.** Once any span of a trace is stored with `status: "ERROR"`, the trace stays `ERROR`. A later batch of successful spans can't clear it, and neither can resending the trace with a trace-level `status: "OK"`. To clear it, resend the failed span itself with `status: "OK"`.
* **Last write wins per span.** If two versions of the same `span_id` arrive, whichever is stored last is kept, in full.
* **Last write wins for trace labels.** `root_name`, `environment`, `source` and `tags` take the values from the most recent batch. Send the same values on every batch.

## Retry guidance

| Response | Retry? | What to do |
| - | - | - |
| `202` | No | Done. |
| Network error, timeout, connection reset | Yes | Retry with the same body and the same `Idempotency-Key`. |
| `500 INTERNAL_ERROR` | Yes | Retry with backoff. Nothing from the failed request was cached. If it persists, contact your CorePillar administrator. |
| `502`, `503`, `504` from a proxy or load balancer | Yes | Retry with backoff. |
| `429` | Yes | Back off and retry. The ingest endpoint doesn't apply its own rate limit today, but infrastructure in front of it can still return `429`. |
| `400 VALIDATION_ERROR` | No | The payload is wrong. Fix it using `error.details` (see [Trace format](/agenthub/developers/trace-format#validation-errors)). Retrying the same body fails the same way. |
| `401 UNAUTHORIZED` | No | The key is missing, malformed or revoked. Fix the key first. |

Use exponential backoff with jitter, cap the number of attempts, and keep the body and `Idempotency-Key` identical across attempts. For example:

<CodeGroup>
  ```python Python theme={null}
  import random
  import time

  import requests

  RETRYABLE = {429, 500, 502, 503, 504}


  def ingest(trace: dict, tracing_key: str, idempotency_key: str, attempts: int = 5) -> dict:
      for attempt in range(attempts):
          try:
              response = requests.post(
                  "https://pillarforge.corepillar.ai/api/v1/traces/ingest",
                  json=trace,
                  headers={
                      "Authorization": f"Bearer {tracing_key}",
                      "Idempotency-Key": idempotency_key,
                  },
                  timeout=10,
              )
          except requests.RequestException:
              response = None

          if response is not None and response.status_code == 202:
              return response.json()["data"]
          if response is not None and response.status_code not in RETRYABLE:
              # 400 / 401: retrying won't help.
              raise RuntimeError(f"ingest rejected: {response.status_code} {response.text}")

          time.sleep(min(30, 2 ** attempt) + random.random())

      raise RuntimeError("ingest failed after retries")
  ```

  ```typescript TypeScript theme={null}
  const RETRYABLE = new Set([429, 500, 502, 503, 504]);

  export async function ingest(
    trace: unknown,
    tracingKey: string,
    idempotencyKey: string,
    attempts = 5,
  ) {
    for (let attempt = 0; attempt < attempts; attempt++) {
      let response: Response | null = null;
      try {
        response = await fetch("https://pillarforge.corepillar.ai/api/v1/traces/ingest", {
          method: "POST",
          headers: {
            Authorization: `Bearer ${tracingKey}`,
            "Content-Type": "application/json",
            "Idempotency-Key": idempotencyKey,
          },
          body: JSON.stringify(trace),
          signal: AbortSignal.timeout(10_000),
        });
      } catch {
        response = null; // network error or timeout
      }

      if (response?.status === 202) return (await response.json()).data;
      if (response && !RETRYABLE.has(response.status)) {
        // 400 / 401: retrying won't help.
        throw new Error(`ingest rejected: ${response.status} ${await response.text()}`);
      }

      const delayMs = Math.min(30_000, 2 ** attempt * 1000) + Math.random() * 1000;
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    }
    throw new Error("ingest failed after retries");
  }
  ```
</CodeGroup>

## Throughput and limits

| Limit | Value |
| - | - |
| Traces per request | 1 |
| Spans per request | 1 to 500 |
| Request rate | No per-key rate limit is enforced by AgentHub today. |
| Trace size | No hard cap on spans per trace for JSON ingestion; send more than 500 as several batches with the same `trace_id`. |

Each request is processed synchronously, so a large backlog is faster sent with a few requests in parallel than one at a time. Keep the concurrency modest (a handful of requests in flight), and don't let tracing block your agent's own work: send traces from a background task or queue, and drop or buffer them if AgentHub is unreachable rather than failing the user's request.

## OpenTelemetry exports

The OTLP endpoint (`POST /api/v1/traces`) doesn't use `Idempotency-Key`, because OpenTelemetry exporters have no such concept. It relies on the same span-level deduplication instead: a retried export rewrites the same spans. See [OpenTelemetry (OTLP)](/agenthub/developers/opentelemetry).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.