Idempotency and retries - CorePillar Docs

Networks fail, workers restart, and queues redeliver. The ingest endpoint is built so that sending the same data twice, or sending it in the wrong order, never corrupts a trace. This page explains the guarantees and how to write a client that relies on them.

The short version

A 202 Accepted response means the batch is stored. There’s no background job to wait for.
Resending a span with the same trace_id and span_id replaces it. It never creates a duplicate.
Trace totals (duration, LLM time, counts, tokens, cost, status) are recomputed from the stored spans after every request, so retries and multi-batch traces never double-count.
An optional Idempotency-Key header makes an exact retry return the original response without reprocessing anything.
Retry network errors and 5xx responses with backoff. Don’t retry 400 or 401 until you’ve fixed the request.

The Idempotency-Key header

POST /api/v1/traces/ingest HTTP/1.1
Host: pillarforge.corepillar.ai
Authorization: Bearer tk_live_...
Content-Type: application/json
Idempotency-Key: support-7f3a9c21-batch-1

Behavior	Detail
Required	No. Leading and trailing whitespace is trimmed; an empty value is treated as no key.
Format	Any string. Use something you can regenerate on retry, such as `<trace_id>` for a single-batch trace or `<trace_id>-batch-<n>` for multi-batch traces.
Scope	Per project. The same key value used with keys for two different projects is two different keys.
When it’s recorded	Only after a batch is stored successfully. A request that fails (bad payload, bad key, server error) leaves nothing behind, so you can retry it with the same key.
Replay	A later request with the same key returns the first request’s response body with `202`, and does nothing else. The new body isn’t read beyond validation, even if it’s different.
Order of checks	The tracing key is authenticated and the body is validated before the key is looked up. A replay with a malformed body still gets `400`.
Expiry	Keys don’t expire. A key stays bound to its first response for good.

Never reuse an idempotency key for different data. If you send batch 2 of a trace with batch 1’s key, you get batch 1’s response back and batch 2 is silently dropped. Give each distinct batch its own key.

Do I need it?

Not for correctness. Because spans are stored by (trace_id, span_id), a retry without a key rewrites the same rows with the same values and the totals come out identical. The key is still worth sending because:

A replay is cheaper: nothing is reprocessed.
It protects the trace-level fields. Without a key, a stale retry that arrives after a newer batch rewrites root_name, environment, source and tags with the stale values (spans and totals stay correct either way).

Concurrent duplicates

The key is recorded after the batch is stored, so two copies of the same request that arrive at the same moment may both be processed. That’s harmless for the reasons above: both write the same spans, and the totals are restated from storage.

Response

Every successful ingest returns 202 Accepted with the standard envelope:

{
  "data": {
    "trace_id": "support-7f3a9c21",
    "project_id": "6f1c2a9e-3b0d-4c55-9a51-2f7d8e4b1c3a",
    "span_count": 6,
    "status": "ERROR"
  }
}

Field	Meaning
`trace_id`	The trace the batch was written to.
`project_id`	The project, from your tracing key.
`span_count`	Spans in this batch, not the trace’s running total.
`status`	`ERROR` if this batch’s trace-level `status` was `ERROR` or any span in it was `ERROR`; otherwise `OK`. The stored trace status can differ: it’s `ERROR` if any span ever stored for the trace is `ERROR`.

Despite the 202, ingestion is synchronous. The spans and the recomputed totals are written before the response is sent, so a GET of the trace right after returns the new data.

Out-of-order and partial delivery

You don’t have to send a trace in one piece or in order:

Children before parents. A span’s parent_span_id isn’t checked on arrival. The tree is assembled from the stored spans when the trace is displayed, so a parent that arrives later is linked in.
Batches in any order. The trace’s started_at is always the earliest stored span start and duration_ms the full extent, whichever batch arrives first.
Errors are sticky. Once any span of a trace is stored with status: "ERROR", the trace stays ERROR. A later batch of successful spans can’t clear it, and neither can resending the trace with a trace-level status: "OK". To clear it, resend the failed span itself with status: "OK".
Last write wins per span. If two versions of the same span_id arrive, whichever is stored last is kept, in full.
Last write wins for trace labels. root_name, environment, source and tags take the values from the most recent batch. Send the same values on every batch.

Retry guidance

Response	Retry?	What to do
`202`	No	Done.
Network error, timeout, connection reset	Yes	Retry with the same body and the same `Idempotency-Key`.
`500 INTERNAL_ERROR`	Yes	Retry with backoff. Nothing from the failed request was cached. If it persists, contact your CorePillar administrator.
`502`, `503`, `504` from a proxy or load balancer	Yes	Retry with backoff.
`429`	Yes	Back off and retry. The ingest endpoint doesn’t apply its own rate limit today, but infrastructure in front of it can still return `429`.
`400 VALIDATION_ERROR`	No	The payload is wrong. Fix it using `error.details` (see Trace format). Retrying the same body fails the same way.
`401 UNAUTHORIZED`	No	The key is missing, malformed or revoked. Fix the key first.

Use exponential backoff with jitter, cap the number of attempts, and keep the body and Idempotency-Key identical across attempts. For example:

import random
import time

import requests

RETRYABLE = {429, 500, 502, 503, 504}


def ingest(trace: dict, tracing_key: str, idempotency_key: str, attempts: int = 5) -> dict:
    for attempt in range(attempts):
        try:
            response = requests.post(
                "https://pillarforge.corepillar.ai/api/v1/traces/ingest",
                json=trace,
                headers={
                    "Authorization": f"Bearer {tracing_key}",
                    "Idempotency-Key": idempotency_key,
                },
                timeout=10,
            )
        except requests.RequestException:
            response = None

        if response is not None and response.status_code == 202:
            return response.json()["data"]
        if response is not None and response.status_code not in RETRYABLE:
            # 400 / 401: retrying won't help.
            raise RuntimeError(f"ingest rejected: {response.status_code} {response.text}")

        time.sleep(min(30, 2 ** attempt) + random.random())

    raise RuntimeError("ingest failed after retries")

const RETRYABLE = new Set([429, 500, 502, 503, 504]);

export async function ingest(
  trace: unknown,
  tracingKey: string,
  idempotencyKey: string,
  attempts = 5,
) {
  for (let attempt = 0; attempt < attempts; attempt++) {
    let response: Response | null = null;
    try {
      response = await fetch("https://pillarforge.corepillar.ai/api/v1/traces/ingest", {
        method: "POST",
        headers: {
          Authorization: `Bearer ${tracingKey}`,
          "Content-Type": "application/json",
          "Idempotency-Key": idempotencyKey,
        },
        body: JSON.stringify(trace),
        signal: AbortSignal.timeout(10_000),
      });
    } catch {
      response = null; // network error or timeout
    }

    if (response?.status === 202) return (await response.json()).data;
    if (response && !RETRYABLE.has(response.status)) {
      // 400 / 401: retrying won't help.
      throw new Error(`ingest rejected: ${response.status} ${await response.text()}`);
    }

    const delayMs = Math.min(30_000, 2 ** attempt * 1000) + Math.random() * 1000;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }
  throw new Error("ingest failed after retries");
}

Throughput and limits

Limit	Value
Traces per request	1
Spans per request	1 to 500
Request rate	No per-key rate limit is enforced by AgentHub today.
Trace size	No hard cap on spans per trace for JSON ingestion; send more than 500 as several batches with the same `trace_id`.

Each request is processed synchronously, so a large backlog is faster sent with a few requests in parallel than one at a time. Keep the concurrency modest (a handful of requests in flight), and don’t let tracing block your agent’s own work: send traces from a background task or queue, and drop or buffer them if AgentHub is unreachable rather than failing the user’s request.

OpenTelemetry exports

The OTLP endpoint (POST /api/v1/traces) doesn’t use Idempotency-Key, because OpenTelemetry exporters have no such concept. It relies on the same span-level deduplication instead: a retried export rewrites the same spans. See OpenTelemetry (OTLP).

​The short version

​The Idempotency-Key header

​Do I need it?

​Concurrent duplicates

​Response

​Out-of-order and partial delivery

​Retry guidance

​Throughput and limits

​OpenTelemetry exports

The short version

The Idempotency-Key header

Do I need it?

Concurrent duplicates

Response

Out-of-order and partial delivery

Retry guidance

Throughput and limits

OpenTelemetry exports