jwuthri-cloud/tracely-ai/tracely

Tracely

Trace-native CI/CD for AI agents. One loop:

production trace → failure detection → regression test → CI gate

The trace is the source of truth. Evaluators, failure clusters, regression cases, gates and trends are all derived from it — there are no hand-authored datasets. Docs: <https://doc.tracely-ai.com>.

Pick the path first

Do not start writing spans. Ask what the code already looks like, then pick:

Default to automatic. Manual spans are the escape hatch, not the starting point, and the two compose — manual spans nest inside auto-instrumented traces in the same tree.

Connect

import tracely_sdk as tracely   # pip install "tracelyai-sdk[openai]"  ([anthropic] [langchain] [all])
tracely.init(
    endpoint="http://localhost:8000",   # hosted: https://api.tracely-ai.com
    api_key="tracely_dev_key",          # an ingest key — Settings → API keys. The key IS the workspace.
    service_name="support-agent",
    env="prod",                         # prod | staging | ci | dev — the gating axis
    instrument="auto",
)

Python ≥ 3.10. Import name is tracely_sdk; the CLI is tracely. init() is idempotent — call it once at startup. Prefer os.environ for endpoint/api_key in real code; never inline a key.

The 90% path

with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
    client.chat.completions.create(model="gpt-4o", messages=[...])   # traced, no span code

trace() opens no span — it stamps run context (agent, conversation, turn, user, env, arbitrary metadata) onto every span inside it, including the ones the instrumentor created. It also works as a decorator on sync or async functions.

Six rules that decide whether Tracely actually works

Getting these wrong doesn't error — it silently produces a useless workspace. Check them in every review of someone's instrumentation.

1. The agent name is declared, never inferred. It's the dimension gates, clusters, scenarios

and trends group by, and Tracely reads exactly one attribute for it: tracely.agent.id — set by init(service_name=…), overridden by init(agent=…) for the app or trace(agent=…)/agent(...) per run. A framework's own gen_ai.agent.name is deliberately ignored: a harness stamps one on every sub-agent it spins up, which would register dozens of agents nobody chose. Name neither and every trace in the workspace lands under a single default agent. One codebase serving many customers → `trace(tenant=customer_id)`: each tenant is its own Agent (endpoint, scenarios, gate, clusters, cases; the traces list filters by it) and agent= stays the per-span label.

1. `conversation=` is what makes a conversation. Each turn is its own trace; passing the same

conversation id to every turn is the only thing that threads them. Without it, a 12-turn support thread is 12 unrelated rows, and every conversation-level evaluator has nothing to grade.

1. `env` is the gating axis. prod failures become regression cases; ci traces are what the

gate grades. Tagging CI runs as prod poisons the case pool with test data.

1. Errors are the failure signal. A failed tool must be marked — tracely.error(span, msg), an

exception inside @observe, or an OTel ERROR status. Detection, clustering and the gate all key off it. A tool that returns {"error": "..."} as a successful span is invisible.

1. `flush()` before the process exits. Scripts, Lambdas, CLI runs and tests lose their last

spans otherwise. Long-lived servers don't need it.

1. If Tracely calls your agent, honour the `traceparent` header. Scenarios and simulate mint

the trace id and send it. Ignore it and the gate sees only text in / text out — blind to your tool calls, so tool expectations report SKIP instead of grading. See references/ci-gate.md.

Two more that bite on specific stacks:

• Streaming OpenAI calls need stream_options={"include_usage": True} or token counts (and

therefore cost) are lost.

• LangChain + a provider instrumentor double-traces. Under "auto" LangChain wins and the provider

instrumentors are skipped; pass an explicit list only if you also make direct provider calls.

Your own spans — @observe

The cheapest way to see business logic. Args → input, return → output, exceptions → level=ERROR, auto-nested via OTel context with no parent wiring.

@tracely.observe(as_type="tool")
def get_weather(city: str) -> dict:
    return {"city": city, "tempF": 64}

as_type ∈ agent · delegate · generation · tool · skill · chain · retriever · thinking · embedding · guardrail · span. Decorating tools also makes them hermetically replayable in CI for free.

Manual spans

When the auto path can't express it. Every helper is a context manager; nesting builds the tree.

with tracely.agent("support-agent", version="v4", conversation="conv-1", turn=0) as a:
    tracely.set_io(a, input=user_msg, output=answer)
    with tracely.tool("get_order", agent="support-agent") as t:
        try:
            tracely.set_io(t, input={"order_id": oid}, output=get_order(oid))
        except Exception as e:
            tracely.error(t, str(e))

Two types are worth naming even on an otherwise-automatic agent, because they're the shape a multi-agent system fails in:

• skill(name, version=…) → SKILL, a named capability the agent chose to run (a refund flow, an

escalation playbook, a loaded agent-skill file). Makes "which skill did this?" a filter and a failure cluster instead of a tree shape you have to infer.

• delegate(to, agent=…, task=…) → DELEGATE, the handover itself. Open the callee's agent(...)

inside it and a judge can grade the routing decision separately from the work.

Full cookbook — every span type, set_io / set_usage / set_metadata / set_state / set_agents, RAG pipelines, multi-agent handoffs, multimodal content, structured I/O rendering: `references/manual.md`.

Evaluators are columns

An evaluator is a column on the trace table, graded automatically as traces land. Two kinds: structural (deterministic, no model, no cost) and llm_judge (a rubric prompt).

A trace fails iff a non-advisory evaluator scored it FAIL. Mark subjective quality judges advisory: true so they show their verdict without flipping the roll-up.

Reach for structural checks before a judge — run_outcome, tool_success, tool_consistency, latency, required_tools cost nothing and never flake. Levels, sequential grading, @VARIABLE advanced templates, targeting and sampling: `references/evaluators.md`.

Gate the PR

Three ways, by what CI can reach:

All exit 0 PASS / 1 FAIL / 2 never-got-an-answer, and post a GitHub commit status + PR comment. A composite action ships at .github/actions/tracely-gate/. Details, scenario authoring, red-team runs and hermetic replay: `references/ci-gate.md`.

Drive it from the editor (MCP)

Every backend serves MCP at /mcp — read traces, inspect clusters, create evaluators, no glue code.

claude mcp add --transport http tracely https://api.tracely-ai.com/mcp \
  --header "Authorization: Bearer $TRACELY_KEY"

Then ask: "look at the last 20 traces, find what's failing, and add an evaluation column that catches it." The key scopes every call to its workspace. Nothing deletes; there is no tool for sending traces — that's the SDK's job.

Verify before claiming it works

1. Run the agent, then tracely.flush().
2. Open the UI → Traces. The run appears within a few seconds (evaluation is debounced ~4s).
3. Check the tree is a tree, not a flat list — if every span is a root, trace() isn't wrapping the

call or the framework escaped the context.

1. Check the conversation groups its turns.
2. Check a deliberately-broken run shows red.

No trace at all, spans not nesting, conversations not grouping, evaluators not running, gate suspiciously green: `references/troubleshooting.md` — symptom → cause → fix.

References

Self-hosting (docker compose up -d --build --wait → UI :3001, API :8000, dev key tracely_dev_key) and Railway one-click: <https://doc.tracely-ai.com/self-hosting>.