ob-labs/agentseek/langsmith-trace

LangSmith Trace

Add tracing to your agent and query traces for debugging. Supports Python and TypeScript.

[!IMPORTANT]

This skill is tuned for AgentSeek template backends (LangGraph + middleware stacks). For general LangSmith concepts, see the upstream langsmith-skills repo.

[!CAUTION]

Never pass `--api-key` as a CLI flag or expose API keys in shell commands / tool calls. The CLI reads LANGSMITH_API_KEY from the environment automatically. Using --api-key <value> leaks secrets into shell history, process listings, and agent tool-call logs. Always rely on the environment variable set in your shell profile or .env file.

Installation & Setup

1. Install the CLI

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

The binary installs to ~/.local/bin/langsmith. If langsmith is not found after install, add to your shell profile:

echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

2. Set your API key (environment variable only)

Add to ~/.zshrc (or the template's .env file) — the user must fill in their own key:

export LANGSMITH_API_KEY=<your-key-here>  # starts with lsv2_pt_

Key format must start with lsv2_pt_. Get yours at https://smith.langchain.com/settings.

The CLI reads this env var automatically — never use `--api-key` flags. Do not echo, print, or pass the key value in any shell command or tool call argument.

3. Verify

langsmith project list

If you see a JSON array of projects, you're set. Common failures:

• `command not found` — ~/.local/bin not in PATH (see step 1)
• 401 Unauthorized — key is wrong or expired; regenerate at LangSmith settings
• Empty array `[]` — valid auth but no projects yet; create one in the UI or run a traced app

Adding Tracing

LangGraph / LangChain apps (automatic)

Just set environment variables — no code changes needed:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY=<your-key-here>  # must be set; do NOT pass via --api-key flag
export LANGSMITH_PROJECT=my-project  # optional, defaults to "default"

For serverless (Python): also set LANGCHAIN_CALLBACKS_BACKGROUND=false to ensure traces flush before function exit.

Non-LangChain apps

Use the @traceable decorator (Python) or traceable() wrapper (TypeScript) and wrap your LLM client:

from langsmith import traceable
from langsmith.wrappers import wrap_openai
from openai import OpenAI
client = wrap_openai(OpenAI())
@traceable
def my_pipeline(question: str) -> str:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": question}],
    )
    return resp.choices[0].message.content

import { traceable } from "langsmith/traceable";
import { wrapOpenAI } from "langsmith/wrappers";
import OpenAI from "openai";
const client = wrapOpenAI(new OpenAI());
const myPipeline = traceable(async (question: string) => {
  const resp = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: question }],
  });
  return resp.choices[0].message.content || "";
}, { name: "my_pipeline" });

Debugging a Trace (Step-by-Step)

This is the recommended workflow for investigating agent behavior:

Step 1: Find the right project

langsmith project list

Look at last_run_start_time to find which project has recent activity. Don't assume — LangGraph apps default to the "default" project, not a named one.

Step 2: List recent traces

langsmith trace list --project default --limit 5
 
# Or with full hierarchy inline (combines steps 2+3):
langsmith trace list --project default --limit 5 --show-hierarchy

Step 3: Get the trace hierarchy

langsmith trace get <trace-id> --project <name>

This returns the full run tree — use it to understand the agent's execution flow.

Step 4: Get IO for all runs in a trace

langsmith run list --trace-ids <trace-id> --project <name> --include-io

This gives you inputs and outputs for every run (LLM calls, tool calls, middleware).

Step 5: Drill into a specific run

langsmith run get <run-id> --include-io

AgentSeek Trace Structure

Our templates produce traces with this typical hierarchy:

<agent_name> (root chain)
├── SkillsMiddleware.before_agent
├── PatchToolCallsMiddleware.before_agent
├── MemoryMiddleware.before_agent
├── model (chain) ← LLM turn
│   ├── TodoListMiddleware.awrap_model_call
│   ├── SkillsMiddleware.awrap_model_call
│   ├── FilesystemMiddleware.awrap_model_call
│   ├── SubAgentMiddleware.awrap_model_call
│   ├── SummarizationMiddleware.awrap_model_call
│   ├── AnthropicPromptCachingMiddleware.awrap_model_call
│   ├── MemoryMiddleware.awrap_model_call
│   └── ChatOpenAI (llm) ← actual LLM call (inputs/outputs here)
├── TodoListMiddleware.after_model
├── tools (chain) ← tool execution
│   ├── FilesystemMiddleware.awrap_tool_call
│   └── <tool_name> (tool) ← actual tool (inputs/outputs here)
├── model (chain) ← next LLM turn
│   └── ... (same middleware stack)
└── TodoListMiddleware.after_model

Key points:

• The actual LLM call is always the innermost ChatOpenAI run
• Tool results are in the <tool_name> run (e.g., generate_cover, execute)
• Middleware wrappers are transparent — they add latency but the IO you care about is at the leaf nodes

Gotchas

--full vs --include-io on individual runs

Problem: langsmith run get <id> --full can return null for inputs/outputs, even though the data exists. Despite --full being documented as equivalent to --include-metadata --include-io --include-feedback, the underlying API behavior differs for individual run fetches in some CLI versions.

Solution: Always use --include-io explicitly when inspecting specific runs:

# DO THIS
langsmith run get <run-id> --include-io
 
# NOT THIS (may return null IO despite docs saying it includes --include-io)
langsmith run get <run-id> --full

--full works reliably on trace export and run list, but has inconsistent behavior on individual run get calls. If this is fixed in a future CLI version, --include-io still works correctly — it's always safe.

Null IO even with --include-io

If inputs/outputs come back null even with --include-io, the project has IO logging disabled. Check for these environment variables in the template's .env:

LANGCHAIN_HIDE_INPUTS=true    # hides inputs from traces
LANGCHAIN_HIDE_OUTPUTS=true   # hides outputs from traces

Remove or set to false to enable IO capture for debugging.

Project confusion

LangGraph apps trace to the "default" project unless LANGSMITH_PROJECT is explicitly set. Always run langsmith project list first and check last_run_start_time to find where your traces actually landed.

Tips

• Add --format pretty for human-readable output during interactive debugging
• Use LANGSMITH_ENDPOINT env var if connecting to a self-hosted LangSmith instance
• The middleware ordering in the trace tree is configurable — your template may differ slightly from the diagram above

CLI Reference

Full command reference: reference/cli-commands.md