context-labs/halo/claude

HALO Loop — diagnostic-led harness improvement

When to invoke this skill

Use this skill when:

• The user is iterating on an agent harness that emits HALO-shaped JSONL trace files (e.g. an OpenAI Agents SDK app wired with tracing.py, an AppWorld run, or any harness that follows docs/integrations/openai-agents-sdk.md).
• The user wants to find and fix harness-level issues (prompts, tool descriptions, retry logic, error handling, configuration) — not model-level issues.
• The user explicitly says "use HALO", "improve the harness", "look at the traces and figure out what's wrong", or similar.

Do not invoke this skill for:

• General agent-development questions unrelated to a specific trace dataset.
• Cases where there are no captured traces yet — you'd need to set tracing up first (see Setup below) and run the harness.
• Cases where the user wants you to fix an LLM model (HALO is for harness, not model weights).

Mental model — what HALO Engine is and isn't

Halo Engine is a trace-exploration runtime. It's an LLM agent that has tools to read a JSONL trace dataset and answer questions about it. It is not a code-modification tool, not a fix proposer, and not a verifier.

The engine's hard-coded root system prompt (engine/agents/prompt_templates.py) starts with:

"You are the root agent in the HALO engine. You explore OTel trace data using the tools the runtime provides."

Treat its output as trace evidence, not as a directive. It can identify patterns, cite trace_ids, surface error strings, count failure modes — that's what it's for. It cannot see your code; if it names a file path or claims a constraint is missing, verify before acting.

The HALO loop is therefore a two-actor system:

You should never ask HALO Engine to propose code changes. You should ask it questions whose answers help you decide where to look in the code.

Setup — getting HALO Engine ready to use

Prereqs:

• A clone of https://github.com/inference-net/HALO somewhere on disk. The path is referred to below as $HALO.
• Python 3.10+, uv for environment management.
• An OPENAI_API_KEY env var (the engine itself uses an LLM to drive tool calls).

Install the engine + CLI once:

cd $HALO
uv sync                       # creates .venv with halo-engine (CLI bundled)

That's it for the engine side. The CLI entry point is halo (registered by pyproject.toml).

Trace-format prerequisites — what the harness must produce

The engine reads a single JSONL file. Each line is one OTel-shaped span as a JSON object. The format is documented at $HALO/docs/integrations/openai-agents-sdk.md. Required top-level fields:

trace_id, span_id, parent_span_id, name, kind, start_time, end_time,
status: { code, message },
resource: { attributes: {...} },
scope: { name, version },
attributes: { ... }

Required attributes.* keys for HALO indexing to work:

• inference.export.schema_version (always "1")
• inference.project_id — used by the engine's index/filter
• inference.observation_kind — one of AGENT, LLM, TOOL, CHAIN, GUARDRAIL, SPAN

Optional but valuable for richer analysis:

• inference.llm.model_name, inference.llm.input_tokens, inference.llm.output_tokens (LLM spans)
• tool.name, input.value, output.value (TOOL spans)
• agent.name (AGENT root span)

Important format constraints:

1. Plain JSONL only — not gzip. The engine opens the file with Path.open("rb") and parses one span per line. Gzip files won't work; gunzip first.
2. Append mode accumulation: If the harness uses HALO's vendored tracing.py, the file is opened in append mode. Repeated runs stack in the same file. Before a fresh diagnostic pass, clean stale outputs (e.g. task clean:run-outputs in the AppWorld demo, or just delete the file and re-run the harness).
3. Parallel runs produce per-process files (traces-p0.jsonl, traces-p1.jsonl, ...). Merge them into a single traces.jsonl before invoking the engine — the AppWorld demo has a task traces:merge task that does this; for ad-hoc cases cat traces-p*.jsonl > traces.jsonl works fine since the engine indexes by trace_id and ignores order.

If the user has a harness that doesn't yet emit traces, the canonical setup pattern is in $HALO/demo/openai-agents-sdk-demo/:

• Copy tracing.py verbatim into the harness's source tree.
• At startup, before any Agent(...) is constructed, call:

``python import agents agents.set_trace_processors([]) # clear SDK default OpenAI uploader processor = setup_tracing(service_name="myapp", project_id="myproj") ``

• Call processor.shutdown() in a finally block before exit so the file flushes.

For non-SDK harnesses, refer to the schema in $HALO/engine/traces/models/canonical_span.py — the engine accepts any JSONL that matches that shape.

Don't accidentally break the mechanism:

• Don't replace set_trace_processors([]) with set_tracing_disabled(True) — that disables HALO's processor too.
• Don't strip the inference.* keys from tracing.py — those are what the engine indexes on. If a span loses inference.project_id, the engine still parses it (the model is extra="allow") but filters and overviews silently drop it.
• Don't use OPENAI_AGENTS_DISABLE_TRACING=1 — same effect as set_tracing_disabled. The engine's default OpenAI-dashboard uploader is what we want to disable, not tracing as a whole. The pattern is "clear default, add HALO's processor on top".

For deeper format details: $HALO/docs/integrations/openai-agents-sdk.md and $HALO/engine/traces/models/canonical_span.py.

The HALO loop — the workflow you should follow

Once the harness emits valid traces:

Step 1 — Capture baseline

Run the harness once and capture eval results. Note the score so you can measure improvement later. If running an existing demo (e.g. AppWorld), the run command typically also produces an evaluation report.

Step 2 — Diagnose with halo

Invoke the CLI with a question, not a command:

cd $HALO
uv run halo /absolute/path/to/traces.jsonl \
    --prompt "What are the most common failure modes across the failed traces? For each, give me trace_id evidence and the precise error string." \
    --model gpt-4.1-2025-04-14 \
    --max-turns 15

Important parameters:

• `TRACE_PATH` (positional, required): absolute path to a JSONL trace file. Must exist; must be plain JSONL.
• `--prompt` / `-p` (required): the question. Frame as a diagnostic question, never as "propose a fix".
• `--model` / `-m` (default gpt-5.4-mini): the engine's driver model. Bump to gpt-4.1-2025-04-14 (1M context) for large traces — multiple runs at default model OOM on AppWorld-sized data even with attribute truncation. The engine still hits context limits when conversation history accumulates across many tool calls; bigger model = more headroom.
• `--max-turns` (default 8): how many reasoning turns the root agent gets. 12-25 is reasonable for deep analyses; lower numbers force tighter answers.
• `--max-depth` (default 1): subagent recursion depth. Default is fine in most cases.
• `--max-parallel` (default 2): concurrent subagent cap. Default is fine.

Step 3 — Verify the engine's claims against the actual repo

The engine's tool descriptions and instructions don't let it touch your filesystem. Anything it says about a file path, line number, or "this constraint is missing" is dead reckoning from convention — verify before treating it as truth.

After a diagnostic answer comes back, do one or more of:

# Find the file the engine alluded to:
rg -l "page_limit" $HALO/demo/appworld/src/appworld/apps/spotify/
 
# Check whether the constraint is actually missing:
rg "le=20|maximum.*20" $HALO/demo/appworld/src/appworld/apps/spotify/apis.py
 
# Check whether the prompt says what the engine claimed:
rg "page_limit" $HALO/demo/appworld/experiments/prompts/function_calling_agent/

If the engine said file X.json exists and rg/Read show it doesn't, that's a hallucinated path — re-grep based on the concept (the tool name, the error string) rather than the engine's path guess.

Step 4 — Form a hypothesis, make the smallest change

Based on verified evidence:

• Identify the precise file(s) involved.
• Choose the minimal-blast-radius edit (description text, a single prompt line, a config value, an error-recovery branch).
• Make the change. Do not make sweeping refactors based on a single diagnostic pass.

Step 5 — Re-run + re-diagnose

Re-run the harness, capture new traces, ask the same diagnostic question to compare. Look at the eval delta numerically; look at whether the cited error pattern reduced in frequency.

If it did, ship. If it didn't, you misread the evidence — go back to step 2 with a sharper question.

Diagnostic-mode discipline — questions to ask vs. avoid

Good prompts (HALO answers these well):

• "How many traces had at least one TOOL span with status.code == STATUS_CODE_ERROR? List the top 5 by error count and cite trace_ids."
• "Across the failed traces, what's the most frequent literal error string in output.value of TOOL spans? Group near-duplicates."
• "Look at trace <id>. What was the tool sequence? Did the agent ever recover after a tool error?"
• "Which tool names appear in function.* spans most often, and which of those are most associated with error outcomes?"
• "Compare two specific traces — one that completed cleanly and one that hit max-turns. What did the failing one do differently after turn 5?"
• "Are there traces where the same tool was called more than 3 times with similar arguments? Cite trace_ids."

Bad prompts (HALO will answer but the result is unreliable):

• "What change should I make to fix this?" — engine has no repo access; will hallucinate file paths.
• "Write a patch for prompts/instructions.txt." — same, plus you should write the patch yourself once you've verified the diagnosis.
• "Refactor the harness to be more robust." — too vague; engine will give generic advice.
• "Is the API predictor under-fetching?" — leading; the engine will agree because the question implies it. Ask "How often does the trace contain a tool call to a tool that wasn't in the predicted list?" instead.

The pattern: ask about the trace data, not about the code. Use the answer to navigate the code yourself.

Halo Engine tool inventory — what the engine can do internally

Knowing the engine's own tool surface helps you frame prompts that route to the cheapest tools: