darwin-agent/harnessx/tau2-playbook

τ²-bench: Benchmark-specific Guidance

Current Benchmark

• What: τ²-bench (Barres et al., Sierra, arXiv 2506.07982) — multi-turn

dialogue eval where an LLM-driven user converses with the agent, the agent calls domain tools that mutate a shared database, and a three-component evaluator scores the run. Evolves out of τ-bench (ICLR'25, retail+airline) → τ²-bench (adds telecom as dual-control Dec-POMDP) → τ³-bench (Oct'25, adds banking+voice). Repo: github.com/sierra-research/tau2-bench. HarnessX targets τ².

• Domains used by this evolver:

Full telecom pool is 2285 tasks; the runner always uses base.

• Reward ∈ [0, 1] = product of three components; any zero zeros everything:
• DB check — final database matches ground truth.
• Action checks — per-action action_match booleans.
• NL assertions — LLM judge rates claims about the dialogue (temp 0.3).

• pass^k vs pass@k — Sierra's signature framing. pass@k = at least

one of k runs passes; `pass^k` = all k runs pass. A 70 % pass^1 agent collapses to ~25 % pass^8. We currently run pass^1 (defaults.NUM_TRIALS = 1); raise NUM_TRIALS to surface reliability.

• Realistic baselines (τ²-verified avg@3, Nova-2 report Mar 2026 —

citable public numbers). Airline is uniformly hardest (brittle single-solution grading); telecom is highest (multiple valid paths):

HarnessX's own runs are in benchmarks/tau2/README.md.

Task Structure

Each task is a multi-turn dialogue built around one customer-service goal: user opens with an ambiguous request → agent info-gathers via read tools → diagnoses and proposes → user confirms → agent executes write tools → user acknowledges or adds a follow-up, then stops (###STOP###).

Tools split into two classes by name convention (basis of PhaseAwareToolFilter in benchmarks/tau2/tool_filter.py):

Per-Domain Cheat-Sheets

Cross-domain guidance is usually wrong in subtle ways. Anchor any instruction-layer fix to the specific domain that surfaced the failure.

retail — order status drives tool choice: modify_pending_order_items and cancel_pending_order need pending; exchange_delivered_order_items needs delivered. Returns must use the original payment method (no swapping). Exchanges must be same product, different option. Courtesy credits often require escalation, not silent issue.

airline (hardest by pass^1) — basic-economy cannot be modified (only cancellable via insurance); strict baggage rules per cabin; user ID and reservation ID must be elicited, never fabricated. Hidden user constraints ("I only accept EWR, not JFK") live in user_instruction and surface only when the agent asks. Single-solution ground truth means reward is brittle — a correct alternative can still score zero.

telecom (dual-control) — task-class hierarchy: service_issue < mobile_data_issue < mms_issue; harder classes require resolving underlying issues first. The agent cannot toggle user-side settings (airplane mode, cellular data, roaming toggle, APN); it must instruct the user in natural language, wait for confirmation, then re-read state. PolicyHintProcessor (benchmarks/tau2/policy_hint.py) is an off-by-default telecom companion that injects positive remediation hints when conditions like data_exceeded, roaming_disabled_abroad, check_correct_line are detected in tool results.

Analyzing Failures

Frontmatter fields emitted by the evolver (always present):

Cross-reference with the trajectory body's ## Result block, which prints raw reward_info: db_match, db_reward, per-action action_checks, and nl_assertions outputs. Those are the most direct evidence for which component let you down.

Known Techniques That Improve tau2 Scores

Already wired in `harness_config.yaml`:

Public-writeup techniques worth authoring as new candidates:

SystemAppendProcessor pattern

tau2 injects its own system prompt (the full policy document + tool descriptions). Never replace it with `TemplateSystemPromptBuilder` — that deletes the policy text and scores collapse. SystemAppendProcessor reads tau2's system message on on_task_start, appends a markdown guidance file, and sets task_system_prompt to the combined text.

# output_dir/config.yaml
processors:
  - _target_: recipe.tau2_evolver.system_append_processor.SystemAppendProcessor
    append_path: /abs/path/to/recipe/tau2_evolver/guidance_retail.md

The shipped guidance_retail.md encodes execute-on-stated-intent, exchange-vs-modify disambiguation, and attempt-tool-calls discipline. For airline/telecom, author a new file drawn from patterns seen in that domain's trajectories — this is where the Quesma/Barres decision-tree rewrite lives.

Common Failure Modes

A. Premature action call

Agent fires a write on turn 0 or 1 from the user's opening message alone, before any read tool.

• Signal: first entry of tool_call_counts is an action tool;

small num_messages; db_match=False.

• Fix: raise PhaseAwareToolFilter.read_only_steps; or guidance

rule "always call at least one read before any action".

B. Over-confirmation / re-asking loop

Agent proposes a plan, user confirms, agent asks for re-confirmation, exchange loops until max_steps.

• Signal: num_messages ≥ 15 on a 4-6 turn task; many reads, no

writes; exit_reason = max_steps or user_stop.

• Fix: instruction rule "execute on stated intent, do not ask for

re-confirmation" (already in guidance_retail.md); tighter LoopDetectionProcessor.

C. Hallucinated arguments or unsupported claims

Two shapes:

• In a tool call: invented order ID / customer ID / parameter not

present in dialogue or read-tool output. Signal: tool_error_counts spikes on a mutation tool; body contains Order X not found.

• In an assistant message (no tool call): confident numeric or

policy claim the agent didn't look up. Signal: nl_reward < 1.0 with db_match=True; no corresponding read tool in the turn.

Fix: guidance rule "every ID, number, or date must appear verbatim in a prior tool result or user message"; or a trust-score / before-tool processor that pattern-matches arguments against recent read outputs.

D. Exchange-vs-modify tool confusion (retail)

modify_pending_order_items on a delivered order, or exchange_delivered_order_items on a pending order.