darwin-agent/harnessx/reference

Reference — HarnessX component authoring

Where things live

All authored files are referenced from config.yaml with absolute paths. Relative paths are rejected by the loader.

Authoring a new @tool

This section assumes you've already decided Action is the right lever. For whether to author a new tool at all (vs. retune an existing one, write a Control hook, or edit the prompt), see the analyze skill — lever choice lives there, authoring mechanics live here.

Before you type — write the shape spec

For anything larger than a short stdlib helper, write the spec in _meta_scratch/TOOL_SPEC.md before any code. Short sections, no prose padding:

# TOOL_SPEC — <one-line capability name>
 
## Capability name
What the agent literally cannot do right now.
 
## Class, not single task
Which class of failing tasks would this help? If only one exotic
task — abandon, this isn't a capability gap.
 
## Inputs the agent will pass
Real examples from failing trajectories — URLs, file tokens,
queries, record ids. Synthetic examples are disallowed; they would
pass dry-fire but tell you nothing about how the tool behaves on the
inputs the benchmark keeps presenting.
 
## Output shape the caller needs (not what you want to produce)
From the caller's POV: structured string? count? page-indexed text?
When the agent gets this back, what can it do with it?
 
## Failure modes
Named modes — auth denied, timeout, unsupported format, empty
result, rate-limited. What does the tool return in each? Never
return `""` silently — always a diagnostic string the caller can
detect.

A spec you cannot write honestly is a spec that is too broad — narrow it. The spec's value is framing the inputs concretely and pinning anti-patterns (no generic replacements, no hardcoded task ids) at the design stage.

Pre-flight

For pure-stdlib utilities, the pre-flight is small:

1. Dump the `Tool` dataclass fields — learn what the decorator

builds before you write a call site for it: ``bash python -c "from dataclasses import fields; from harnessx.tools.base import Tool; \ [print(f.name, '=', f.type) for f in fields(Tool)]" ``

1. Read the nearest builtin in harnessx/tools/builtin/ that is

structurally similar (same return shape, same env access pattern, same failure mode). Match its signature and error-handling style; don't invent a new convention.

If your tool touches the network or imports a third-party package:

1. Write the imports as usual. If dry-fire reports

ImportError, either fall back to stdlib or record the dep in _meta_scratch/NEEDS_FROM_HUMAN.md and try a different approach.

1. Check prior art under harnessx/tools/builtin/ for the

keyword in your diagnosis — but decide by observed behaviour on this benchmark's inputs, not by whether a similarly named builtin exists (see Prior-art rule).

Minimal shape

# output_dir/tools/<name>.py
from harnessx.tools.base import tool
@tool(description="One-line capability sentence — what can the agent now do that it couldn't before.")
async def <fn_name>(<typed args>) -> str:
    # Import third-party deps at module top-level when possible;
    # inline imports are fine for narrow capabilities that don't
    # justify package-level visibility.
    #
    # Return a diagnostic string on every failure mode named in
    # TOOL_SPEC.md. Never return "" — empty output gets absorbed by
    # the caller; a tagged message like "[<tool> failed: <reason>]"
    # lets the agent fall through deliberately.
    ...

Concrete tool examples (search fallbacks, file-format parsers, domain-specific clients) belong in the recipe's <bench>-playbook skill — they encode the benchmark's input distribution and failure classes, which this skill deliberately does not.

Rules:

• Exactly one module-level Tool object per file (produced by

the @tool decorator on a function).

• async def or regular def — both are supported.
• Input args must have type hints; they become the JSON Schema.
• Return a string, or something stringifiable. Tool outputs feed

back to the model as text.

• Third-party imports: just import at module top-level. Dry-fire
• replay-gate handle env uncertainty downstream.
• No hardcoded task ids, URLs, or answers in fn bodies — the

literals validator rejects these (R4).

Reference from config.yaml:

tools:
  builtin: [Read, Write, Edit, Glob, Grep, Bash, WebSearch, WebFetch]
  custom:
    - file:///abs/path/output_dir/tools/<name>.py::<fn_name>

The string after :: is the function name (what the tool will be called as). If the inner agent should use your tool instead of a builtin, remove the builtin's PascalCase name from tools.builtin.

Authoring a new MultiHookProcessor

Mandatory pre-flight

Most contract violations in evolve rounds come from skipping one of these. Walk them in order every time, regardless of how trivial the hook looks:

1. Pick the hook deliberately. Cross-check against the

hook dispatch table — an on_* method whose suffix isn't a known event becomes an import-time TypeError.

1. Dump the event dataclass (see

Event fields) before typing any event.<field> access. Field names drift; don't trust memory or pattern-match from another processor.

1. If your hook mutates `event.messages` (only on_step_start

and on_before_model are allowed to), read harnessx/core/processor.py::_validate_messages_contract end-to-end, plus check_post_hook_invariants in the same file. Then re-check your hook body against every violation category listed in Messages-mutation contract.

1. Grep prior art under harnessx/processors/control/,

context/, memory/. If a builtin's observed behaviour is close to what you want, a config-layer retune beats a new file.

Minimal shape

# output_dir/processors/tool_error_tail_injector.py
from __future__ import annotations
import dataclasses
from harnessx.core.events import ToolResultEvent, Message
from harnessx.core.processor import MultiHookProcessor
class ToolErrorTailInjector(MultiHookProcessor):
    """When a tool errors, append a short tail note to its result so
    the agent sees the failure mode without re-reading the full
    stderr block."""
    _singleton_group = "tool_error_tail_injector"
    _order = 25  # after loop_detection(20), before otel(30)
    def __init__(self, tail_chars: int = 200):
        self.tail_chars = int(tail_chars)
    async def on_after_tool(self, event: ToolResultEvent):
        result = event.result or ""
        if not isinstance(result, str) or not event.is_error:
            yield event
            return
        tail = result[-self.tail_chars:].strip()
        annotated = f"{result}\n\n[tail] {tail}"
        yield dataclasses.replace(event, result=annotated)

Zero-arg-friendly constructor: the loader may instantiate with no kwargs if the YAML entry doesn't pass any.

(Benchmark-specific processors — for example, one that enforces a particular answer-format marker — belong in the recipe, authored by the meta-agent when the benchmark's playbook identifies a grader requirement that the current config isn't meeting.)

Hook dispatch table

Name a method on_* but not one of those → import-time TypeError. Use @on(CustomEventClass) to handle a custom event under a free name.

Event fields — look them up, never guess

python -c "
from dataclasses import fields
from harnessx.core.events import BeforeModelEvent  # swap in your event
for f in fields(BeforeModelEvent):
    print(f.name, '=', f.type)
"

Referencing a non-existent field (event.message when the field is messages) passes canonicalize but explodes on the next benchmark run. Copy-paste field names from the dataclass.

Messages-mutation contract

Only two hooks may touch event.messages: on_step_start and on_before_model. Violations surface at runtime as CONTRACT [<type>] hook=... processor=... warnings, and raise ContractViolationError under HARNESSX_CONTRACT_MODE=strict.

Single source of truth is `harnessx/core/processor.py::_validate_messages_contract` (plus check_post_hook_invariants for chain-level rules). The list below is a scanning checklist.

Violation categories:

• Empty messages (empty_messages) — before_model output must

be non-empty.

• Length delta out of range (length_exceeded) — before_model:

a processor may not remove messages, and may add at most +1 per step.

• Tail role & insertion rules — when before_model adds +1:

the pre-hook tail must not already be user (content-edit only in that case), and the inserted message itself must have role="user".

• Chain-level insertion cap — across all processors in a

single before_model chain, at most +1 user insertion total.

• `system_window` invariance — step_start may not change

messages[0] when its role is "system".

• Structural-vs-content edits — step_start may not ONLY change

the tail (that's before_model's job). before_model with delta==0: if tail is user, only its content may change and the prefix must be identical; if tail is not user, no change allowed.

Chain-level invariants in check_post_hook_invariants:

• system count ≤ 1 and must sit at index 0.
• raw_track and effective_track must have equal length, matching

roles at every index, and matching tool_call_id for role=='tool'.

Safest default when unsure: a pure content rewrite of the existing tail user message (delta==0, prefix untouched) — valid in both step_start and before_model's user-tail branch.

Yielding

A hook is an async def generator. Yield one or more events:

• yield event — pass through unchanged
• yield dataclasses.replace(event, messages=new_tuple) — mutate a

frozen dataclass field

• yield ProcessorTriggerEvent(...) — emit an observability event

alongside the main one

Yielding nothing blocks the hook; don't do this unless you intentionally want to suppress a tool call or model turn.

Ordering

_singleton_group prevents two processors of the same kind stacking. _order: 10 = pre, 20 = normal, 30 = post. Builtins use token_budget=10, cost_guard=10, checkpoint=10, loop_detection=20, otel=30. Slot new guards at 15–25 unless you specifically need to run before or after a named one.

Reference from config.yaml:

processors:
  - _target_: file:///abs/path/output_dir/processors/tool_error_tail_injector.py::ToolErrorTailInjector
    tail_chars: 200

Kwargs in the YAML map directly to the processor's __init__ args.

Editing the inner agent's system prompt (instruction layer)