yuanpingsong/ultracodex/agent-script-authoring

Writing Agent Scripts

An Agent Script is one JavaScript file that orchestrates a fleet of AI agents deterministically: the script decides what fans out, what gets verified, and how results flow; the agents do all the actual reading, writing, and thinking. You are writing the harness, not the workers. The format is model-agnostic and engine-portable — the same file runs on Claude Code's Workflow tool and on ultracodex (engine-implementer spec: docs/agent-script-spec.md in the ultracodex repo).

Part 1 is the contract — violating it breaks the script. Part 2 is the craft — what separates a good workflow from a merely valid one.

Part 1 — Core contract

1. File shape

• One plain-JavaScript ES module. TypeScript syntax is rejected. No import/export beyond the meta export, no dynamic import(), no require, process, filesystem, or network — the script itself cannot touch the outside world; only its agents can.
• The first statement is export const meta = {...}, and the initializer is a pure literal: objects, arrays, strings, numbers, booleans, null only — no variables, calls, spreads, or ${} interpolation.
• meta fields: name (kebab-case) and description (one line) required; whenToUse and phases (array of { title, detail? }) optional. Every phases[].title must exactly string-match a phase() call in the body, and every phase the body runs should be declared. A step deserves its own phase when it groups agent calls; bookkeeping gets a log() line instead.
• Everything after the meta export is evaluated as an async function body: top-level await and return are legal, and the return value is the run's result.
• Banned as nondeterministic (they break resume/replay): Date.now(), Math.random(), argless new Date(). Pass timestamps in via args; vary prompts by index when you need variety. Inside the agents' own work, time and randomness are fine — the ban is on the orchestration script.

2. Skeleton

export const meta = {
  name: 'find-flaky-tests',
  description: 'Find flaky tests and propose fixes',
  phases: [
    { title: 'Scan', detail: 'grep test logs for retries' },
    { title: 'Fix', detail: 'one agent per flaky test' },
  ],
}
const FLAKY_SCHEMA = {
  type: 'object',
  properties: { tests: { type: 'array', items: { type: 'string' } } },
  required: ['tests'],
  additionalProperties: false,
}
phase('Scan')
const scan = await agent('Grep the CI logs under ./logs for retry markers; return every flaky test name.', { schema: FLAKY_SCHEMA })
if (!scan) return { error: 'scan agent failed' }
phase('Fix')
const fixes = (await parallel(scan.tests.map((t) => () =>
  agent(
    `Diagnose why test "${t}" is flaky and fix it. Run the test 5x to confirm stability.`,
    { label: `fix:${t}` },
  ),      // <- closes agent(
)))       // <- closes map(, then parallel(, then the (await ...) wrapper
  .filter(Boolean)
log(`${fixes.length}/${scan.tests.length} tests fixed`)
return { fixed: fixes.length, of: scan.tests.length }

The closer of the nested fan-out idiom is the single most common parse failure: in (await parallel(items.map((x) => () => agent(...)))), the wrapping paren closes before .filter(Boolean). Count the parens on every multi-line map/parallel/pipeline close.

3. The eight globals — the complete API

There is nothing else. If you want another import or primitive, restructure so an agent does that work.

`agent(prompt, opts?) → Promise<string | object | null>` — spawns one agent. Without schema it resolves the agent's final message text; with schema (a JSON Schema object) it resolves a validated object, no parsing needed. opts:

• label — display name; use verb:target (review:auth, fix:parser).
• phase — assigns this call to a progress group. Use this instead of global `phase()` inside pipeline/parallel callbacks — the global mutates shared state and races across concurrent items.
• model / effort — advisory capability tiers (e.g. 'sonnet' | 'opus' | 'haiku' | 'fable'; effort 'low' | 'medium' | 'high' | 'xhigh' | 'max'), mapped by each engine, so scripts stay portable. Default: omit both. Override only for clearly mechanical stages (effort: 'low') or the hardest verify/judge stages.
• isolation: 'worktree' — a fresh git worktree per agent; costly. The deciding test is downstream visibility: if a gate or integrator must see all the parallel agents' edits in one tree, keep them on the shared tree — safe exactly when their file ownership is disjoint. Worktree-isolating disjoint-ownership builders strands their edits where the gate never sees them. Isolate only when parallel writers touch the same files, and plan how the surviving worktree merges back.
• agentType — an engine-defined agent profile (e.g. a read-only explorer type); the script still runs on engines that map it differently.

`parallel(thunks) → Promise<any[]>` — runs an array of () => Promise thunks concurrently and waits for all of them (a barrier). A thunk that throws — or whose agent fails — yields null at its index; the call itself never rejects.

`pipeline(items, ...stages) → Promise<any[]>` — each item flows through all stages independently, with no cross-item barrier. Stage callbacks receive (prev, item, index); the first stage's prev is the item itself. A stage that throws drops its item to null and skips its remaining stages; a stage that resolves null does NOT drop the item — later stages run with prev === null.

`phase(title)` / `log(message)` — progress grouping and narration. Neither affects results, but a script that narrates its coverage decisions is far more trustworthy to whoever is watching it run.

`args` — the run's input value, verbatim (undefined if none); how one script serves many runs. Read tunables defensively — const DB = args?.dbPath ?? './data.db' — a bare top-level args.x throws the moment the run launches without args. Structured inputs arrive as real JSON values, never re-encoded strings.

`budget` — { total: number|null, spent(), remaining() }, counting output tokens across the whole run; total is null when no target was set (then remaining() is Infinity — guard loops on budget.total first). The ceiling is hard: once spent() ≥ total, every further agent() call throws.

`workflow(nameOrRef, args?)` — runs a saved workflow (by name) or a script file ({ scriptPath }) as a child sharing this run's caps and budget. Nesting is one level only. Throws catchably on unknown name/path or child syntax error.

Caps, identical on every engine: concurrency auto-capped around 10–16 agents (excess calls just queue), at most 4096 items per parallel/pipeline call, at most 1000 agent calls per run.

4. Nulls and joins — design around failure

The core invariant: a failed agent call resolves `null`; it does not throw. Only the budget ceiling and the two caps throw — so don't wrap agent() in try/catch: with budget guarded, a catch adds nothing, and a genuine cap overrun should abort the run.

• .filter(Boolean) after every parallel(); null-check prev in later pipeline stages and every single awaited result before property access.
• Identify failures before you filter. To name what failed ("tell me which probe came back empty"), compute identities from the unfiltered results — ITEMS.filter((it, i) => !results[i]) — because filtering first destroys the index→identity mapping; a survivor count can never say which one is missing.
• Key results by id, not position. Put the item's id inside the result schema; never join by array position, indexOf, or by indexing two independently filtered arrays — filtering shifts indices and silently misaligns on the first null.
• In loops, the builder AND the verifier can both return null — decide explicitly whether null means retry, skip, or abort. Null-check into a temporary and commit only on success (const next = await agent(...); if (next) draft = next) — direct reassignment erases the prior good state exactly when a retry needs it. Guard anything a later iteration reads back; degrading a dead verifier to a synthetic failing verdict ({ pass: false, issues: ['verifier failed'] }) beats continue, and beats silently losing the item.

5. Pre-flight checklist

1. It parses. Run ultracodex validate --strict <file> FIRST when the CLI is available — it must print ok: no issues. Without it, mentally lint and re-read every fan-out closer.
2. Parens balanced on every nested fan-out (§2).
3. meta is a pure literal with name + description; every phase() declared in meta.phases and exact-matched.
4. No imports/require/fs/process/network, no TypeScript, no Date.now()/Math.random()/argless new Date().
5. Every null path handled per §4; joins by id; failures identified before filtering.
6. Loops capped or budget-guarded; dedup vs everything seen; no later iteration reads state a skipped one never populated.
7. Schemas strict per §7: additionalProperties: false, required = all keys, closed sets as enums (headline field graded, derived enums gated), strictest schema on the terminal producer, no aggregate rows in item arrays.
8. Fan-outs sized with overhead counted; budget thresholds ≥ one wave's cost; early stops return a slice-bounded resume tuple; all coverage bounds logged.
9. Tunables via args?.x ?? default — never a bare top-level args.x.
10. Parallel same-file writers isolated per the §3 deciding test; write stages gated by an agent that runs the checks itself (§9).
11. No added stage violates a deliverable-shape constraint (§6).
12. Prompts self-contained; verifiers separate and framed to refute; phase passed as an option inside concurrent callbacks.
13. The body returns one compact structured result — one shape per record set.

Part 2 — Craft reference

6. Choosing structure

Default to `pipeline()`. A barrier (parallel() between stages) is justified only when stage N genuinely needs all of stage N−1: dedup/merge across the full set, early-exit when the total is zero, or a prompt that references "the other findings". Not justifications: "I need to flatten/filter first" (do it inside a stage), "the stages are conceptually separate" (that's what pipeline models), "it's cleaner" (barrier latency is real — fast workers idle while the slowest finishes). Smell test: await parallel(...) → pure transform → await parallel(...) means the transform belongs inside a pipeline stage. When in doubt: pipeline.

The mirror error is adding stages the deliverable forbids: a fan-out may legitimately end at the fan-out. When results must stay raw, verbatim, or exactly structured, do not append a synthesizer — the schema is the aggregation, and a free-text consolidator cannot be trusted to preserve exact paths and counts even when told not to summarize. Before adding any stage, ask whether its output replaces a deliverable the problem defined more strictly than the stage can guarantee. Extra verification layers carry the same altitude question: stack refute passes when false positives are expensive, not by reflex.

Loops are the depth axis (breadth = parallel, flow = pipeline). There is no loop primitive; use plain while/for:

let draft = null, verdict = null
for (let round = 1; round <= MAX_ROUNDS; round++) {
  const next = await agent(round === 1 ? DRAFT_PROMPT : revisePrompt(draft, verdict.issues), { phase: `Round ${round}` })
  if (!next) break
  draft = next
  verdict = await agent(critiquePrompt(draft), { schema: VERDICT_SCHEMA, phase: `Round ${round}` })
  if (verdict?.pass) break                      // early exit on first pass
}