statelyai/agent/improve-machine

Improve an agent machine

You improve a policy: the machine that decides how the agent works toward its goal, written in TypeScript or as a JSON AgentWorkflowConfig. It runs inside an environment: a TypeScript machine whose states and guards encode the rules of the world and the business invariants. One round changes the policy once and measures it:

locate inputs → read traces → diagnose → propose one diff → gate (≤ 3 repairs) → replay train → replay held-out → recommend or discard → record → pull request

This is step 4 of 4, after loop-to-machine, add-traces, and add-evals. To author a JSON policy from scratch, use the generate-machine skill. Concept page: Environment and policy.

Hard rules

• Never edit the environment. Not its states, guards, invariants, event schemas, or the vocabulary it exposes. When the goal is unreachable without an environment change, write a change request for a person (template in reference.md) and stop.
• Never edit a machine while it runs. A promoted policy applies to new runs only.
• The objective comes from ground truth. Score outcomes from the traces (terminal states, outputs) or from effects recorded in the world. Never ask a model to grade its own runs.
• One cause per round, smallest diff. A diff that fixes two things cannot tell you which one helped.
• Diagnose from training scenarios only. Held-out scenarios are for the promotion decision. Never read their failures to shape a change, and never move a scenario between splits. Eval files may list baseline failures for every split; read only the training rows.
• Stop after 3 rounds without a promotion (or the limit the user gives).
• Propose, never promote. A round ends in a recommendation. A recommended candidate goes to a person as a pull request with its scorecard; merging it is the promotion. Never merge, deploy, or switch the running policy yourself.

Policy format

The rounds are the same for both formats. What differs is what the diff may touch and how it is gated.

No policy yet. loop-to-machine leaves a single model decision inside the environment when the old code had no policy logic. Then:

• Record v0 in a new CHANGELOG.md next to the environment (in a policy/ directory): "the model decides directly". This entry goes in the working tree.
• The first recommended candidate introduces the policy as v1: a policy file that reproduces the current behavior (it passes the environment's prompt and facts through to the model) plus the one rule the diagnosis found, and the routing line in the runner (withPolicy(modelExecutors)). The policy runs as the decider through createPolicyDecider from @statelyai/agent; the pattern is in ../loop-to-machine/policy.md.
• Its description is the environment's goal with measurable success criteria. If the environment's goal has none, draft them from the confirmed objective and list the draft under questions for the user.
• If the policy needs a fact the environment does not pass in the decision's metadata.facts, that is an environment change request.

For a TypeScript policy, before the first round:

• List the environment files (the environment machine, its guards, the world and its tools) and record a hash of each. Any change to them fails the round. The runner is not one of them, but its diff may contain only the routing line.
• Keep the policy a machine. A diff may add states, choice branches, guards over facts, helpers in the policy file, or change a prompt or a model ref. It may not turn the policy into a free function, call tools the environment owns, or import from the environment anything but its schemas, types, constants, and exported guard functions.

1. Locate the inputs

Find all six before changing anything. If one is missing, stop and ask.

The objective defines "better". Do not start a round until the user has confirmed it, including how two versions compare when it has more than one field. A good objective has:

• a primary metric from the goal (accuracy against expected outcomes and effects, resolution rate). With several, state the rule: one primary, or none worse and one better,
• a hard constraint: invariant violations, which must be 0,
• a cost term (model calls, tokens, guard rejections) used as a tie-breaker or a stated budget. Count the calls that reach a model executor, not the environment's decision requests: a policy that answers with a rule makes no model call, and a cost term that counts requests cannot see the saving.

2. Read

Read both machines as text. describeMachine puts the goal first:

import { describeMachine, setupAgent } from "@statelyai/agent";
// A TypeScript policy: import it. A JSON policy: lower it.
const { machine: policy } = setupAgent.fromConfig(policyConfig, { compileSchema });
const environmentText = describeMachine(environment);
const policyText = describeMachine(policy);

Then find the training threads and compute one outcome row per episode:

import { getDecisionAttempts, getUsageFromEvents, replay } from "@statelyai/agent/log";
// Thread ids name the split and the decider version, as add-traces set up: "train:<deciderVersion>:<scenario>:r<k>".
// With a policy, the decider version is the policy's version ("v3"); before one exists, the model's label.
const prefix = `train:${deciderVersion}:`;
const threadIds: string[] = [];
let cursor: string | undefined;
do {
  const page = await store.listThreads!({ prefix, limit: 100, cursor });
  threadIds.push(...page.threadIds);
  cursor = page.cursor;
} while (cursor !== undefined);
const rows = [];
for (const threadId of threadIds) {
  const scenario = scenarioOf(threadId);
  const entries = await store.read(threadId);
  const { snapshot } = replay(environment, entries); // the recording machine; pure fold, no model calls
  rows.push({
    threadId,
    expected: scenario.expected,
    state: snapshot.value,
    status: snapshot.status,
    output: snapshot.output,
    steps: entries.length,
    tokens: getUsageFromEvents(entries).totalTokens ?? 0,
    failedAttempts: getDecisionAttempts(entries).flatMap(({ attempts }) => attempts),
  });
}

• store.listThreads({ prefix, limit, cursor }) pages through the threads that hold at least one entry. cursor is present only while more ids remain. It is optional on a custom AgentEventLogStore; createInMemoryEventLogStore implements it. Without it, use the thread ids recorded with each scenario.
• Prefix thread ids with the split and version so a listThreads prefix returns one version's training runs and never a held-out thread. scenarioOf is your mapping from a thread id back to its scenario.
• store.read(threadId) returns a thread's entries in log order.
• replay(machine, entries) needs the machine, at the version, that recorded the thread. Here that is the environment: its runtime records every thread, and the policy runs only as a decider at its boundary, so never replay with the policy machine. A thread recorded by a different environment version throws AgentMachineVersionMismatchError. Score each thread with the version that recorded it.
• getDecisionAttempts(entries) returns { index, eventType, attempts } for each entry whose decision needed retries: the failed tries before the event landed, or every try of an exhausted decision (eventType is then xstate.error.actor.*). Each attempt has failure (unknown-event, invalid-payload, or rejected-by-guard), reason, and the attempted event when there was one. A first-try decision records nothing.
• agent.decide invokes journal their attempts on their own. With decideNext at the boundary, the host must pass them with the event, runToQuiescence(runtime, { event, attempts }) or runtime.transition(state, event, { attempts }), or they never reach the log.
• Compute the objective over the rows. This is the baseline for the round.

3. Diagnose

Group the failing training episodes by cause. Check these clusters in order:

1. Invariant violations. Any violation is a stop: the environment is wrong, not the policy. Write a change request.
2. Wrong terminal state. The episode ended in escalated when the label says issued, or the reverse. Look at which decision led there.
3. Right terminal state, wrong effect. The episode ended issued with the wrong amount, or reached the right state by a costlier path. Compare the chosen event with the offered options at each decision.
4. Guard rejections. The policy keeps proposing events the environment refuses: rejected-by-guard entries in the rows' failedAttempts. The policy lacks a check the environment already makes.
5. Loops and budget exhaustion. Repeated state values, or a terminal reached because a turn or retry budget ran out.
6. Cost hotspots. Correct episodes that spend model calls where a choice over known facts, or a cheap judgment, would decide.

Pick the cluster with the largest effect on the objective. Write the hypothesis down before you edit anything (template in reference.md): cause, evidence (thread ids), the change, and which metric should move.

4. Propose

Make the smallest policy diff that addresses the one cause.