radiantlogicinc/fastworkflow/build-task-benchmarks

Building task benchmarks

A per-command smoke test proves a command reaches its backend. It cannot tell you whether a user can get a job done, because a job is many commands in sequence, and everything that makes sequences hard — navigation, handles passed between turns, the context a turn lands in — is exactly what a per-command test removes.

For a long-task reliability study, build the benchmark in two tiers: establish the short conversation blocks before composing longer tasks.

The tiers are not two views of the same thing. Tier 1 is your unit-test layer and the block library tier 2 is assembled from. That is what makes a tier-2 failure attributable: every block is independently green, so a step that only fails at length is a length problem, not a command problem. If you write tier 2 first you will spend your time debugging whether the command or the chain is broken.

Publish and inspect these tasks

This skill designs multi-turn content; its conversation/step dataclasses are driver conventions, not the native benchmark schema. For a small regression, keep the task as short as the behavior requires; the length ladder below is for testing long-task reliability, not a requirement for creating every benchmark.

Use create-workflow-benchmarks to publish tasks under benchmarks/, preserve task IDs, create a version-pinned experiment, and map content into the runner. run_chatbot lists tasks in benchmark/experiment details and shows recorded conversations under experiments. Inspect turns and steps on the right and attach human feedback to the relevant component. Use optimize-workflow-with-feedback to turn those findings into a paired comparison. Runtime success alone does not grade business outcomes.

Tier 1 — conversations as unit tests

Structure

A conversation is a name, a purpose, and an ordered list of turns. A turn is what the user says, the command that should handle it, its parameters, and optionally what to capture from the response and which context it should land in. Full dataclass and runner skeleton: reference.md.

Four rules make the difference between a real harness and a loop over command calls:

1. Navigation is real. The context a turn lands in is the context the next turn runs in. Do not

reset between turns. A conversation that only works because each turn was handed a fresh context is not testing anything a user will do.

1. Handles come out of the rendered response. Parse the uid a later turn needs from the text

an earlier turn produced, not from internal state. The rendered response is all an agent ever sees, so if a listing does not expose its handles legibly, that is a defect this suite should catch.

1. Run the validation hook. Signature.validate_extracted_parameters is part of the runtime

path, and normalising rules write back onto the parameter object the command body then reads. Skipping it tests a call the runtime never makes. Return a validation failure as a soft outcome, not an exception — the runtime hands the message back for correction, so a bad call is still a conversation.

1. Writes are opt-in and paired. Every write turn gets a restore turn, so a default run is

net-neutral. Anything genuinely destructive is validated (its input object is constructed) and never sent.

Verdicts

One boolean per turn destroys the signal. Distinguish, at minimum:

EMPTY versus FAIL is the distinction that keeps a suite honest. A suite that scores an empty dataset as failure gets muted; one that scores it as success hides real regressions. Keep them apart and report both.

Seeding

Do not seed from the first page of a picker. Seed objects with enough structure hanging off them to sustain a whole chain — a person who has rights and accounts and group memberships, a container that has children, a record that has a parent. Rare shapes are exactly the ones that break chains and exactly the ones missing from page one.

• Query with explicit bounds. An unbounded scan of a realtime view can take the backend down.
• Where a shape is rare, walk the catalogue rather than the first page, and say so in a comment.
• Seed per variant when a command's answer is typed. If findings are typed by entity family,

one seed per family, and ask each only for the family it has — otherwise you record an empty result as a command defect.

• Fail loudly at seed time if the dataset cannot support the suite. A missing seed should stop the

run, not silently produce twenty EMPTYs.

Coverage

Enumerate the routable command set from the routing definition and diff it against the commands your conversations exercised. Print the uncovered ones. Without this, a new command is untested-by-default and nobody notices.

Tier 2 — real-world tasks

Composition, not copy-paste

Write each tier-1 errand as a function returning a list of turns, then build tasks by concatenating them. Record on each task which conversations it combines. Copy-pasted steps drift from the blocks they came from and destroy attributability.

Pick task subjects a practitioner would recognise as a unit of work — an investigation, a recertification, an onboarding or offboarding sweep, a periodic review, an audit of one object across every system it touches. The test of a good subject is that you can state its purpose in two sentences without naming a command.

The ladder is the experiment

Build roughly six or seven tasks spanning 20 to 100 steps, each about 1.4× the last. The ladder is not padding — it is how you separate two hypotheses that a single long task cannot:

• Long tasks are harder per step (per-step quality falls as the task grows), versus
• Long tasks have more chances to fail (per-step quality is flat; only exposure grows).

In practice it is the second, and that changes the fix entirely.

Crossing contexts

A task that stays in one workspace is a tier-1 conversation wearing a costume. Real tasks move between them, and movement is a command, not a harness operation.

fastWorkflow puts navigation commands in every context — reset to root, go up to parent, list what is available here, report the current context. They live in the framework's metadata-extraction workflow rather than the workflow's own command folder, and they read the app workflow off their workflow's context. So a harness must run the real command bodies through a small shim object that carries {"app_workflow": workflow}, never reimplement them. Reimplementation is how you end up testing your own parent-chain logic instead of the framework's.

Treat these verbs as first-class steps: they get a say, they get routed, and they are scored like everything else. They are usually the least-trained commands in a workflow and the most load-bearing in a long task.

Static context planning

Because navigation is declared — a turn either names the context it enters, or is a go-up, or is a reset — the context every step runs in is computable before running anything. Maintain a current context and a stack; a navigation turn pushes, a go-up pops, a reset clears.

This pays twice: it is what lets you score routing with no backend at all, and in live mode it is a free assertion that the workflow's real navigation agrees with the declared plan.

Score two axes, never blended

Run them from one harness over one task definition so the two numbers describe the same steps. Keeping routing backend-free matters: it is the half that can run in CI.

Expect these to diverge sharply, and the divergence is the finding. A workflow can execute every step of a 100-step task perfectly and still be unusable, because the harness names commands explicitly while a user types English.

Routing outcome taxonomy

Scoring routing as right/wrong loses the two outcomes that matter most:

Score escalation by walking the ancestor stack exactly as the framework's wildcard handler does. An escalation label returned beside local candidates takes the ambiguity branch at runtime and never escalates, so it is ambiguous, not escalated.

Report compounding explicitly

This is the number the benchmark exists to produce. With a per-step clean rate p over n steps, unattended completion is roughly p^n. At p = 0.95, a 20-step task completes 36% of the time and a 100-step task 0.6% of the time. At p = 0.55 nothing above twenty steps is viable.

Report per-task: steps, per-step clean rate, p^n, and the index of the first derailing step. The first-derail index is what a reader acts on; the percentage is what they quote.

State the independence caveat honestly — a command that derails derails every time it appears, so real runs cluster their failures rather than sampling independently. It does not change the direction of the conclusion.

Phrasing discipline

The routing half is only as honest as its wording.

• Hold the phrasings out of the training seeds, and make the harness refuse to run if any

leaks in. Compare on a normalised form so dropping a full stop does not evade the check.

• Rotate three or four phrasings per command across tasks, deterministically, so no task's

score rests on one lucky wording and two runs stay comparable.

• Hold one business vocabulary steady across a whole session. A real practitioner says

"credential" or "login" consistently, not a different synonym each time. This scores lower than per-case varied wording and is the more representative number — a consistent vocabulary the seeds never trained on will fail consistently, which is precisely the failure users hit.

• Offline runs have no seeded ids; substitute readable placeholders, and carry the labels from

the last live run so searches are worded as they really would be.

What long tasks surface that short ones cannot

Look for these; they are the recurring findings.