zvadaadam/az-skills/verify-harness

Verify Harness

Agents that change a product need to prove the change the way a person would see it. Without a harness they write a fresh Playwright or curl script every task, look at one screenshot, and throw the script away. The next agent starts from nothing, and nobody can re-run the proof.

A verification harness fixes that with two things that live in the repository:

1. `control-<app>`, a driver CLI. It starts the real app in an isolated, throwaway session and acts on it the way a person does: roles and names, real modifier keys, drags, menus, files, keyboard. It also acts the way an agent does, through the app's own CLI, API or MCP. Every observable result is stated as an expect, and the evidence is kept as a proof.
2. A feature map, one file per user-facing area, written from the user's point of view. Each file says what the area does, every way to reach it, a recipe of control-<app> commands that proves it, and the gotchas that mislead. A recipe replays on its own, so the map is also the regression checklist.

Here, “harness” means the tools that run and check the application. The guide the agent reads is verify-<app> (SKILL.md and features/). Put the driver in the repository's normal CLI or development tooling, covered by its build, types, lint and CI; keep executable test helpers with the repository's tests. The skill links to those commands rather than owning a parallel application. This skill builds that workflow, grows it, uses it and keeps it true.

For the broader repository setup—development commands, architecture and automated constraints, agent guidance, task skills and maintenance—use setup-agent-workflow when available. This skill owns the verification portion.

Keep the proof and the investment proportional

The feature map is navigation and executable product knowledge, not a list of test files. Start from a report's words, find the user's entry point and expected behaviour, drive it, inspect the result, then preserve that route for the next agent. Existing tests are fast supporting checks; a green test suite alone does not prove the user's journey.

Start with one useful vertical slice and reuse the app's existing CLI, API, browser or device controls. Add a persistent daemon only when the selected recipes need a session across commands. The driver contract is a capability catalogue, not a requirement to implement every command before the first proof. Add gestures, faults, tracker integration and other controls when a real recipe needs them. Keep unsupported controls and entry points in a visible gaps list.

Every proof names its layer: test, component/integration, local application, or live provider. Record the entry point, fixture/stand-ins, source/build identity, action, expected and observed results, evidence, and unverified paths. A detector and archive check can prove a build handoff, but cannot claim the app compiled or launched. A local app with fake external services does not qualify those services. Refuse a requested proof when its required path is unavailable; do not silently substitute a lower layer.

When a failure recurs, prefer removing the invalid state through the product's types or architecture, then an enforceable test/lint rule, then guidance in the feature map. Preserve useful tests; do not generate tests that merely restate the implementation or grow a parallel model of the product inside the driver.

Modes

Pick the one the request needs, and say which:

The prompts that drive each mode are in `references/prompts.md`. Copy them into the repo's notes and adapt them.

Set up

1. Interview the repository, not the person

Answer from the code, and ask only what you cannot observe:

• Surfaces. What does a person touch: a web UI, a desktop app, a CLI or TUI, an API? What do agents touch: a CLI, an API, MCP tools, an embedded app inside a chat host? Both kinds of user belong on the map.
• Run. The documented dev or start command, ports, env vars, seed data, auth. How do you tell it is ready?
• Drive. Existing harnesses first: E2E helpers, Playwright specs, PTY helpers, the app's own CLI. The driver is usually an extraction of what the tests already do.
• Observe. Screenshots, the accessibility tree, the app's saved state, logs, exit codes, files on disk.
• Isolate. Every per-person state folder (home, profile, device account), every offline or preview switch, and every outbound service a verification must not really call (trackers, payments, paid model APIs).
• Escape routes. Everything that can leave the session: the system opener (Finder, the default browser), downloads, native dialogs, the clipboard, file pickers, third-party calls. Also every host the app runs inside: an iframe, a chat app, an extension.
• Registries. Where the product lists its entry points: UI tool catalogues, CLI usage text, API routes, MCP tool definitions. Completeness is checked against these, not against memory.

If the checkout does not build or start, fix that first or report it precisely. A harness written against a broken base teaches wrong steps.

2. Build control-<app>

Follow `references/driver-contract.md`. In short:

• Name it apart from the product's own commands, so verify or test never collide.
• Sessions. A bounded one-shot command may create and clean up its own isolated session. For persistent workflows, up starts a temporary workspace/profile and the app through its own start command; down stops only what this session started. Default sessions are per conversation so agents cannot stop each other. Preview/offline modes and captured outbound calls prevent accidental live operations.
• Doctor. Answers "is this worth driving?": build/source identity, processes alive when required, isolation holding, open gaps. Refuse stale compiled bundles. For direct source execution, record a source digest instead of inventing a build-freshness check.
• Two reads for every result: once as a person sees it (the accessibility tree, the visible state) and once as the product saved it (its own CLI or API). A result is an expect that retries until it holds, and answers expected and actual when it does not.
• Stand-ins at every boundary from the interview. Each one records what the app asked for, and the product's own checks still run: the opener, downloads, dialogs, the clipboard, outbound services, pages to fetch, the host an embedded app runs in.
• Faults for failure paths (a read-only folder, failing fonts, slow or failing reads), reversible and undone by down.
• Evidence: screenshots, recordings, traces, and proof <feature>, which bundles them with the commit and build stamp into a folder that survives cleanup.
• The gaps loop. script <file> --why "…" runs a custom step inside the session and logs it in a gaps ledger. Before the task ends, each gap is either promoted into the driver (preferred) or reported to the tracker. See "No silent scripts".
• The map commands: features, route "<words from a report>", checks <feature> (the tests annotated for it), coverage, replay <feature>, sweep, and issues <feature> against the tracker.

3. Seed the feature map

Write features/README.md and one file per selected area, starting with the change or report at hand. Use the four-section format in `references/feature-file-format.md`:

• a paragraph saying what the area does;
• a header line: Rules: … (where the rule lives), Words people use: … (the words reporters use, which are rarely the code's), and Checks: control-<app> checks <feature>;
• ## Sub-features with stable ids;
• ## How to get to it (user POV), listing every entry point, including the agent ones;
• ## Driving it with control-<app>: preconditions, then bullets. Each bullet names its sub-features and runs commands in order in one continuous session, and every observable result in it is an expect;
• ## Gotchas, most valuable when they say where a rule deliberately does not apply.

4. Wire it into the repository

• Put the skill in one real folder, using the repository's existing convention or .agents/skills/verify-<app>/. Link it from the hosts the repository actually uses, such as .claude/skills/ (Claude Code) or .cursor/skills/ (Cursor).
• Add one paragraph to AGENTS.md: prove behaviour with verify-<app>, never write standalone probe scripts, and update the feature file in the same PR.
• Add annotations like // @verifies <feature>#<sub> above the existing tests that pin a sub-feature, so checks and coverage are generated, not hand-kept.
• Add a CI test for the map (`references/map-checks.md`). It checks each file's structure, that every recipe command exists in the driver, that every annotation resolves, that the index lists every file, and that fixtures exist. It also checks that every entry point in every registry appears in a feature file: report while mapping, assert once complete.

5. Prove it before handing it over

Replay every seeded recipe in fresh state and check the proof survives cleanup. State the layer each replay reached. A component-only first slice is useful when labelled as such, but its unproven application paths remain gaps; a claimed application harness that was never run end to end is a draft.

Expand the map

To map a whole product, run two background mappers, each owning a group of areas. Keep every driver change in the main session. The mappers drive, write files, replay each twice, annotate tests and prepare findings. They send the main session the capabilities they need, most important first, and keep working with script until those land. The main session:

• promotes capabilities within minutes and tells the mapper;
• reviews every finding before it goes to the tracker (next section);
• adds the files to the index;
• ends with a full sweep.

Prompt 7 in `references/prompts.md` is the brief. Map the agent surfaces too: CLI commands, API and MCP tools, the security boundary (host and origin checks, loopback only), sign-in and account paths, and every promise in the onboarding prompt people paste.

Verify a change

1. control-<app> doctor (build if stale), then issues <feature> when a tracker is configured: a fix awaiting review is something to confirm, and an open issue may be exactly what you are about to see.
2. Read the feature file, and run what checks <feature> lists, fastest first.
3. Start the session if needed, or run the bounded recipes. Drive every supported entry point the change touches, including relevant success, cancel, empty, error and persistence paths. Confirm each side effect through the second read; report unsupported paths explicitly.
4. Retain the automatically saved proof or run proof <feature> --note "…", and cite the folder.
5. Changed behaviour means the feature file and its recipe change in the same PR, and replay passes.

The proof bar: