nearform/lastlight/verify

Verify

Test a specific claim about behaviour and report whether the evidence supports or refutes it. The deliverable is a verdict with evidence, not an opinion.

This skill uses the building skill for installing dependencies and running the repo's build/test commands in the sandbox.

Scope (text-evidence path). In the default text phase the sandbox agent

has bash, file read, and the github tools — no browser, no screenshots, no

video. Prove claims with what bash can capture: test output, command

stdout/stderr, curl against a dev-server you start, log excerpts, file

contents, exit codes. Driving a real browser and attaching screenshots is the

separate browser-qa skill, which is staged only in the gated browser phase

running on the docker QA image (Playwright + Chromium). If that skill isn't in

your catalogue and a claim genuinely can only be shown in a rendered UI, say

so under INCONCLUSIVE rather than guessing — the browser phase will pick

it up when available.

Ground rule — investigator, not advocate

Your job is to find out whether the claim is true, not to make it look true. A conclusive "this is broken" with clear evidence is exactly as valuable as a "this works". Specifically:

• Never fabricate, hardcode, mock, or stage evidence to match an expected

outcome. Do not edit the code under test to make a result appear.

• If the behaviour you observe contradicts the claim, that is the result —

report it as REFUTED with the evidence inline. Don't bury it.

• Don't retry a failing test hoping for a different answer. Retry only after

changing the environment or procedure (wrong branch, missing build step), and only when you have reason to believe the test setup — not the code — was wrong. One unexpected result that reproduces is a finding.

Parse the target

The Context block gives you the claim and (usually) a PR or issue. The claim may arrive as:

• A direct claim — "ESC cancels streaming in bash mode", "the --fork flag

creates a new session".

• A PR reference + claim — verify the claim against that PR's head.
• A PR reference only — read the PR description + diff and identify the most

important, most testable claim it makes; state which claim you chose.

When a PR/issue is given, read its description and diff for context (from the local checkout — see the building skill's workspace notes; don't pull the diff through the API).

Decide what would convince a skeptic

Name the single behaviour to observe and the evidence that settles it:

• Functional claim (a feature works, a flow completes) → run it and capture

the output / exit code / resulting state.

• Regression / fix claim ("no longer clears the screen", "stops erroring") →

show the before and after: reproduce on the base ref, then show it gone on the head ref. Both states must appear in the evidence — an off-camera "it's fixed" doesn't count.

• Output/encoding claim → capture the exact bytes/text (xxd, od -c, raw

stdout) rather than describing them.

Run the test

Follow the building skill to install and build. Then exercise the claim with the minimal sequence that demonstrates it, capturing evidence at the decisive step. Start a dev-server in the background and curl it when the claim is about a running service; run the CLI directly when it's a CLI claim; run the targeted test when the repo already encodes the behaviour as a test.

If the environment blocks a clean test (missing dependency the sandbox can't provide, build failure you can't resolve, infra that won't run), capture what blocked it — that drives an INCONCLUSIVE result, not a guess.

Report

Produce your report as your final message in this shape — the workflow posts it for you (don't github_add_issue_comment yourself; that would double-post). Keep it concise: a one-line environment summary, the decisive evidence (commands + key output, not every line), and a one-paragraph conclusion — not an exhaustive walkthrough.

## Verify: <claim>
 
**Environment:** <branch / commit, package manager, how the app was run>
 
### Evidence
<commands run + their captured output — fenced. Include both states for a
before/after claim.>
 
### Conclusion
**CONFIRMED** | **REFUTED** | **INCONCLUSIVE**
 
<one paragraph: what the evidence shows and why it settles the claim.>

• REFUTED — state the expected behaviour (from the claim), the observed

behaviour (from the evidence), include the evidence inline, and note any environmental factors (branch, commit, OS). This is a valuable finding, not a failed run.

• INCONCLUSIVE — report exactly what blocked the test and what would be

needed to resolve it. Do not guess at the outcome.

Do NOT

• Retry a failing test more than once without changing the environment/procedure.
• Hardcode expected output, mock responses, or edit the code under test to force

a result.

• Assume unexpected behaviour means you made a mistake — it may be a real bug.
• Omit evidence that contradicts the claim.