nearform/lastlight/qa-test

QA Test

Drive a target through a sequence of steps and report step-level pass/fail with evidence for each step. The deliverable is a QA report, not a single verdict — partial coverage with documented failures is the expected output.

This skill uses the building skill for installing dependencies and running the repo's build/run commands in the sandbox.

Scope (text-evidence path). In the default text phase the sandbox agent

has bash, file read, and the github tools — no browser, no screenshots. QA a

CLI directly, or a web service by starting its dev-server in the

background and exercising it with curl (status codes, response bodies,

headers). Capture stdout/stderr, exit codes, and log excerpts as per-step

evidence. Driving a real browser UI and attaching screenshots is the separate

browser-qa skill, staged only in the gated browser phase running on the

docker QA image (Playwright + Chromium). If that skill isn't in your

catalogue, mark a step that genuinely needs rendered-UI interaction

BLOCKED with the reason rather than faking a result — the browser phase

will pick it up when available.

Parse the target

The Context block gives you what to test and (usually) a PR/issue. The target may be:

• A CLI command (my-cli --flag, a subcommand) → run it directly.
• A local web service (an app the repo serves on a port) → start it, hit it

with curl.

• A PR reference + focus area → read the diff and infer the flow that

exercises what changed.

• A free-text flow ("create a project, then list it") → infer concrete steps.

When a PR/issue is given, read its description + diff (from the local checkout — see the building skill) to decide what's worth testing.

Define the steps

If the request specifies steps, use them. Otherwise design a reasonable flow:

• CLI: invoke with valid input → assert output/exit code → invoke an edge

case (bad flag, missing arg) → assert it's handled → done.

• Web service: start the server → wait until it's ready → hit the primary

endpoint(s) → assert status + body → hit an error case → assert handling → stop the server.

State the steps and their success criteria before running. If the flow or the success criteria are genuinely ambiguous and you can't infer them from the diff, say what's unclear in the report rather than inventing a pass.

Run each step

Follow the building skill to install and build first. Then run each step in order, capturing evidence (command + stdout/stderr + exit code) at every step.

• Record each step's result as you go — PASS, FAIL, or BLOCKED.
• On failure, continue to the next step for maximum coverage — unless the

failure blocks everything downstream (e.g. the server never starts, or a login step fails). If it's a hard blocker, record it, mark the rest BLOCKED, and stop.

• Treat a wrong result as a real finding, not your mistake. Don't edit the code

under test to make a step pass.

Report

Produce your report as your final message in this shape — the workflow posts it for you (don't github_add_issue_comment yourself; that would double-post). Keep it concise: a one-line environment summary, the results table (terse Evidence cells), the failures, and a short coverage note — not a step-by-step narration.

## QA Test: <target>
 
**Environment:** <branch / commit, package manager, how the app was run>
 
### Results
| Step | Status | Evidence |
|------|--------|----------|
| <step description> | PASS / FAIL / BLOCKED | <command + key output / exit code> |
 
### Issues found
- <each FAIL with the expected vs observed behaviour and the evidence>
 
### Coverage
<what was tested, and what was not tested and why (e.g. a UI-only step that the
text-evidence sandbox can't drive).>

Every defined step must have a row and a result. Never report a flow as "passed" with steps you didn't actually run — list them as untested instead.