cyborgtests/playwright-reports-server/playwright-reports

Playwright Reports

Access to the Playwright Reports Server. Every command returns JSON; error messages, LLM analyses, and cluster members pass through verbatim.

Use this skill when the user asks why a test is failing, what's flaky, what changed between runs, or any aggregate question over a time window.

Skip this skill for writing new tests, modifying playwright.config.ts, running tests locally, or any task that doesn't touch historical run data.

The CLI is read-only. Write commands (analysis-submit, summary-submit, feedback) live in authoring.md and are loaded only when the user explicitly asks to author or dissent.

Quick reference

All list-returning commands carry total + hasMore. Defaults: report list/test search/report compare = 20, cluster list = 10, test history = 20 (max 50), regression list = 25 (max 200). For exact flags run pwrs-cli help <group> or pwrs-cli <command> --help.

Project identification

Most commands accept --project to scope data. If the user's query doesn't name a project, identify the project first - don't guess or omit it silently.

pwrs-cli project list    # → { "projects": ["head:main", "staging:main", "e2e:local:sh"] }

project list returns all known project names - use exact string values from this list in --project. Projects are named by the reporter config, not by you.

When to pass `--project`:

• User names a project ("how's staging?") → use the matching string from project list (e.g. staging:main).
• User doesn't name a project and only one project exists → use it implicitly, don't ask.
• User doesn't name a project and multiple projects exist → ask which project they mean, or if the question is aggregate ("overall health"), omit --project to get cross-project results. Commands without --project return data across all projects.
• `PWRS_PROJECT` env var is set → it serves as the default; explicit --project still wins. You can check if it's set via pwrs-cli config get.

Cross-project (`--project` omitted or `all`):

• test find / test search return results from all projects - the project field on each row tells you which project it belongs to.
• project summary without --project produces a cross-project digest. Per-project summaries are more specific; prefer them when the user asks about one area.
• report list / stats / cluster list aggregate across projects when --project is omitted.

Don't assume project names. Project names vary by server (e.g. chromium, staging:main, head:main, nightly). Always start from project list output.

Worked examples

User: "Why is the checkout test failing?"

pwrs-cli project list                   # → identify available projects first
pwrs-cli test find "checkout"           # → matches[]; each row has a `project` field
# If matches span multiple projects, ask the user which one, or pick the project they're working in.
pwrs-cli test brief <testId>            # → signals + latestFailure + llmAnalysis + cluster

If cluster is non-null, fix at cluster.anchor, not the test. If llmAnalysis: null, fall back to pwrs-cli test failure-context <testId> --report-id <reportId> and reason from the evidence envelope yourself.

User: "How did staging do yesterday?"

pwrs-cli project list                   # → confirm exact project name (e.g. "staging:main")
pwrs-cli report list --project "staging:main" --from 2026-06-08 --to 2026-06-08
# → reports[0].reportId
pwrs-cli report brief <reportId>        # compact: 5 KB even on a 50-failure report
pwrs-cli report summary <reportId>      # → verdict ('isolated' | 'clustered' | 'widespread' | 'systemic')

Read summary.llmSummaryStructured.verdict first - widespread/systemic means the run is broken at a layer above any single test (infra, fixtures, deploy); don't drill into individual tests in that case.

Workflows

Discovery (no ID yet)

pwrs-cli project list                                       # what projects are tracked
pwrs-cli tag list [--project p]                             # what tags exist (pairs with report list --tags)
pwrs-cli category list                                      # valid --failure-category values
 
pwrs-cli report list [--project p] [--from --to] \
    [--pass-rate failing|below-threshold|passing|all] \
    [--search text] [--tags a,b]                            # reports in a window
 
pwrs-cli test search [--project p] [--tier flaky|critical|stable] \
    [--status quarantined|not-quarantined] \
    [--failure-category c] [--sort slowest] \
    [--from --to] [--search text]                           # tests matching open-ended filters
 
pwrs-cli stats [--project p] [--from --to] [--failed-only]  # numeric digest
pwrs-cli project summary [--project p]                      # LLM-written digest + verdict
 
pwrs-cli cluster list [--project p] [--from --to] \
    [--include-resolved]                                    # failure clusters (active only by default)

Drill-down (you have an ID)

# Test:
pwrs-cli test brief <testId>                                # everything we know about this test
pwrs-cli test history <testId> [--limit N]                  # per-run history + signatureGroups
pwrs-cli test analysis <testId>                             # full LLM markdown (use when brief.llmAnalysis lost a section)
 
# Test + report:
pwrs-cli test failure-context <testId> --report-id <r>      # fresh prompt + typed evidence envelope (no LLM call)
pwrs-cli test analysis-prompt <testId> --report-id <r>      # verbatim prompt + response from the last completed task
 
# Report:
pwrs-cli report brief <reportId>                            # compact: stats + clusterSummary + sample failures
pwrs-cli report brief <reportId> --with-failures            # full brief per failed test (~100 KB on a 50-failure report)
pwrs-cli report summary <reportId>                          # persisted LLM summary + verdict
pwrs-cli report compare <reportIdA> <reportIdB>             # diff buckets (accepts `latest` / `prev` keywords)
 
# Cluster:
pwrs-cli cluster brief <clusterId>                          # full member roster + per-member brief

--project is optional for test brief / test history / test analysis* - the server resolves the canonical (fileId, project) lane from the test's most recent run. Pass it only when auto-resolution picks the wrong lane (e.g. an obsolete project sharing a testId).

From a file path or stack frame

pwrs-cli test from-file tests/checkout.spec.ts              # all tests in that file
pwrs-cli test from-file tests/checkout.spec.ts:200          # sorted by proximity to line 200

failure-context evidence envelope

test failure-context returns { markdown, segments, evidence, attachments, heuristicCategory, meta }. The evidence envelope is the typed shape the LLM queue uses:

errorMessage, stackTrace, testSourceFrame (codeframe), stepTree, pageSnapshot (ARIA), stdout, stderr, testMeta, gitCommit, ciBuild, gitDiff, environment, consoleEvents, networkEvents, actionLog.

Use this when you want to reason from raw signals (faster + cheaper than the LLM-mediated path) or when no analysis exists yet.

Interpreting signals

Decision rules - apply top-to-bottom. Field shapes live in the workflow section above.

1. `feedback` present → trust the team's note, override everything else.
2. `signals.quarantined: true` → skip; the team already marked it broken.
3. Report-level verdict first. Before assuming it's your test, read report summary <reportId> → summary.llmSummaryStructured.verdict. widespread/systemic means fix the run (infra, fixtures, deploy), not the test. For multi-day patterns, escalate to project summary → summary.structured.verdict.
4. `regression` present → a tracked green→red transition. regression.regressedAtCommit + regression.lastGreenCommit define the suspect range; frame the fix around what landed there, not chronic flake. On cluster brief, regressionContext.sharedRegressionCommit (when set) means ≥80% of the cluster regressed at one commit.
5. `cluster` present → fix at cluster.anchor, not the test. Every member likely resolves together.
6. `signals.flakyTier` → critical urgent, flaky investigate, stable ignore.
7. `signals.signatureOccurrenceCount > 1` with a stable `signature` → recurring failure. Counts prior runs of the same latestFailure.signature, not total. Distinct from regression: a signature can recur without a green→red transition.
8. `signals.signatureFirstSeen` → likely correlate with deploys / PRs.
9. `llmAnalysis.rootCause` present → start from this hypothesis. Use test analysis for the unmodified markdown, test analysis-prompt to inspect what the model saw.
10. `llmAnalysis: null` with a failing run → analysis hasn't been generated. Pull test failure-context and reason from evidence yourself rather than waiting.

Cluster anchors

cluster.anchor is the deterministic fix target, discriminated by anchor.kind: