itamarzand88/cli-anything-web/capture

Traffic Capture (Phase 1)

Assess the site, then capture comprehensive HTTP traffic. This skill combines site assessment with full traffic recording in a single browser session.

Copy this checklist and check off items as you complete them:

Phase 1 Progress:
- [ ] Prerequisites: playwright-cli available, checkpoint restored if one exists
- [ ] Step 1: Browser open on target URL
- [ ] Step 2: Fingerprint run + auth handled + profile classified + assessment.md written
- [ ] Step 3: Full capture — exploration targets met for the profile
- [ ] Step 4: Trace parsed, validate-capture.py PASSED (the gate)
- [ ] Step 5: Browser closed, checkpoint + phase-state marked complete

CRITICAL EXECUTION RULES

NEVER use `run_in_background: true` for ANY playwright-cli command.

All playwright-cli commands must run in the foreground with appropriate timeouts.

Background execution causes task ID tracking failures — the command completes

before you can read the output. See references/playwright-cli-commands.md

for the timeout table.

NEVER use `eval` for complex expressions. eval fails silently on ternaries,

comma operators, and multi-branch logic with "not well-serializable" errors.

Use run-code instead. See references/framework-detection.md for details.

ESM context — no `require()`. run-code uses ESM. Use await import('fs')

instead of require('fs'). See references/playwright-cli-commands.md.

Prerequisites (Hard Gate)

Do NOT start unless:

• [ ] playwright-cli is available (npx @playwright/cli@latest --version)
• [ ] Target URL is known

Default capture method: playwright-cli tracing (standard workflow below).

Optional --mitmproxy mode

When the --mitmproxy flag was passed to /cli-anything-web (or the capture needs untruncated bodies / enhanced metadata), follow this workflow with the step substitutions in references/mitmproxy-mode.md — Steps 1, 3, and 4 differ; everything else is identical. When playwright-cli itself is unavailable (e.g., no Node), fall back to chrome-devtools-mcp via launch-chrome-debug.sh — see HARNESS.md Tool Hierarchy.

Public API Shortcut

If the target site has a documented public REST/JSON API (e.g., Hacker News Firebase API, Dev.to API, Reddit API, Wikipedia API), browser capture is optional:

1. Probe the API endpoints directly with httpx or curl
2. Save responses as <app>/traffic-capture/raw-traffic.json
3. Skip to Phase 2 (methodology)

This applies when:

• API docs exist (OpenAPI/Swagger, developer docs page, /api/ prefix)
• The API is publicly accessible without browser-specific auth
• Endpoints return JSON (not HTML)

If unsure whether a public API exists, proceed with browser capture as normal.

Resume from Checkpoint

Before starting, check if a previous capture session exists:

python ${CLAUDE_PLUGIN_ROOT}/scripts/capture-checkpoint.py restore <app>

If a checkpoint exists, read the guidance field and resume from the last completed step instead of starting over. This prevents duplicate work when sessions are interrupted.

Step 1: Setup

# Create output directory
mkdir -p <app>/traffic-capture
 
# Clear any stale sessions
npx @playwright/cli@latest kill-all 2>/dev/null || true
 
npx @playwright/cli@latest -s=<app> open <url> --headed --persistent
# Note: heavy SPAs (Next.js, React) may show "TimeoutError: page._snapshotForAI" on open.
# This is non-fatal — verify with: npx @playwright/cli@latest list
#
# IMPORTANT — "Browser opened with pid..." in command output means the daemon
# RE-ATTACHED to the existing browser, NOT that a new session was created.
# Do NOT re-navigate or restart when you see this. The session is still open.

`--mitmproxy` mode: use the Step 1 substitution in

references/mitmproxy-mode.md (start the proxy, then open through it).

Do NOT ask the user to log in yet — Step 2 will determine if auth is needed.

Step 2: Site Fingerprint (Single Command)

Run the all-in-one site fingerprint command instead of individual eval calls. This is faster, more reliable, and detects framework + protection + iframes + auth requirements in one shot.

Use the script file — multi-line JS with arrow functions and optional chaining fails in playwright-cli's single-line command parser. The script file approach has been tested and works reliably:

npx @playwright/cli@latest -s=<app> run-code "$(grep -v '^\s*//' ${CLAUDE_PLUGIN_ROOT}/scripts/site-fingerprint.js | tr '\n' ' ')"

IMPORTANT: The site-fingerprint.js script must be loaded via the command

above. Do NOT copy-paste the JS inline — it will fail with SyntaxError.

The grep -v strips comments and tr joins lines for single-line execution.

Interpret fingerprint results

The fingerprint returns four groups: framework, protection, auth, iframes. Map each true flag to the next action:

Claude-facing shortcuts:

• googleBatch: true → generate rpc/ subpackage (batchexecute protocol).
• cloudflareManagedChallenge: true → tier 4 (camoufox) is required; curl_cffi alone will fail.
• awsWaf: true → capture aws-waf-token cookie; use curl_cffi for GraphQL, cookie-only for SSR.
• akamai: true or datadome: true → 1–2 s delays between requests are mandatory.
• serviceWorker: true → note in assessment.md; generated CLI uses service_workers="block".
• iframeCount > 0 → re-run the fingerprint inside the iframe. Google Labs apps (Stitch / MusicFX / ImageFX) follow this pattern — parent has WIZ_global_data, iframe has the real app.

Note: snapshot and click <ref> auto-resolve iframes. Only drop down to run-code for iframe interaction when built-in commands fail.

Auth detection (BEFORE exploration)

Check the fingerprint auth fields:

If auth is needed:

1. Tell the user: "This site requires login. Please log in in the browser window."
2. Wait for user confirmation
3. Save auth state and tighten permissions (CLAUDE.md mandates chmod 600):

npx @playwright/cli@latest -s=<app> state-save <app>/traffic-capture/<app>-auth.json
chmod 600 <app>/traffic-capture/<app>-auth.json

If NO auth is needed: Skip directly to Step 2b.

2b. Classify Site Profile

Based on fingerprint results AND what you see in the UI, classify the site:

2c. Quick API Probe (Force SPA Navigation Trick)

Start a SHORT trace, click 3-4 internal links, stop. This reveals hidden API endpoints that SSR hides on initial page load.

npx @playwright/cli@latest -s=<app> tracing-start
npx @playwright/cli@latest -s=<app> click <internal-link-1>
npx @playwright/cli@latest -s=<app> click <internal-link-2>
npx @playwright/cli@latest -s=<app> click <internal-link-3>
npx @playwright/cli@latest -s=<app> tracing-stop
 
# Quick parse to see what endpoints appeared (saved alongside the full capture
# so it survives the session — don't output to /tmp).
python ${CLAUDE_PLUGIN_ROOT}/scripts/parse-trace.py .playwright-cli/traces/ --latest \
  --output <app>/traffic-capture/probe-traffic.json

This probe trace is separate from the full capture in Step 3 — Step 3 will start a fresh trace that overwrites the .network file in .playwright-cli/traces/. The parsed probe-traffic.json is kept in traffic-capture/ so it stays available for cross-referencing during Step 4.

Check the probe results — what API patterns did you find? See references/api-discovery.md for the priority chain and decision tree.

2d. Write Assessment Summary

Create <app>/traffic-capture/assessment.md to consolidate all findings:

# Site Assessment: <app>
 
-**URL**: <url>
-**Framework**: <detected framework or "none/custom">
-**Protocol**: <REST / GraphQL / batchexecute / HTML scraping / hybrid>
-**Protection**: <none / cloudflare / captcha / aws-waf / etc.>
-**Auth required**: <yes (type: Google SSO / cookie / JWT / API key) / no>
-**Iframes**: <yes (N frames, app in frame N at <url>) / no>
-**Site profile**: <Auth+CRUD / Auth+Generation / Auth+Read-only / No-auth+CRUD / No-auth+Read-only>
-**Capture strategy**: <API-first / SSR+API hybrid / batchexecute / HTML scraping / protected-manual>
-**Key observations**: <any quirks, localized UI, rate limits, special patterns>

Step 3: Full Traffic Capture

Now do the comprehensive capture based on what Step 2 revealed.

# Optional: Start HAR recording alongside trace for standard-format capture
# HAR files enable mitmproxy2swagger (auto OpenAPI spec) and third-party analysis tools
npx @playwright/cli@latest -s=<app> run-code "async page => {
  await page.context().routeFromHAR('<app>/traffic-capture/capture.har', {
    update: true,
    updateContent: 'embed',
    updateMode: 'full'
  });
  return 'HAR recording started';
}"
 
# Start fresh trace for full capture (note the trace ID from output!)
npx @playwright/cli@latest -s=<app> tracing-start
# Output: "trace-<ID>" — record this ID
 

`--mitmproxy` mode: skip tracing-start and HAR recording — the proxy

is already capturing (see references/mitmproxy-mode.md).

HAR recording is optional but recommended. It produces a standard HAR file

alongside the trace. This enables mitmproxy2swagger to auto-generate an

OpenAPI spec: pip install mitmproxy2swagger && mitmproxy2swagger -i capture.har -o api-spec.yaml -p <base-url>

The HAR file is saved when the browser context is closed (Step 5).

Exploration by site profile

Use the concrete targets in references/exploration-checklists.md for the profile identified in Step 2b. Each profile has an explicit entry count, distinct-path count, and WRITE-op target that validate-capture.py (Step 4) will enforce. Minimum bar across all profiles:

• ≥ 15 entries, ≥ 3 distinct URL paths, protocol ≠ unknown
• ≥ 1 WRITE op (unless the site is genuinely read-only — pass --read-only to the validator)
• < 50% error rate (dominant 4xx/5xx means auth or rate-limit failure)

Pacing for protected sites

If any of cloudflare, cloudflareManagedChallenge, akamai, datadome, awsWaf, or rateLimit fired in the fingerprint, leave 1–2 s between clicks / form submits. Faster exploration triggers per-IP challenges within ~30 requests and corrupts the trace.

General interaction rules