itamarzand88/cli-anything-web/methodology

CLI-Anything-Web Methodology (Phase 2)

Analyze captured traffic, design the CLI command structure, and implement the complete Python CLI package. This skill owns the core transformation from raw HTTP traffic to a production-ready CLI.

Copy this checklist and check off items as you complete them:

Phase 2 Progress:
- [ ] Prerequisites: raw-traffic.json exists (+ auth state if the site needs auth)
- [ ] Step A: traffic analyzed, protocol identified, <APP>.md written
- [ ] Step A: api-spec.json written (every endpoint cites raw-traffic.json evidence)
       and passes `cli-web-devkit spec validate`
- [ ] Step B.0: scaffolded via scaffold-cli.py (.manifest.json present)
- [ ] Step B: client endpoint methods implemented from the spec
- [ ] Step B: command modules implemented + registered, REPL help in sync
- [ ] Smoke check passed (no protocol leaks), phase-state marked complete

Prerequisites (Hard Gate)

Do NOT start unless:

• [ ] raw-traffic.json exists (with WRITE operations, or read-only GET-only traffic)
• [ ] Auth state was captured during Phase 1 (if the site requires auth)

If raw-traffic.json is missing or has no WRITE operations, invoke the capture skill first. If Phase 1 state shows failed, follow skills/shared/RECOVERY.md §phase-state Check Failures before re-running.

Exception for read-only sites: If the site is genuinely read-only (search engine, dashboard, analytics viewer with no create/update/delete), the trace may contain only GET requests. In this case, note "read-only site — no write operations" in <APP>.md and proceed. The generated CLI will have read-only commands (list, get, search) but no create/update/delete commands. This is valid.

No-auth sites: If the target site requires no authentication (public API, no login needed), the "Auth state captured" prerequisite does not apply. Note "no-auth site" in <APP>.md and proceed.

Step A: Analyze (API Discovery)

Goal: Map raw traffic to a structured API model.

Process:

1. Read `traffic-analysis.json` first (if it exists alongside raw-traffic.json).

This file is auto-generated by parse-trace.py or mitmproxy-capture.py → analyze-traffic.py and contains pre-detected protocol type, auth pattern, endpoint grouping, GraphQL operations, batchexecute RPC IDs, and suggested CLI commands. Use it as a starting point — verify its findings and fill in anything marked "unknown" by reading raw-traffic.json manually.

Enhanced analysis (present only when captured via mitmproxy): request_sequence (timeline-ordered requests with auth-flow detection), session_lifecycle (cookie inventory, auth-cookie identification, session pattern), and endpoint_sizes (response-size classification). If these are missing (has_timestamps: false), the capture came from the default trace path — rely on manual analysis for sequence/session detail.

If traffic-analysis.json doesn't exist, run the analyzer: ``bash python ${CLAUDE_PLUGIN_ROOT}/scripts/analyze-traffic.py \ <app>/traffic-capture/raw-traffic.json --summary ``

1. Parse raw-traffic.json (for details the analyzer couldn't extract)
2. Group requests by base path (e.g., /api/v1/boards/, /api/v1/items/)
3. For each endpoint group, identify:

• HTTP method (GET/POST/PUT/DELETE/PATCH)
• URL pattern (extract path parameters like :id)
• Query parameters and their types
• Request body schema (JSON fields, types, required/optional)
• Response body schema
• Authentication method (Bearer token, cookie, API key)
• Rate limiting signals (429 responses, retry-after headers)

1. Identify RPC protocol type -- classify the API transport:

This determines client architecture in Step B -- REST uses simple client.py, non-REST protocols need a dedicated rpc/ subpackage with encoder/decoder/types.

1. Detect data model:

• Entity types (boards, items, users, projects...)
• Relationships (board has many items, item belongs to board)
• ID formats (UUID, numeric, slug)

1. Detect auth pattern:

• Cookie-based sessions
• Bearer/JWT tokens
• OAuth refresh flow
• API key headers
• Browser-delegated auth: tokens embedded in page JavaScript (e.g., WIZ_global_data),

not in HTTP headers. Requires CDP for initial cookies, HTTP for token extraction. See references/auth-strategies.md "Browser-Delegated Auth" section.

• No auth / public access: fully public API, no login required. CLI may

optionally support API key auth for write operations (e.g., dev.to).

1. Write <APP>.md -- software-specific SOP document

1. Write agent-harness/api-spec.json -- the machine-readable API spec.

Every endpoint MUST carry an evidence field citing its captured traffic entry (raw-traffic.json#<index>) — never invent endpoints (this is the structural enforcement of the RPC-ID verification rule). Schema and validator:

``bash cli-web-devkit spec validate <app>/agent-harness/api-spec.json ``

Downstream consumers: client method implementation (Step B), the gap-analyzer (cli-web-devkit gaps), and the traffic-fidelity review in Phase 4 (spec-vs-traffic becomes a deterministic diff).

Output: <APP>.md (human SOP) + api-spec.json (machine spec, validated).

References: traffic-patterns.md, google-batchexecute.md, ssr-patterns.md

Step B: Implement (Code Generation)

Study Existing CLIs First (Critical for Accuracy)

Before implementing, read an existing CLI that uses the same protocol as your target. These are battle-tested implementations that solved the same problems you'll face.

How to use reference CLIs:

1. Read the reference CLI's core/client.py — understand the request/response pattern
2. Read core/auth.py — copy the login_browser() pattern exactly for Google apps
3. Read core/rpc/ (for batchexecute) — understand encoder/decoder, DO NOT reinvent
4. Read commands/ — see how Click commands are structured, how --json works
5. Read utils/helpers.py — see handle_errors(), _resolve_cli(), repl patterns

For batchexecute apps specifically, the notebooklm CLI is your bible:

• Copy the encoder/decoder architecture (don't reinvent the batchexecute wire format)
• Copy the auth token extraction pattern (CSRF, session ID, build label)
• Copy the cookie domain priority logic (critical for Israeli/international users)
• Adapt the RPC method IDs and param structures to your target app

The agent implementing the CLI MUST read these files before writing code. Use the Agent tool to dispatch a research agent that reads the reference implementation while you design the command structure.

Design Before You Code

Before writing any code, note the command structure in <APP>.md (10 minutes max):

• Map each API endpoint group to a Click command group:
• /api/v1/boards/* → boards command group
• /api/v1/items/* → items command group
• Map CRUD operations to subcommands (GET list → list, GET single → get,

POST → create, PUT/PATCH → update, DELETE → delete)

• Note auth design: auth login, auth status, auth refresh; credentials at

~/.config/cli-web-<app>/auth.json

• Note REPL design: bare command enters REPL, branded banner via repl_skin.py

Goal: Generate the complete Python CLI package.

Package Structure

See HARNESS.md "Generated CLI Structure" for the complete package template. Key points: cli_web/ namespace (NO __init__.py), <app>/ sub-package (HAS __init__.py), core/, commands/, utils/, tests/ directories.

Step B.0: Scaffold Core Modules

Run the scaffold generator script (v2 — Jinja2 templates, requires pip install jinja2) to create all boilerplate files:

python ${CLAUDE_PLUGIN_ROOT}/scripts/scaffold-cli.py <app>/agent-harness \
  --app-name <app> \
  --protocol <rest|graphql|html-scraping|batchexecute> \
  --http-client <httpx|curl_cffi> \
  --auth-type <none|cookie|api-key|google-sso> \
  --resource <name> [--resource <name> ...] \
  [--has-polling] [--has-context] [--has-partial-ids]

This renders exceptions.py, client.py skeleton, the unified auth.py (google-sso handled via a template conditional), helpers.py, config.py, output.py, the CLI entry point with REPL, one commands/<resource>.py per --resource flag, setup.py, conftest.py, test_e2e.py skeleton, README/SKILL skeletons, repl_skin.py, and (for batchexecute) the rpc/ subpackage. It also writes .manifest.json (template version + profile) at the harness root — keep it; fleet tooling depends on it. See skills/boilerplate/SKILL.md for the template → output map and per-profile flag recipes.

Fallback: If the script is unavailable, follow

skills/shared/RECOVERY.md §scaffold-cli.py Unavailable — adapt from the

newest generated CLI (e.g., capitoltrades/agent-harness/), do NOT

reconstruct boilerplate from memory.

After scaffolding, review the generated files and customize client.py with actual endpoint methods from <APP>.md.

Implementation Rules

All rules below are DEFINED in skills/shared/CONVENTIONS.md — this section tells you when to apply them during implementation.

• `exceptions.py` -- implement first. Required hierarchy and error-code mapping: CONVENTIONS.md §Exception Hierarchy. Complete code: references/exception-hierarchy-example.py.