wochap/webscoop/webscoop-use-recipe

Use a webscoop recipe

Webscoop replays recorded scrapers ("recipes") in the user's browser and prints the rows as JSON. Use it to get data from a site instead of fetching pages yourself.

1. Discover recipes

Run:

webscoop recipes --json

It prints a JSON array (nothing when there are no recipes). Each entry has:

• name: the recipe name to pass to webscoop run.
• description (optional): what the recipe scrapes.
• url: the start URL, possibly with {variable} placeholders.
• vars: variables, each with name, optional default, and optional description.
• tables: each with name, optional description, and fields (name, type).
• captures (optional): each with name and optional description. A capture is a table of request URLs the page loaded, such as video playlist URLs that never appear on the page itself.
• agent (optional, true): the recipe has an agent step, which drives the page with a language model.
• downloads (optional, true): the recipe saves files and has a downloads table.

2. Choose a recipe

Match the user's request against each recipe's description, its tables' descriptions and field names, its captures' names and descriptions, and its variables' descriptions. The URL helps when descriptions are missing.

When no recipe fits, use the webscoop-control-tab skill to interactively inspect and scrape the page. Alternatively, suggest that the user records a recipe with webscoop record <url> (the user runs it; it needs them in the browser).

3. Fill the variables

Pass each variable as --var name=value. Supply every variable that has no default; ask the user when the request does not give a value. Variables with a default may be omitted.

4. Run it

webscoop run <recipe> --var name=value --jsonl --quiet

Add --table <name> when only one table of a multi-table recipe is needed.

Stdout has one JSON row per line. Each row has the table's fields plus _page (page number) and _index (position on the page). When the recipe has several tables and --table is not given, each row also has _table, the table name.

A capture is a table named after it. Its rows are request URLs seen while the page loaded, each with url, method, status, mime, and frame (the URL of the frame that made the request), plus _page and _index. Read url for the data, such as a video's playlist URL.

A downloads table lists the files a run saved. Each row has file, the absolute path of the saved file, plus name, url, bytes, _page, and _index. A recipe that only downloads has no other table, so its output is just those rows; read file and give the user that path.

A recipe with agent: true hands a goal to a language model that acts on the page, so a run takes longer, may cost model calls, and the model may not reach the goal. Run it like any other recipe; do not retry it in a loop. Before submitting a form or clicking a consequential button (such as Pay, Order, Delete, or Send), an agent step pauses to ask a person to allow it. --submit stop is a dry run that stops before a submit and does not reach the goal. NEVER pass --submit allow on your own; only use it when the user explicitly asked for the submit to go ahead.

Example: the user asks for the PDF behind a URL and the catalog lists grab-pdf, described "download the PDF a page offers", with variable url and agent: true. Run the command below and read file from the downloads row:

webscoop run grab-pdf --var url=<url> --jsonl --quiet

Example: the user asks for the stream URL of lesson 42 and the catalog lists lesson-video with variable id and capture videos described "HLS playlist URL". Run the command below and read url from the videos row:

webscoop run lesson-video --var id=42 --jsonl --quiet

Example: the user asks "search cats" and the catalog lists bing-search, described "Bing web search results for a query", with variable query described "search terms":

webscoop run bing-search --var query=cats --jsonl --quiet

5. Read the exit code

• 0: success. Use the rows.
• 1: error. Report it to the user with the stderr text.
• 2: the browser needed the user (login, captcha, or a submit confirmation) and nobody answered in time. When caused by a submit confirmation, ask the user whether to allow the submit; do not pass --submit allow on your own. Then retry the command.
• 3: the page changed and a required field or capture was not found. Tell the user the recipe needs repair with webscoop edit <recipe>. When the recipe has agent: true, exit 3 instead means the model gave up on that page or did not reach the goal (for example, no file was downloaded); tell the user that, since it is not a recipe repair case.

Parallel runs and pauses

You may start several runs at once. Webscoop queues or shares them, so no extra coordination is needed. A run may pause while the user answers a prompt in the browser; wait for it to finish.

Never

• Never edit recipe files.
• Never run webscoop record, webscoop edit, or webscoop daemon stop yourself. These are for the user.
• Never pass --submit allow unless the user explicitly requested it.