decodo/agent-skills/decodo-web-scraping

Decodo web scraping

Decodo is a web scraping API that returns clean web data without you managing proxies, browsers, or anti-bot logic. This skill teaches you to reach for it through the `decodo` CLI (primary), the hosted MCP server (when there is no shell), or the raw HTTP API (fallback).

When to use Decodo

Use Decodo instead of curl / requests / BeautifulSoup / Playwright when the task is:

• Fetching a page that is JS-heavy, geo-restricted, rate-limited, or behind anti-bot/CAPTCHA.
• A plain fetch returned a 403/429/empty body/CAPTCHA challenge.
• Web search results (Google or Bing SERP).
• Listing the pages of a site (map) or fetching many pages of one site or section (crawl).
• E-commerce data (Amazon, Walmart, Target product/search pages).
• Social media data (Reddit, TikTok, YouTube).
• A screenshot of a rendered page.

If a basic fetch/curl clearly works and none of the above apply, you don't need Decodo.

Setup (do this first, in order)

Pick the lowest-friction path that works.

1. Check whether the CLI is already usable

decodo whoami          # installed + authenticated?  prints auth source + masked token
# or, with nothing installed:
npx -y @decodo/cli whoami

• Exit 0 with a token printed → you're ready, skip to Usage.
• "auth required" / exit 3 → continue to step 2.
• decodo: command not found → use npx -y @decodo/cli ... for everything, or install (step 3).

2. Authenticate

The user needs a Decodo API key from the dashboard at <https://dashboard.decodo.com/web-data/playground> (free account = up to 10K requests, no card). One key covers scraping, search, map and crawl.

Older plans only have a basic auth token. It works for scrape, screenshot and target commands, but search, map and crawl need an API key and reject a basic token with exit code 3. decodo whoami prints api key: or token: so you can tell which one is set.

Prefer the environment variable — it needs no interaction and is easy to scope to a session:

export DECODO_AUTH_TOKEN='<api-key>'

The variable is named DECODO_AUTH_TOKEN, which holds an API key.

To persist it to the CLI config non-interactively:

decodo setup --token '<api-key>'   # validates, then saves to config

Do not run a bare decodo setup — it opens a hidden interactive prompt you cannot drive. Credential precedence: --token flag → DECODO_AUTH_TOKEN env → saved config.

Treat the key as a secret. If whoami reports no auth, ask the user to set DECODO_AUTH_TOKEN or run decodo setup --token <api-key>. Never cat, read, or print the key from a config file (e.g. ~/.config/decodo/config.json) to work around missing auth.

3. Install for repeat use (optional)

curl -fsSL https://decodo.github.io/cli/install.sh | sh   # macOS / Linux
npm install -g @decodo/cli                                # any platform

npx -y @decodo/cli <command> works with zero install if you'd rather not install anything.

Choosing a surface

1. Shell available (Claude Code, Cursor, Codex CLI, Gemini CLI, Windsurf, terminal) → use

the `decodo` CLI. This is the default and the rest of this skill assumes it.

1. No shell (Claude Desktop, claude.ai, an MCP-only client) → use the hosted MCP server

at https://mcp.decodo.com/mcp (Bearer auth with the same API key). Per-client config in `references/mcp-setup.md`.

1. Neither → call the raw HTTP API with curl — recipes in

`references/api-curl.md`.

Usage (CLI)

Core commands

decodo scrape https://example.com                  # markdown by default
decodo scrape https://example.com --country US     # geo-target the request
decodo search "best web scraping api" | jq -r '.organic[].link'   # web search, JSON
decodo search "query" --geo DE --locale de-DE --page 2           # geo, locale, next page
decodo map https://example.com --limit 100                       # site URLs, one per line
decodo screenshot https://example.com -o shot.png                # PNG must go to a file

search always returns parsed JSON (organic, top_stories, people_also_ask, …) and takes only --geo, --locale and --page. It has no --engine, --limit or --parse. For a specific engine or its full SERP, use the google-search or bing-search target (below).

Search, map and crawl a site

search, map and crawl run on the Data API and need an API key (see Setup step 2).

Map first. map lists a site's URLs without fetching them: cheap, fast, small output. Use it to see what exists, then either scrape the few pages you need or crawl a scoped section. Don't crawl a whole site to find one page.

decodo map https://docs.example.com --limit 200 -o urls.txt
wc -l urls.txt; grep -i pricing urls.txt | head
decodo map https://example.com --select-paths "/blog/*" --exclude-paths "/blog/tag/*" --limit 100

Scope every crawl. Always pass --limit, and narrow with --select-paths / --exclude-paths (repeatable) or --max-depth. Unscoped crawls are slow and burn requests.

Start crawls async, collect later. decodo crawl <url> returns at once and prints only the crawl id. Keep working, then collect the pages with crawl results --wait, which polls until the crawl finishes:

id=$(decodo crawl https://docs.example.com --select-paths "/docs/*" --limit 50)
# ... other work ...
decodo crawl status "$id"                                 # discovered/completed/failed, status
decodo crawl results "$id" --wait --progress -o pages.ndjson
decodo crawl cancel "$id"                                 # stop early; crawled pages stay readable

For a small crawl you'll wait on anyway, start and collect in one call: decodo crawl <url> --limit 20 --wait -o pages.ndjson (the id goes to stderr). --wait polls every --poll-interval seconds (default 5) and gives up after --timeout seconds (default 3600, exit 6); the crawl keeps running until you cancel it. truncated/cancelled crawls print a warning; a faulted crawl writes its pages, then exits 7.

Never print crawl output into your context. Always write it to a file with -o, then read it in slices. Each NDJSON line is one page: url, depth, status, content, …

wc -l pages.ndjson                                        # how many pages
jq -r '[.status, .url] | @tsv' pages.ndjson | head -50    # page list, no content
jq -r 'select(.url | test("/pricing")) | .content' pages.ndjson | head -c 4000
jq -r 'select(.content_error) | .url' pages.ndjson        # pages that failed
decodo crawl results "$id" --no-content -o index.ndjson   # metadata only, no content

Don't cat the file, and don't head -n it either: one line can hold a whole page.

Discover targets — don't guess

Target subcommands are generated from the live API schema, so list and introspect them rather than memorizing them:

decodo targets                 # all targets, grouped (e.g. google-search, amazon-product, reddit-post)
decodo google-search --help    # exact flags for a target (--parse, --geo, ...)
decodo amazon-product --help

Prefer a dedicated target (reddit-subreddit, amazon-product, …) over a generic scrape <url> when one exists — it returns parsed, cleaner data and avoids guesswork. Check decodo targets first.

Then call a target directly:

decodo google-search "query" --parse        # structured JSON instead of raw SERP
decodo amazon-product B09H74FXNW --parse
decodo universal https://example.com         # generic target with the most flags — see --help

Output handling (important for piping)

By default a command prints the first result's content (markdown for scrape, parsed JSON for targets called with --parse). These modifiers apply to scrape and target commands. search, map and crawl support a subset; check decodo <command> --help.

The parsed JSON shape is target-specific. Pipe to jq 'keys' (then drill down, e.g. jq '.results | keys') to discover the structure before writing a deeper query.

Keep context small. Pipe straight to jq and select only the fields you need; make one request and extract from it. Don't dump raw, --pretty, or --full payloads just to "inspect" — use jq 'keys'. --full includes the response envelope (large headers/cookies), so use it only when you actually need status/headers.

# Parsed SERP → organic result titles (use a target command for --parse, not `search`)
decodo google-search "rust web scraping" --parse | jq -r '.results.results.organic[].title'
 
# Extract a field from a scraped JSON page (inspect with `jq keys` first if unsure)
decodo universal https://ip.decodo.com/json --no-markdown | jq -r '.proxy.ip'
 
# Stream results as NDJSON; each record nests the parsed payload under `.content`
decodo google-search "rust" --parse --format ndjson --full | jq -c '.content.results.results.organic'

stdout is data; logs and errors go to stderr — pipes stay clean.

Exit codes (use these to self-correct)

Search, map and crawl over MCP

The hosted MCP server has the same Data API tools; they also need an API key, not a basic token.

• search — web search; limit caps organic results (10 by default).
• map — site URLs as a JSON list; limit defaults to 10, so set it and narrow with

selectPaths / excludePaths. Then scrape_as_markdown only the pages you need.

• Crawl is async, in four tools:

1. crawl_start with url, a limit and path filters → returns crawl_id at once.
2. crawl_status with crawlId → poll every 10–30 s, doing other work in between,

until status is no longer pending.

1. crawl_results with crawlId → about 50 KB per call. Pass nextCursor back as cursor

until it is missing. A long page is split across calls; contentPart gives the range.

1. crawl_cancel with crawlId → stop early; pages crawled so far stay readable.