thinkscape/agent-smart-fetch/smart-fetch

Smart Fetch Tools

When to use which tool

Sites and pages smart_fetch handles well

smart_fetch uses Defuddle for content extraction and wreq-js for browser-grade TLS fingerprinting. This combination works especially well on:

Limitations — escalate to the browser tool for:

• JS-heavy SPAs — content that only appears after JavaScript execution
• Login-protected pages — no session/cookie management
• Interactive flows — anything needing clicks, form fills, or scrolling

smart_fetch

Why smart_fetch over web_fetch

• TLS fingerprinting — impersonates real browsers at the TLS/HTTP2 level (JA3/JA4).

Sites that return 403 or empty pages to plain HTTP clients often serve full content to smart_fetch.

• Better extraction — Defuddle removes more noise (nav, sidebars, ads,

footers, social widgets) and keeps more signal (code blocks, footnotes, math, callouts, schema.org metadata).

• Richer metadata — returns author, publish date, site name, language,

word count.

• `raw` format — returns the full unmodified server response (HTML,

JSON, markdown, etc.) without extraction or truncation. Use when you need to parse the raw markup yourself.

• No API key required — works out of the box.

batch_smart_fetch

• Default concurrency: 8 parallel requests (configurable via plugin config).
• Results are ordered matching the input array — labelled [N/total].
• Individual failures don't fail the batch — each item has its own status.

Workflow escalation

1. `smart_fetch` — first choice for any URL.
2. `batch_smart_fetch` — when you need multiple URLs at once.
3. `web_fetch` — if smart_fetch is unavailable.
4. Browser tool — JS-heavy or login-protected pages only.

Automatic web_fetch fallback

When the smart-fetch plugin is enabled, it registers as a web fetch provider. The built-in web_fetch tool will automatically use smart_fetch's TLS-fingerprinted pipeline when its own Readability extraction fails — no configuration needed.