neuron-mr-white/unipi/web

Web Tools

Use these tools to access web content. The read path uses a local smart-fetch engine by default — free, fast, and no API key required.

web_search

Search the web for information. Lower source = simpler/cheaper providers.

• Quick facts: source 1-2 (DuckDuckGo, Jina Search)
• Research: source 3-5 (SerpAPI, Tavily, Perplexity)

Parameters:

• query (required): Search query string
• source (optional): Provider selection (1-5)

Examples:

web_search(query: "TypeScript generics tutorial")
web_search(query: "latest AI research", source: 5)  # Use Tavily

multi_web_content_read

Read and extract content from URLs. Uses the smart-fetch engine by default (source=0 or omitted) — free, local, no API key required. Supports single URL or batch URLs.

Default behavior (source=0):

• Browser-grade TLS fingerprinting via wreq-js
• Intelligent content extraction via defuddle
• Returns clean markdown with metadata (title, author, site, word count)
• No API key required

Parameters:

• url (required): Single URL string or array of URLs for batch
• source (optional): Provider selection (0=smart-fetch, 1=Jina Reader, 2=Firecrawl, 3=Perplexity)
• browser (optional): TLS fingerprint profile (default: chrome_145)
• os (optional): OS fingerprint (default: windows)
• format (optional): Output format — markdown, html, text, json (default: markdown)
• maxChars (optional): Maximum content characters (default: 50000)
• timeoutMs (optional): Request timeout in ms (default: 15000)
• removeImages (optional): Strip image references (default: false)
• includeReplies (optional): Include comments/replies (default: extractors)
• proxy (optional): Proxy URL
• batchConcurrency (optional): Concurrent requests for batch (default: 8)
• verbose (optional): Include metadata header (default: true)

Examples:

# Single URL (uses smart-fetch engine by default)
multi_web_content_read(url: "https://example.com/article")
 
# Batch URLs
multi_web_content_read(url: ["https://example.com/a", "https://example.com/b"])
 
# Use provider fallback (Jina Reader)
multi_web_content_read(url: "https://example.com/article", source: 1)
 
# Custom options
multi_web_content_read(url: "https://example.com/article", format: "json", maxChars: 10000)

web_llm_summarize

Summarize URL with LLM. Higher cost (LLM tokens + provider).

• Use for complex content that needs analysis
• Custom prompts supported for targeted summaries

Parameters:

• url (required): URL to summarize
• prompt (optional): Custom summarization prompt
• source (optional): Provider selection for content fetch (1-3)

Examples:

web_llm_summarize(url: "https://example.com/long-article")
web_llm_summarize(url: "https://example.com/research", prompt: "Extract key findings")

Provider Selection

• Omit source for auto-selection (smart-fetch engine for read, cheapest for search)
• Specify source number for specific provider
• If provider unavailable, tool throws descriptive error

Auto-selection falls through on failure. With source omitted, a failing provider is skipped and the next-ranked one is tried, so an uninitialized wigolo never blocks a search. With an explicit source, the choice is respected and the error is reported instead.

Provider Rankings

Search providers:

1. wigolo (free, local) — default
2. DuckDuckGo (free)
3. Jina AI Search (freemium)
4. SerpAPI (paid)
5. Tavily (paid)
6. Perplexity (paid)

Read providers:

1. Smart-Fetch Engine (free, local) — default
2. wigolo (free, local)
3. Jina AI Reader (freemium)
4. Firecrawl (paid)
5. Perplexity (paid)

Summarize providers:

1. Perplexity (paid)
2. LLM Summarize (uses pi's LLM)

wigolo

wigolo is a local-first web engine: multi-engine search (18 direct adapters) with rank fusion and on-device reranking, plus a tiered fetch router that escalates to a headless browser on anti-bot challenges. No API key, nothing leaves the machine, $0 per query.

It is not bundled — wigolo is AGPL-licensed and UniPi is MIT, so it is an optional dependency loaded at runtime only when the user installed it:

npm install -g wigolo && npx wigolo init

If it is missing or uninitialized, the provider raises an actionable error and auto-selection falls through to the next provider. Never tell the user wigolo is broken — tell them to run npx wigolo init, or to disable it in /unipi:settings (Web API).

Smart-Fetch Engine

The smart-fetch engine is a local content extraction pipeline:

• wreq-js: Browser-grade TLS fingerprinting (bypasses Cloudflare, etc.)
• defuddle: Intelligent content extraction from HTML
• linkedom: Server-side DOM parsing

Features:

• No API key required
• Browser-level anti-bot bypass
• Clean markdown output with metadata
• Batch concurrent fetching with progress
• Client-side meta redirect following
• Multiple output formats

Configure defaults via /unipi:settings (Web API) → "Smart Fetch Defaults"

Cost Awareness

• Smart-Fetch Engine: Free (read only, no API key)
• wigolo: Free (search + read, local, no API key) — prefer this
• DuckDuckGo: Free (search only)
• Jina: Freemium (search + read)
• SerpAPI/Tavily: Paid (search)
• Firecrawl: Paid (read)
• Perplexity: Paid (search + summarize)
• LLM Summarize: LLM token cost

Configuration

Configure providers via /unipi:settings (Web API) command.

• Add/remove API keys
• Enable/disable providers
• Configure smart-fetch defaults
• View provider status

Cache

Web content is cached for 1 hour by default.

• Clear cache: /unipi:settings (Web API → "Clear web cache…")
• Cache includes smart-fetch results (keyed by URL + browser + format + maxChars)