netclaw-dev/netclaw/web-content-retrieval

Web Content Retrieval

This skill helps you choose the right tool for fetching web content.

Quick Decision

When web_fetch Works

web_fetch sends a simple HTTP request and returns the response. It works for:

• Static HTML pages
• Server-rendered content
• Public APIs and JSON endpoints
• News articles, blog posts, documentation

When web_fetch Fails

web_fetch will return empty, partial, or error content when:

• The page requires JavaScript to render (SPAs, React/Vue apps)
• The site blocks non-browser user agents
• Content is behind a login wall
• The site uses anti-bot protections

Common sites that require browser automation:

• Twitter/X (x.com, twitter.com)
• Instagram
• LinkedIn
• Facebook
• Most modern social media platforms

Using Browser Automation

If a browser MCP server is available (e.g., Playwright), use it for JS-heavy sites:

# First, check if browser tools are available
search_tools(query: "browser navigate")
 
# If available, use browser navigation
browser_navigate(url: "https://x.com/username/status/123")

Browser automation:

1. Renders JavaScript
2. Handles dynamic content loading
3. Can interact with pages (scroll, click)
4. Takes screenshots for verification

When Browser Automation Is Unavailable

If no browser MCP is configured, tell the user honestly:

"This URL appears to be from a JavaScript-heavy site (Twitter/X) that typically

requires browser automation to fetch content. I don't have a browser tool

available in this session. You could:

1. Share the content directly (copy/paste the tweet text)

2. Enable a browser MCP server like Playwright

3. I can try web_fetch, but it will likely return incomplete content"

Do not silently fail. If you attempt web_fetch on a known JS-heavy site and get empty/blocked content, explain why and suggest alternatives.

Fallback Strategy

1. Check the domain — is it a known JS-heavy site?
2. If yes, check for browser tools — search_tools(query: "browser")
3. If browser available — use it
4. If no browser — inform the user, offer alternatives
5. If uncertain — try web_fetch first, but be ready to explain if it fails

Cross-References

• Citation and source rules: load search-citation
• Tool discovery: see netclaw-operations → Tool Discovery