version: "1.0.1" name: scraping description: Web scraping via progressive escalation and social media platform scrapers. USE WHEN scraping, crawl, scrape URL, bot detection, CAPTCHA, spider. metadata: author: pai version: 1.0.0
Scraping Skill
Progressive Escalation
- Direct fetch — curl/fetch with standard headers
- Browser rendering — Headless browser for JS-heavy sites
- Proxy rotation — For rate-limited or geo-restricted content
- Specialized APIs — Platform-specific scrapers
Rules
- Respect robots.txt
- Rate limit requests
- Handle pagination
- Extract structured data
- Store raw + processed versions