yrzhe/claude-skills/intelligent-web-scraper

Name: intelligent-web-scraper
Author: yrzhe

Claude Code and agent skill by yrzhe/claude-skills

Trust tier 3

community

By yrzhe
Coding
yrzhe/claude-skills
Plugin intelligent-web-scraper
plugins/intelligent-web-scraper/skills/intelligent-web-scraper/SKILL.md
Updated 9 Oct 2026
Version 1.0.01 version recorded

Source check pending

Checking that the SKILL.md file is still in the repository.

Self-learning intelligent web scraper agent - automatically analyzes page structure, handles pagination, anti-blocking, and discovers article series. No user configuration needed - AI decides everything.

Install

npx vskill@latest install yrzhe/claude-skills/intelligent-web-scraper

SKILL.md

Intelligent Web Scraper Agent

A self-aware, self-learning web scraping agent that accumulates experience over time.

Prerequisites

IMPORTANT: Before using this skill, ensure these dependencies are installed.

One-Click Setup (Recommended)

bash

# Navigate to skill directory and run setup
cd ~/.claude/skills/intelligent-web-scraper
chmod +x setup.sh && ./setup.sh
 
# Or for VPS/headless environment:
./setup.sh --headless

The setup script will:

Detect your environment (macOS/Linux, GUI/headless)
Install all Python dependencies
Install Playwright browsers
Set up Crawl4AI
Create necessary directories

Check Dependencies

bash

# Check if everything is installed correctly
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py
 
# Auto-install missing dependencies
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --install
 
# Output as JSON (for programmatic use)
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py --json

Manual Setup (Alternative)

If you prefer manual installation:

bash

# 1. Install Python dependencies
pip install -r ~/.claude/skills/intelligent-web-scraper/requirements.txt
 
# 2. Install Playwright browser
python -m playwright install chromium
 
# 3. Set up Crawl4AI
crawl4ai-setup
 
# 4. For VPS/headless Linux, also run:
python -m playwright install-deps chromium

Required Tools

Tool	Purpose	Installation
Playwright MCP	Browser automation	Must be configured in Claude Code MCP settings
Python 3.9+	Script execution	`brew install python` or system package manager
Crawl4AI	Advanced crawling	Included in setup.sh

Verify Installation

bash

# Quick verification
python -c "from crawl4ai import AsyncWebCrawler; print('Crawl4AI OK')"
python -c "from playwright.sync_api import sync_playwright; print('Playwright OK')"
 
# Full check
python3 ~/.claude/skills/intelligent-web-scraper/scripts/check_deps.py

VPS/Headless Deployment

For servers without GUI (VPS, Docker, CI):

bash

# 1. Run setup in headless mode
./setup.sh --headless
 
# 2. The scraper will automatically use headless browser
# No GUI required - all scraping works via headless Chromium
 
# 3. Check configuration
python3 ~/.claude/skills/intelligent-web-scraper/scripts/config.py

Note: local_browser_scraper.py (CDP mode) requires a GUI and is not available in headless environments. Use crawl4ai_wrapper.py instead.

Local Browser Scraping (CDP)

Use this when the user wants to scrape using their local browser (Comet, Chrome, etc.)

This allows scraping while preserving user's login sessions and cookies!

Quick Start

bash

# 1. Install websockets (one time)
pip install websockets
 
# 2. Launch Comet/Chrome with debugging (or use the script)
python ~/.claude/skills/intelligent-web-scraper/scripts/local_browser_scraper.py \
    --launch comet \
    --url "https://example.com"
 
# 3. Scrape data from current page
python ~/.claude/skills/intelligent-web-scraper/scripts/local_browser_scraper.py \
    --extract articles \
    --output data.json

Using local_browser_scraper.py

Launch browser with debugging:

bash

python scripts/local_browser_scraper.py --launch comet --url "https://douban.com"
python scripts/local_browser_scraper.py --launch chrome --url "https://example.com"

Scrape from current tab:

bash

# Get page text
python scripts/local_browser_scraper.py --extract text
 
# Get all links
python scripts/local_browser_scraper.py --extract links
 
# Get articles
python scripts/local_browser_scraper.py --extract articles
 
# Custom JavaScript
python scripts/local_browser_scraper.py --extract "document.querySelectorAll('h1').length"

Scrape specific tab (by URL pattern):

bash

python scripts/local_browser_scraper.py --url-pattern "douban.com" --extract articles

Built-in extractors: text, html, title, links, images, tables, articles, metadata

Manual CDP (Alternative)

CRITICAL: Always use user's EXISTING profile to preserve login sessions!

bash

# Comet - use existing profile
/Applications/Comet.app/Contents/MacOS/Comet \
    --remote-debugging-port=9222 \
    --user-data-dir="$HOME/Library/Application Support/Comet" \
    "https://example.com" &
 
# Chrome - use existing profile
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
    --remote-debugging-port=9222 \
    --user-data-dir="$HOME/Library/Application Support/Google/Chrome" \
    "https://example.com" &

Browser Profile Locations (macOS)

Browser	User Data Directory
Comet	`~/Library/Application Support/Comet`
Chrome	`~/Library/Application Support/Google/Chrome`
Edge	`~/Library/Application Support/Microsoft Edge`
Brave	`~/Library/Application Support/BraveSoftware/Brave-Browser`

NEVER use temp directory like /tmp/browser-debug - this creates empty profile without logins!

Why use existing profile:

Preserves all login sessions (no re-authentication needed)
Browser extensions work normally
User's bookmarks, history available

When to Use Local Browser vs Playwright

Use Local Browser	Use Playwright
User requests specific browser	Automated bulk scraping
Need user's login session	Don't need authentication
User wants to see the page	Headless scraping OK
Site blocks headless browsers	Standard sites

Core Capabilities

1. Intelligent Page Analysis

Auto-detect page type (product list, article, series, etc.)
Identify data containers and selectors
Recognize pagination mechanisms

2. Smart Pagination & Scroll Loading

Page numbers, Next button, Infinite scroll, Load more, API pagination
Auto-detect and handle appropriately

CRITICAL: Scroll Loading Requirements

MUST automatically scroll down the page until all content is loaded
Many modern websites use lazy loading/infinite scroll, initially showing only partial content
Scroll strategy:

Record current page height
Scroll to bottom
Wait 1-2 seconds for new content to load
Check if page height increased
Repeat until height stops changing (3 consecutive times)

Use browser_evaluate to execute scrolling:

``javascript window.scrollTo(0, document.body.scrollHeight) ``

3. Detail Link Following

CRITICAL: Detail Page Scraping Requirements

List pages may only show title/summary, full content is on detail pages
MUST detect and follow detail links to get complete data
Patterns to identify detail links:
"View more", "Read more", "Details", "Full article"
Title itself is a link
Arrow links like "→"
Scraping workflow:

First scrape all entries from list page
Identify detail link for each entry
Visit detail pages one by one to extract full content
Merge data and save

Note: Detail page scraping needs appropriate delays (2-5s) to avoid blocking

4. Anti-Blocking

Adaptive delays (2-60s based on signals)
User-Agent rotation
Block detection and recovery

5. Series Discovery

Find prev/next links
Detect table of contents
URL pattern analysis
Build complete series from single article

6. Self-Learning (NEW)

Records successful patterns for each domain
Accumulates lessons learned
Reuses knowledge on similar sites

Self-Learning System

How It Works

┌─────────────────────────────────────────────────────────────┐
│                    SCRAPE REQUEST                            │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 1: Check Experience Database                            │
│ - Read experiences/site_patterns.json                        │
│ - Look for matching domain/URL pattern                       │
│ - If found: Use learned selectors and strategies             │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 2: Execute Scraping                                     │
│ - Use learned patterns OR discover new ones                  │
│ - Apply anti-blocking strategies                             │
│ - Extract data                                               │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│ Step 3: Learn & Record                                       │
│ - If successful: Save patterns to site_patterns.json         │
│ - If failed: Record lesson in lessons_learned.md             │
│ - Update success rate and timestamps                         │
└─────────────────────────────────────────────────────────────┘

Experience Database Structure

Location: ~/.claude/skills/intelligent-web-scraper/experiences/

experiences/
├── site_patterns.json    # Learned patterns for each domain
├── lessons_learned.md    # Accumulated wisdom and gotchas
└── README.md             # How the learning system works

Pattern Storage Format

json

{
  "domain.com": {
    "url_patterns": {
      "/movie/*/comments": {
        "selectors": {
          "container": ".comment-list",
          "item": ".comment-item",
          "fields": {
            "author": ".comment-info a",
            "content": ".comment-content span",
            "rating": ".rating"
          }
        },
        "pagination": {
          "type": "page_number",
          "selector": ".paginator a"
        },
        "scroll_loading": {
          "required": true,
          "max_scrolls": 20,
          "scroll_delay_ms": 1500,
          "notes": "Page uses infinite scroll loading"
        },
        "detail_links": {
          "required": true,
          "selector": "a.read-more, a:has-text('→')",
          "fields_in_detail": ["fullContent", "images", "comments"],
          "notes": "List page only has titles, need to scrape detail pages for full content"
        },
        "anti_block": {
          "min_delay": 3,
          "max_delay": 8,
          "needs_login": false
        },
        "success_count": 15,
        "last_success": "2024-01-15",
        "notes": "Rate limit after 50 requests"
      }
    }
  }
}

Execution Guide

When You Receive a Scraping Request:

Phase 1: Check Existing Knowledge

python

# ALWAYS do this first
experiences_path = "~/.claude/skills/intelligent-web-scraper/experiences/site_patterns.json"

Read site_patterns.json
Extract domain from target URL
Check if matching pattern exists
If yes: Report to user "I have experience with this site, using learned patterns"

Phase 2: Analyze Page

bash

# 1. Navigate to URL
browser_navigate(url)
 
# 2. Wait for load
browser_wait_for(time=3)
 
# 3. [IMPORTANT] Scroll to load all content
browser_evaluate('''async () => {
  let prevHeight = 0;
  let currentHeight = document.body.scrollHeight;
  let noChangeCount = 0;
 
  while (noChangeCount < 3) {
    prevHeight = currentHeight;
    window.scrollTo(0, document.body.scrollHeight);
    await new Promise(r => setTimeout(r, 1500));
    currentHeight = document.body.scrollHeight;
 
    if (currentHeight === prevHeight) {
      noChangeCount++;
    } else {
      noChangeCount = 0;
    }
  }
  return document.querySelectorAll('article, [class*="item"]').length;
}''')
 
# 4. Get snapshot (DOM structure)
browser_snapshot()
 
# 5. Take screenshot for visual analysis
browser_take_screenshot()

This is the start of the file. Read the full SKILL.md on GitHub.

Security

Security scanPassed, 90 of 100Pattern checks on the skill's files.
Model reviewPassed

See the full security report

Quality evaluation

Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

SKILL.md in the repositoryCheckingplugins/intelligent-web-scraper/skills/intelligent-web-scraper/SKILL.md

Popularity

7-day trend score: 0

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

Claude Code
Cursor
GitHub Copilot
Windsurf
Codex
Gemini CLI
Cline
Amp
Roo Code
Goose
Continue
Kilo Code
Junie
Trae
OpenHands
Qwen Code
AI Agent
Tabnine Chat
Sourcegraph Cody
Amazon Q
Firebase Genkit
Sweep
Devin
Phind
Mentat
Claude Desktop
Aider
GPT Pilot
Plandex
bolt.new
v0
Replit Agent
Anse
TypingMind
BetterChatGPT
LibreChat
Warp AI
Pieces
Fig AI

Embed badge

[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/yrzhe/claude-skills/intelligent-web-scraper/badge)](https://verified-skill.com/skills/yrzhe/claude-skills/intelligent-web-scraper)

Something wrong with this skill or its page? Tell us.