northseadl/norix-skills/web-scraper

Name: web-scraper
Author: northseadl

Trust tier 3

community

Source check pending

Checking that the SKILL.md file is still in the repository.

Web scraper with SPA/JavaScript rendering, page interaction, and JS execution. Two-tier engine (HTTP → Playwright browser). Smart discovery, batch fetch, interactive content extraction, OpenAPI parsing. Use when read_url_content fails, SPA rendering needed, or page interaction required.

Install

npx vskill@latest install northseadl/norix-skills/web-scraper

SKILL.md

Web Scraper

./scrape <command> [options] — Run ./scrape help for full option reference.

Content Precision Levels

The fetch command outputs filtered content by default. Three precision levels:

Level	Flag	Behavior	When to use
Default	(none)	PruningContentFilter removes boilerplate	Most scenarios — good enough
Precision	`--selector ".css"`	Extract only the matched container	Production-quality docs, zero noise
Raw	`--raw`	Full page, no filtering	Debugging, page structure analysis

Precision workflow: fetch a sample page → inspect remaining noise → identify content container via CSS selector → re-fetch all pages with --selector.

Engine Architecture

L0: Pure HTTP (httpx + selectolax + markdownify)
    fetch → strip boilerplate → markdownify → clean content
 
L1: Browser (crawl4ai + Playwright, networkidle wait)
    fetch → [JS interaction] → PruningContentFilter → fit_markdown → clean content
    Handles: SPAs, anti-bot, JS-rendered content, folded/lazy content
 
Auto mode: L0 probe → detect SPA/empty → fallback to L1
 
Smart Discovery (3-tier):
    1. sitemap.xml → 2. Nav DOM extraction → 3. Browser deep crawl

Workflow Patterns

Single page fetch

bash

# Auto-detects if browser needed
./scrape fetch https://docs.example.com/api/create
 
# Force browser for known SPA sites
./scrape fetch https://open.feishu.cn/document/... --engine cdp
 
# Precision extraction with CSS selector
./scrape fetch https://developer.work.weixin.qq.com/document/path/90196 --engine cdp --selector ".ep-doc-area"

Interactive content extraction (SPA/folded content)

bash

# Auto-expand all collapsed/folded sections (universal preset)
./scrape fetch https://open.feishu.cn/document/... --expand-all --wait 5000
 
# Scroll through entire page to trigger lazy loading
./scrape fetch https://example.com/infinite-scroll --scroll-full
 
# Custom JavaScript before extraction
./scrape fetch URL --js "document.querySelectorAll('.expand-btn').forEach(e => e.click())"
 
# JS from file (for complex interactions)
./scrape fetch URL --js-file /path/to/interact.js --wait 5000
 
# Wait for specific element before extraction
./scrape fetch URL --wait-for ".api-response-table"
 
# Combine: expand + custom JS + wait
./scrape fetch URL --expand-all --js "extraAction()" --wait-for ".loaded" -o /tmp/docs/

Execute JavaScript on a page

bash

# Execute JS and return metadata (no markdown conversion)
./scrape exec URL --js "return document.title"
 
# Extract data from page's JavaScript state
./scrape exec URL --js "return JSON.stringify(window.__NEXT_DATA__)"
 
# Execute JS then fetch page as markdown
./scrape exec URL --js "expandAll()" --then-fetch -o /tmp/result.md

Batch-download documentation

bash

./scrape fetch https://docs.example.com --auto -o /tmp/docs/
./scrape fetch --from-file urls.txt -o /tmp/docs/
./scrape fetch --from-file urls.txt --merge -o /tmp/docs/all.md

Files are automatically named by page title (e.g., 读取成员.md, Getting_Started.md).

Build organized local documentation

Discover site structure: ./scrape discover <url> --json → get URLs + titles
Filter relevant URLs (Agent selects subset based on user's needs)
Write filtered URLs to a file
Fetch to output directory: ./scrape fetch --from-file urls.txt -o /path/to/docs/
Organize (Agent moves/renames files into logical folder structure)

Analyze site structure

bash

./scrape discover https://docs.example.com
./scrape discover https://docs.example.com --engine cdp --deep --max-pages 100

Extract OpenAPI specs

bash

./scrape openapi https://api.example.com/v3/api-docs -o /tmp/api.md
./scrape openapi https://api.example.com/swagger-ui/ -o /tmp/api.md

Key Options

Option	Commands	Description
`--engine MODE`	discover, fetch	`auto` (default), `http` (L0 only), `cdp` (L1 only)
`--selector CSS`	fetch, exec	CSS selector for content area (precision mode)
`--raw`	fetch, exec	Output full page (skip content filtering)
`--js CODE`	fetch, exec	JavaScript to execute before extraction
`--js-file PATH`	fetch, exec	JavaScript file to execute before extraction
`--wait-for CSS`	fetch	Wait for CSS selector to appear before extraction
`--expand-all`	fetch	Auto-expand collapsed/folded sections (universal preset)
`--scroll-full`	fetch	Scroll entire page to trigger lazy loading
`--then-fetch`	exec	After JS execution, also fetch page as markdown
`--session ID`	exec	Reuse browser session for multi-step interactions
`--js-only`	exec	Skip navigation, execute JS on existing session
`--auto`	fetch	Auto-discover pages then fetch all
`--from-file FILE`	fetch	Read URLs from file (one per line)
`-o PATH`	fetch, exec, openapi	Output directory or file path
`--merge`	fetch	Merge all pages into single markdown
`--json`	discover, fetch	Machine-readable JSON output
`--summary-only`	fetch	Only show URL, title, char count
`--max-pages N`	discover, fetch	Limit pages (discover: 200, fetch: 50)
`--deep`	discover	Follow links for browser-based BFS crawl

Requirements

uv: curl -LsSf https://astral.sh/uv/install.sh | sh
Dependencies and Playwright Chromium auto-installed on first run

Security

Security scanPassed, 90 of 100Pattern checks on the skill's files.
Model reviewPassed

See the full security report

Quality evaluation

Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

SKILL.md in the repositoryCheckingweb-scraper/SKILL.md

Popularity

7-day trend score: 0

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

Claude Code
Cursor
GitHub Copilot
Windsurf
Codex
Gemini CLI
Cline
Amp
Roo Code
Goose
Continue
Kilo Code
Junie
Trae
OpenHands
Qwen Code
AI Agent
Tabnine Chat
Sourcegraph Cody
Amazon Q
Firebase Genkit
Sweep
Devin
Phind
Mentat
Claude Desktop
Aider
GPT Pilot
Plandex
bolt.new
v0
Replit Agent
Anse
TypingMind
BetterChatGPT
LibreChat
Warp AI
Pieces
Fig AI

Embed badge

[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/northseadl/norix-skills/web-scraper/badge)](https://verified-skill.com/skills/northseadl/norix-skills/web-scraper)

Something wrong with this skill or its page? Tell us.