codingheader/myskills/21pounder-web-scrape

/web-scrape <url> [options]

/web-scrape https://example.com/article
/web-scrape https://news.site.com/story --format=json
/web-scrape https://spa-app.com/page --scroll --screenshot

1. mcp__playwright__browser_navigate
   url: "<target URL>"
 
2. mcp__playwright__browser_wait_for
   time: 2  (allow initial render)

3. mcp__playwright__browser_evaluate
   function: "async () => {
     for (let i = 0; i < 3; i++) {
       window.scrollTo(0, document.body.scrollHeight);
       await new Promise(r => setTimeout(r, 1000));
     }
     window.scrollTo(0, 0);
   }"

4. mcp__playwright__browser_snapshot
   → Returns full accessibility tree with all text content

5. mcp__playwright__browser_take_screenshot
   filename: "scraped_<domain>_<timestamp>.png"
   fullPage: true

6. mcp__playwright__browser_close

# [Title]
 
**Source:** [URL]
**Date:** [if available]
**Author:** [if available]
 
---
 
[Main content in clean markdown]

# [Product Name]
 
**Price:** [price]
**Availability:** [in stock/out of stock]
 
## Description
[product description]
 
## Specifications
| Spec | Value |
|------|-------|
| ... | ... |

{
  "url": "https://...",
  "title": "Page Title",
  "type": "article|product|docs|list",
  "content": {
    "main": "...",
    "metadata": {}
  },
  "extracted_at": "ISO timestamp"
}

If page load fails:
1. Report the specific error to user
2. Suggest: "Try again?" or "Different URL?"
3. Close browser cleanly
 
If content is blocked:
1. Report what was detected (CAPTCHA/paywall/geo-block)
2. Extract any available preview content
3. Suggest alternatives if applicable

1. Navigate to URL
2. Wait longer (3-5 seconds) for JS hydration
3. Use browser_wait_for with specific text if known
4. Then snapshot

1. Navigate
2. Execute scroll loop (see Phase 1)
3. Snapshot after scrolling completes

1. Snapshot first to identify clickable elements
2. Use browser_click on "Read more" / "Show all" buttons
3. Wait briefly
4. Snapshot again for full content

1. Scrape first page
2. Identify "Next" or pagination links
3. Ask user: "Article has X pages. Scrape all?"
4. If yes, iterate through pages and combine

1. browser_navigate → 2. browser_wait_for → 3. browser_snapshot → 4. browser_close

1. browser_navigate
2. browser_wait_for
3. browser_evaluate (scroll)
4. browser_snapshot
5. browser_take_screenshot
6. browser_close