bug-ops/zeph/web-scrape

Web Page Data Extraction

Extract structured data from web pages by specifying a URL, a CSS selector to target elements, and an extraction mode.

Command Format

{"url": "URL", "select": "CSS_SELECTOR", "extract": "MODE", "limit": N}

Parameters

Extract Modes

CSS Selector Reference

Basic Selectors

Attribute Selectors

Combinators

Pseudo-Classes

Common Extraction Patterns

Page Title

{"url": "https://example.com", "select": "title", "extract": "text"}

Meta Description

{"url": "https://example.com", "select": "meta[name='description']", "extract": "attr:content"}

Open Graph Tags

{"url": "https://example.com", "select": "meta[property^='og:']", "extract": "attr:content"}

All Links on a Page

{"url": "https://example.com", "select": "a[href]", "extract": "attr:href", "limit": 50}

External Links Only

{"url": "https://example.com", "select": "a[href^='http']", "extract": "attr:href", "limit": 30}

All Images

{"url": "https://example.com", "select": "img", "extract": "attr:src", "limit": 20}

Images with Alt Text

{"url": "https://example.com", "select": "img[alt]", "extract": "attr:alt", "limit": 20}

Article / Main Content

{"url": "https://example.com/post", "select": "article p", "extract": "text"}

{"url": "https://example.com/post", "select": "main p", "extract": "text"}

{"url": "https://example.com/post", "select": ".content p, .post-body p", "extract": "text"}

Headings (Document Structure)

{"url": "https://example.com/post", "select": "h1, h2, h3", "extract": "text"}

Navigation Links

{"url": "https://example.com", "select": "nav a", "extract": "text", "limit": 20}

Table Data

{"url": "https://example.com/data", "select": "table thead th", "extract": "text"}

{"url": "https://example.com/data", "select": "table tbody td", "extract": "text", "limit": 100}

{"url": "https://example.com/data", "select": "table tbody tr", "extract": "text", "limit": 50}

Ordered / Unordered Lists

{"url": "https://example.com", "select": "ul.features li", "extract": "text"}

{"url": "https://example.com", "select": "ol li", "extract": "text"}

Definition Lists

{"url": "https://example.com", "select": "dl dt", "extract": "text"}

{"url": "https://example.com", "select": "dl dd", "extract": "text"}

Form Fields

{"url": "https://example.com/form", "select": "input[type='text'], input[type='email'], textarea", "extract": "attr:name"}

Code Blocks

{"url": "https://example.com/docs", "select": "pre code", "extract": "text", "limit": 10}

Specific Data Attributes

{"url": "https://example.com", "select": "[data-price]", "extract": "attr:data-price"}

Multi-Step Extraction Strategy

For complex pages, extract data in multiple steps:

1. Discover structure — get headings and landmark elements first:

``scrape {"url": "https://example.com", "select": "h1, h2, h3, nav, main, article, section", "extract": "text", "limit": 30} ``

1. Narrow down — once you know the page structure, target the specific container:

``scrape {"url": "https://example.com", "select": "#results .item .title", "extract": "text"} ``

1. Extract details — get attributes or nested content from the targeted elements:

``scrape {"url": "https://example.com", "select": "#results .item a", "extract": "attr:href"} ``

Selector Construction Tips

Finding the Right Selector

• Start broad (p, div, a) and narrow down based on results
• Use class names when available — they are the most reliable selectors
• Combine element + class for precision: div.product-card instead of .product-card
• Use attribute selectors for pages that rely on data attributes: [data-testid="price"]
• Chain selectors for nested content: div.product-card > h3 > a

Common Page Structures

Pagination

To scrape multiple pages, modify the URL for each page:

{"url": "https://example.com/items?page=1", "select": ".item-title", "extract": "text", "limit": 25}

{"url": "https://example.com/items?page=2", "select": ".item-title", "extract": "text", "limit": 25}

Look for pagination patterns in the page:

{"url": "https://example.com/items", "select": ".pagination a", "extract": "attr:href"}

Important Notes

• Always use HTTPS URLs when possible
• Set a reasonable limit to avoid overwhelming output — start with 10-20 and increase if needed
• If a selector returns no results, the page may use JavaScript rendering (SPA). Try broader selectors or a different approach
• Some sites block automated access — if a request fails, the site may require authentication or have anti-scraping measures
• Respect robots.txt and site terms of service
• For tables, extract headers (thead th) and data (tbody td) separately for cleaner results
• When extracting links with attr:href, relative URLs (starting with /) need the base domain prepended
• Use extract: "html" when you need to preserve formatting (bold, links, lists) within elements
• If the initial selector is too broad and returns noise, add parent context: instead of p, try article p or main p
• For sites with dynamic class names (React/Next.js), use attribute selectors: [data-testid="..."], [role="..."], [aria-label="..."]