deanwhitex/aiw2.0stack/asset-scraping

Asset Scraping Skill

How this skill is customised

This skill defines the universal asset-harvest mechanism: where to scrape, in what priority order, how to validate, how to write the manifest, and what counts as a halt.

The niche-specific specifics, what asset categories to look for (a contractor needs truck photos and a manufacturer badge wall; a boutique hotel needs golden-hour exteriors and host portraits; an e-commerce brand needs product flat-lays), what alt-text patterns to detect them by, what trust badges to match, what minimum counts to enforce, come from the niche playbook written by Module 2D:

templates/{active-niche-slug}/niche-playbook/asset-patterns.json
templates/{active-niche-slug}/niche-playbook/photo-manifest.json
templates/{active-niche-slug}/niche-playbook/trust-signals.json

asset-patterns.json defines the alt-text and filename keywords that identify each category. photo-manifest.json enumerates required categories with minimum counts and lighting/composition notes. trust-signals.json lists the badges to detect with match-keywords for fuzzy lookup.

If any of these files are missing, the agent halts with an error pointing at Module 2D.

Purpose

Harvest the client's real brand assets from their owned channels so the build agent (Stage 9) has real logos, real trust signals, real project/portfolio photos, and real people photos to work with. Never fabricate. When something cannot be found, flag it for manual drop.

What gets harvested (universal categories, niche playbook can add more)

Sources to scrape (priority order)

1. Client's own website, homepage, about, services/offerings, portfolio/gallery, reviews
2. Google Business Profile, photos tab (often has people/site shots the website doesn't)
3. Facebook Business Page, profile/cover, photo albums
4. Instagram (if linked), recent grid, highlights, profile picture
5. LinkedIn (for owner photo specifically), if found in research
6. Yelp / TripAdvisor / Houzz / niche-specific directories, last resort

The agent matches each <img> against the alt-text and filename patterns in niche-playbook/asset-patterns.json to categorise candidates.

Image quality requirements

Logo

• Prefer SVG → PNG → WebP → JPG
• Minimum 200×200px or 200px in any single dimension
• Save the highest-quality version found

Trust badges

• PNG or SVG with transparent background preferred
• Filename: lowercase kebab-case ({badge-id}.png)
• Match against niche-playbook/trust-signals.json for badge identity; never invent or

alter manufacturer/association marks

Portfolio / project photos

• Minimum 1200×800px
• WebP or JPG
• Compress under 300KB per file but maintain visual quality
• Skip images smaller than 800×600
• Filename: descriptive kebab-case

Owner / team / host photos

• Minimum 600×600px
• Prefer documentary-style (on-site, in-context) over chamber-of-commerce headshots when

both available

• Filename: owner-{firstname}-{lastname}.webp or team-{role}-{firstname}.webp

The niche playbook may set stricter requirements (e.g. boutique hospitality requires 1920×1080 hero photos at golden hour; contractor requires truck photos).

Scraping execution

Step 1, Read research + niche patterns

• clients/[Client Name]/Pipeline Data/research/research-data.json, URLs, social links
• templates/{active-niche}/niche-playbook/asset-patterns.json, what to look for
• templates/{active-niche}/niche-playbook/photo-manifest.json, required counts/types
• templates/{active-niche}/niche-playbook/trust-signals.json, badge lookup table

Halt if any niche playbook file is missing.

Step 2, Scrape client's website

Use Playwright to visit homepage, about, services, gallery, contact, reviews. Extract every <img> with src, alt, and dimensions. Categorise per asset-patterns.json.

Step 3, Hit GBP / social channels

Playwright with stealth settings. Respect rate limits (1 request per 2 seconds for social). Scroll loops where needed to load all photos.

Step 4, Categorise, validate, save

• Move accepted images to their target folders
• Rename per the niche playbook's filename convention
• Generate manifest.json in each folder

{
  "logo": [
    {
      "filename": "logo.svg",
      "source": "https://example.com/wp-content/uploads/logo.svg",
      "dimensions": "240x80",
      "format": "svg"
    }
  ]
}

Step 5, Trust badge library lookup (when applicable)

If the niche playbook ships a curated badge library at niche-playbook/trust-badges/ with a registry.json of match_keywords, fuzzy-match the client's detected certifications against the registry. First match wins. Copy the library badge into the client's badges folder. Record matches in the per-client assets/badges/manifest.json.

Niches that don't have a curated library scrape badges directly from the client's website and manufacturer/association brand kits.

Step 6, Flag gaps

For each category below minimum count from photo-manifest.json, create MANUAL-DROP-NEEDED.md in that folder with explicit instructions and source suggestions.

Step 7, Logo absence = HALT

If no logo found:

• Write MANUAL-DROP-NEEDED.md in logo/
• Update pipeline-state.json → stages.asset_scraping.passed = false
• Halt the pipeline

Stage 9 (hero generation) and Stage 10 (build) require a logo.

Step 8, Pipeline state

Update logs/pipeline-state.json:

{
  "stages": {
    "asset_scraping": {
      "passed": true,
      "timestamp": "ISO",
      "found": {
        "logo": 1,
        "trustBadges": 4,
        "projectImages": 12,
        "founderPhotos": 2
      },
      "manualDropNeeded": []
    }
  }
}

Failure modes

Ethical considerations

• Respect copyright. Only scrape from the client's owned channels (their site, their

social, their GBP).

• Manufacturer / association certification badges are explicitly authorised for use by

certified members, those are safe.

• Stock photos from third-party sites are NOT okay to scrape, use only client-owned

content.

What this skill is NOT

• Not the hero image generator (that's Stage 9 with nano-banana)
• Not the design synthesiser (Stage 7)
• Not responsible for image editing, pipeline takes assets as-is