brunocampos01/ai/scrape-leads

Lead Scraping & Verification

Goal

Scrape leads using Apify (code_crafter/leads-finder), verify their relevance (industry match > 80%), and save them to a Google Sheet. For large scrapes (1000+ leads), use parallel scraping for 3-5x faster performance.

Inputs

• Industry: The target industry (e.g., "Plumbers", "Software Agencies")
• Location: The target location (e.g., "Texas", "United States", "California"). Scripts auto-format to Apify's required format (US states get ", us" suffix automatically).
• Total Count: The total number of leads desired

Scripts

All scripts are in ./scripts/:

• scrape_apify.py - Single scrape, for <1000 leads
• scrape_apify_parallel.py - Parallel scraping, for 1000+ leads
• classify_leads_llm.py - LLM-based lead classification
• enrich_emails.py - Email enrichment via AnyMailFinder
• update_sheet.py - Batch sheet updates
• read_sheet.py - Read data from Google Sheets

Process

Small Scrapes (<1000 leads)

1. Test Scrape

``bash python3 ./scripts/scrape_apify.py --query "INDUSTRY" --location "LOCATION" --max_items 25 --no-email-filter --output .tmp/test_leads.json ``

1. Verification

• Read .tmp/test_leads.json
• Check if at least 20/25 (80%) leads match the Industry
• Pass: Proceed to step 3
• Fail: Stop and ask user to refine keywords

1. Full Scrape

``bash python3 ./scripts/scrape_apify.py --query "INDUSTRY" --location "LOCATION" --max_items TOTAL_COUNT --no-email-filter --output .tmp/leads.json ``

1. [Optional] LLM Classification (for complex niches)

``bash python3 ./scripts/classify_leads_llm.py .tmp/leads.json --classification_type product_saas --output .tmp/classified_leads.json ``

1. Upload to Google Sheet

``bash python3 ./scripts/update_sheet.py .tmp/leads.json --title "Leads - INDUSTRY" ``

1. Enrich Missing Emails

``bash python3 ./scripts/enrich_emails.py SHEET_URL ``

Large Scrapes (1000+ leads)

1. Test Scrape (same as above with 25 items)

1. Parallel Full Scrape

``bash python3 ./scripts/scrape_apify_parallel.py \ --query "INDUSTRY" \ --total_count 4000 \ --location "United States" \ --strategy regions \ --no-email-filter ``

Geographic partitioning is automatic:

• United States: 4-way (Northeast, Southeast, Midwest, West)
• EU/Europe: 4-way (Western, Southern, Northern, Eastern)
• UK: 4-way (SE England, N England, Scotland/Wales, SW England)
• Canada: 4-way (Ontario, Quebec, West, Atlantic)
• Australia: 4-way (NSW, VIC/TAS, QLD, WA/SA)

1. Continue with steps 4-6 from small scrapes

Outputs

The ONLY deliverable is the Google Sheet URL. Local JSON files in .tmp/ are temporary intermediates.

Edge Cases

• No leads found: Ask user to broaden search
• API Error: Check credentials in .env
• Low quality classifications: If >80% "unclear", improve scrape keywords

Environment

Requires in .env:

APIFY_API_TOKEN=your_token
GOOGLE_APPLICATION_CREDENTIALS=path/to/credentials.json
ANTHROPIC_API_KEY=your_key
ANYMAILFINDER_API_KEY=your_key