anton-abyzov/vskill/apify-data-gathering
Claude Code and agent skill by anton-abyzov/vskill
- vendor
Checking that the SKILL.md file is still in the repository.
Get public web data the cheapest legal way, with Apify as the paid fetch layer: choose official API, plain fetch or an Apify Store actor, cap every run's spend, and save evidence. Activate when the user wants to scrape, crawl, extract, monitor or enrich data: lead lists from Google Maps or directories, contact emails from websites, social listening on X, Reddit, Instagram, TikTok or YouTube, Google SERP rank tracking, competitor site or pricing changes, review mining, job-post signals, or inges…
Install
npx vskill@latest install anton-abyzov/vskill/apify-data-gatheringSKILL.md
The same SKILL.md was published earlier as anton-abyzov/vskill/apify-data-gathering.
Apify data gathering
Apify is a fetch layer: maintained scrapers (Actors), browsers, proxies and storage you pay for per result or per compute unit. It is not a strategy. This skill decides when Apify is the right door, keeps every run inside a budget, and leaves a dataset someone can audit.
The official apify/agent-skills teach the platform (Actor catalogue, building Actors). This skill is the operating discipline around them: door choice, legality, budget, caching, evidence. Use both if both are installed.
1. Pick the door, cheapest first
| Tier | Door | Cost | Use when | |
|---|---|---|---|---|
| 0 | Official API, export, RSS, sitemap, JSON-LD, the site's own XHR/GraphQL | $0 | Always check first. GitHub API, HN Algolia API, YouTube Data API, Reddit JSON, Google Places API, a league's JSON endpoint. | |
| 1 | Plain fetch and parse (curl/httpx + selectors, or a single-page fetch tool) | $0 | Static HTML, a handful of pages, no bot wall. | |
| 2 | Apify Store Actor | per result / per event | A maintained Actor exists for the site (Maps, Instagram, X, TikTok, Google SERP, reviews), or you need JS rendering or proxies at volume. | |
| 3 | Generic crawler Actor (apify/website-content-crawler, apify/cheerio-scraper) or a custom Actor | compute units | Long-tail sites at volume, docs ingestion for RAG. |
Rules that override the table:
- For anything that runs on a schedule, compile a deterministic extractor once (selectors or a typed provider) and replay it. Do not call an LLM on every fetch; it creates drift and duplicates. Re-run the LLM only when health checks fail.
- Hash the fetched content and skip unchanged pages before paying for extraction.
- If the only way in is defeating a login, CAPTCHA, WAF challenge or honeypot aimed at you, stop and ask the user. Tooling being legal says nothing about the target allowing it.
2. Legal and data gate (before any run)
- Read the target's terms and robots.txt. If the terms forbid automated access, say so and ask for an alternative (official API, partner export, written permission).
- Public data only. Never use your own logged-in cookies with an Actor (LinkedIn, Facebook, Instagram private views). Logged-in scraping breaks those platforms' terms and risks the account.
- Personal data: collect only business-contact fields the task needs, record the source URL for each row, and honour suppression lists. Cold email needs CAN-SPAM (US) or GDPR/PECR (UK/EU) compliance: opt-out, physical address, lawful basis.
- Respect rate limits and honeypots: fetch only known URL shapes, no blind link following on sites that trap crawlers.
3. Token and network
- Token:
APIFY_TOKENenv var, else macOS Keychain serviceapify-api-token. Read it into a variable, never echo, log, commit or paste it. If neither exists, stop and tell the user to create a token in Apify Console > Settings > API & Integrations and store it withsecurity add-generic-password -U -s apify-api-token -a "$USER" -w(it prompts for the value). - Network:
api.apify.commust be reachable. Sandboxed or cloud agents often deny it; run on the user's machine or ask them to allowlist the host. Do not route around a network policy. - MCP option:
https://mcp.apify.com(OAuth, orAuthorization: Bearer), narrow it with?tools=actors,docs,<owner>/<actor>. Good for exploring Actors interactively; use the script below for budgeted batch runs. Limit: 30 requests per second per user.
4. Budget every run
python3 scripts/apify_run.py limitsshows month-to-date spend and the account cap.- Find the Actor and its pricing model:
python3 scripts/apify_run.py search "<site or task>", then open the Actor's Pricing tab. Prefer Actors with many users and recent updates. - Estimate before running: items x price per 1,000, plus compute for crawler Actors. Say the estimate out loud.
- Smoke run with 10 to 20 items and a $0.50 cap. Apify rejects a
maxTotalChargeUsdbelow $0.50 (max-total-charge-usd-below-minimum); the cap is a ceiling, and a small run is billed per event, typically cents. Inspect fields, then scale. - Every run passes
--max-usd(maps tomaxTotalChargeUsd) and, for pay-per-result Actors,--max-items. The script refuses a run without a cap and refuses one larger than the money left this month. - Reuse data:
--outplus--cache-hoursskips the run while the file is fresh. Dedupe on stable ids (place id, profile URL, post id), never on display text or dates. - The account-wide monthly cap lives in Console > Settings > Limits. Recommend one; do not change it yourself. Free plan: $5 a month, no rollover.
5. Run
The script starts the run asynchronously, polls until it ends, pages through the dataset (clean=true), writes a JSON array and prints runId, item count, usageTotalUsd and the Console URL. Exit codes: 2 no token, 3 over budget, 4 run did not succeed. Input fields differ per Actor: read the Actor's input schema first, never guess field names.
Synchronous shortcut for small jobs under 5 minutes: POST /v2/acts/<owner>~<actor>/run-sync-get-dataset-items?maxTotalChargeUsd=0.5&maxItems=20 (408 after 300 s, so use the script for anything bigger).
6. Deliver
- Save raw output next to the work (
data/or the increment'sreports/), plus one manifest line per run: date, Actor, input file, runId, items, cost. - Clean in a separate step: dedupe, normalise, drop fields the task does not need.
- Report counts, cost and the run link, and state what was not covered (blocked pages, missing emails).
- Never paste scraped personal data into public repos, issues or chats.
References
- references/recipes.md: step-by-step recipes (lead lists, social listening, SERP tracking, competitor monitoring, review mining, docs-to-RAG, news feeds with a daily cap).
- references/actors.md: Actors worth knowing, by job, with what to check before using each.
Security
- Security scanPassed, 100 of 100Pattern checks on the skill's files. Certified 15 Nov 2026.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingplugins/marketing/skills/apify-data-gathering/SKILL.md
Popularity
- Repository stars
- 47
- Repository forks
- 5
- 7-day trend score
- 0
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/anton-abyzov/vskill/apify-data-gathering)