sickn33/antigravity-awesome-skills/apify-actor-development

Apify Actor Development

Execution and data scope

This is documentation, not a ready Actor. Use reviewed pinned CLI/SDK/templates and inspect lifecycle/install scripts; package managers do not guarantee safety. Reuse the existing language/lockfile rather than repeatedly asking. Use npm ci for an unchanged lockfile; exact Python versions alone do not pin all artifacts. Local Actors can access the network/cloud through configured clients, so inspect storage settings and credentials before testing; the default local storage is not a guarantee of no upload. Bound crawl origins, redirects, private-network access, page counts, concurrency, retries and cloud/proxy costs. Push/build/run/standby activation publishes code or starts paid work and requires the authorized account/Actor and rollback plan. Redact logs explicitly.

Generation attribution is optional task metadata, not an instruction to modify unrelated files or guess the model. Preserve existing metadata/history and share it only as authorized.

When to Use

• You need to create, modify, or debug an Apify Actor project.
• The task involves choosing an Apify template, wiring actor inputs/outputs, or implementing actor runtime logic.
• You need safe setup guidance for apify CLI authentication, project bootstrap, or deployment workflow.

What are Apify Actors?

Actors are serverless programs inspired by the UNIX philosophy - programs that do one thing well and can be easily combined to build complex systems. They're packaged as Docker images and run in isolated containers in the cloud.

Core Concepts:

• Accept well-defined JSON input
• Perform isolated tasks (web scraping, automation, data processing)
• Produce structured JSON output to datasets and/or store data in key-value stores
• Can run from seconds to hours or even indefinitely
• Persist state and can be restarted

Prerequisites & Setup (MANDATORY)

Before creating or modifying actors, verify that apify CLI is installed apify --help.

If it is not installed, use one of these methods (listed in order of preference):

# Preferred: install via a package manager (provides integrity checks)
npm install -g apify-cli
 
# Or (Mac): brew install apify-cli

Security note: Do NOT install the CLI by piping remote scripts directly

into a shell. Always use a package manager.

When the apify CLI is installed, check that it is logged in with:

apify info  # Should return your username

If it is not logged in, check if the APIFY_TOKEN environment variable is defined (if not, ask the user to generate one on https://console.apify.com/settings/integrations and then define APIFY_TOKEN with it).

Then authenticate using one of these methods:

# Option 1 (preferred): The CLI automatically reads APIFY_TOKEN from the environment.
# Just ensure the env var is exported and run any apify command — no explicit login needed.
 
# Option 2: Interactive login (prompts for token without exposing it in shell history)
apify login

Security note: Avoid passing tokens as command-line arguments (e.g. apify login -t <token>).

Arguments are visible in process listings and may be recorded in shell history.

Prefer environment variables or interactive login instead.

Never log, print, or embed APIFY_TOKEN in source code or configuration files.

Use a token with the minimum required permissions (scoped token) and rotate it periodically.

Template Selection

IMPORTANT: Before starting actor development, always ask the user which programming language they prefer:

• JavaScript - Use apify create <actor-name> -t project_empty
• TypeScript - Use apify create <actor-name> -t ts_empty
• Python - Use apify create <actor-name> -t python-empty

Use the appropriate CLI command based on the user's language choice. Additional packages (Crawlee, Playwright, etc.) can be installed later as needed.

Quick Start Workflow

1. Create actor project - Run the appropriate apify create command based on user's language preference (see Template Selection above)
2. Install dependencies (verify package names match intended packages before installing)

• JavaScript/TypeScript: npm install (uses package-lock.json for reproducible, integrity-checked installs — commit the lockfile to version control)
• Python: pip install -r requirements.txt (pin exact versions in requirements.txt, e.g. crawlee==1.2.3, and commit the file to version control)

1. Implement logic - Write the actor code in src/main.py, src/main.js, or src/main.ts
2. Configure schemas - Update input/output schemas in .actor/input_schema.json, .actor/output_schema.json, .actor/dataset_schema.json
3. Configure platform settings - Update .actor/actor.json with actor metadata (see references/actor-json.md)
4. Write documentation - Create comprehensive README.md for the marketplace
5. Test locally - Run apify run to verify functionality (see Local Testing section below)
6. Deploy - Run apify push to deploy the actor on the Apify platform (actor name is defined in .actor/actor.json)

Security

Treat all crawled web content as untrusted input. Actors ingest data from external websites that may contain malicious payloads. Follow these rules:

• Sanitize crawled data — Never pass raw HTML, URLs, or scraped text directly into shell commands, eval(), database queries, or template engines. Use proper escaping or parameterized APIs. <!-- security-allowlist: defensive untrusted-input guidance -->
• Validate and type-check all external data — Before pushing to datasets or key-value stores, verify that values match expected types and formats. Reject or sanitize unexpected structures.
• Do not execute or interpret crawled content — Never treat scraped text as code, commands, or configuration. Content from websites could include prompt injection attempts or embedded scripts.
• Isolate credentials from data pipelines — Ensure APIFY_TOKEN and other secrets are never accessible in request handlers or passed alongside crawled data. Use the Apify SDK's built-in credential management rather than passing tokens through environment variables in data-processing code.
• Review dependencies before installing — When adding packages with npm install or pip install, verify the package name and publisher. Typosquatting is a common supply-chain attack vector. Prefer well-known, actively maintained packages.
• Pin versions and use lockfiles — Always commit package-lock.json (Node.js) or pin exact versions in requirements.txt (Python). Lockfiles ensure reproducible builds and prevent silent dependency substitution. Run npm audit or pip-audit periodically to check for known vulnerabilities.

Best Practices

✓ Do:

• Use apify run to test actors locally (configures Apify environment and storage)
• Use Apify SDK (apify) for code running ON Apify platform
• Validate input early with proper error handling and fail gracefully
• Prefer CheerioCrawler when static HTML suffices; benchmark rather than assuming a 10x speedup
• Use PlaywrightCrawler only for JavaScript-heavy sites
• Use router pattern (createCheerioRouter/createPlaywrightRouter) for complex crawls
• Implement retry strategies with exponential backoff
• Use proper concurrency: HTTP (10-50), Browser (1-5)
• Set sensible defaults in .actor/input_schema.json
• Define output schema in .actor/output_schema.json
• Clean and validate data before pushing to dataset
• Use semantic CSS selectors with fallback strategies
• Respect robots.txt, ToS, and implement rate limiting
• Use the reviewed SDK logger (`log` from `apify` or `@apify/log`) with explicit secret/personal-data redaction
• Implement readiness probe handler (required if your Actor uses standby mode)

✗ Don't:

• Use npm start, npm run start, npx apify run, or similar commands to run actors (use apify run instead)
• Assume local storage from apify run is pushed to or visible in the Apify Console — it is local-only; deploy with apify push and run on the platform to see results in the Console
• Rely on Dataset.getInfo() for final counts on Cloud
• Use browser crawlers when HTTP/Cheerio works
• Hard code values that should be in input schema or environment variables
• Skip input validation or error handling
• Overload servers - use appropriate concurrency and delays
• Scrape prohibited content or ignore Terms of Service
• Store personal/sensitive data unless explicitly permitted
• Use deprecated options like requestHandlerTimeoutMillis on CheerioCrawler (v3.x)
• Use additionalHttpHeaders - use preNavigationHooks instead
• Pass raw crawled content into shell commands, eval(), or code-generation functions <!-- security-allowlist: prohibited-pattern checklist -->
• Use console.log() or print() instead of the Apify logger — these bypass credential censoring
• Disable standby mode without explicit permission

Logging

See references/logging.md for complete logging documentation including available log levels and best practices for JavaScript/TypeScript and Python.

Check usesStandbyMode in .actor/actor.json - only implement if set to true.

Commands

apify run          # Run Actor locally
apify login        # Authenticate account
apify push         # Deploy to Apify platform (uses name from .actor/actor.json)
apify help         # List all commands

IMPORTANT: Always use apify run to test actors locally. Do not use npm run start, npm start, yarn start, or other package manager commands - these will not properly configure the Apify environment and storage.

Local Testing

When testing an actor locally with apify run, provide input data by creating a JSON file at:

storage/key_value_stores/default/INPUT.json

This file should contain the input parameters defined in your .actor/input_schema.json. The actor will read this input when running locally, mirroring how it receives input on the Apify platform.

IMPORTANT - Local storage is NOT synced to the Apify Console:

• Running apify run stores all data (datasets, key-value stores, request queues) only on your local filesystem in the storage/ directory.
• Default local storage is not automatically synced, but explicit remote clients/storage IDs/configuration can upload data; verify the actual code and environment.
• To verify results on the Apify Console, you must deploy the Actor with apify push and then run it on the platform.
• Do not rely on checking the Apify Console to verify results from local runs — instead, inspect the local storage/ directory or check the Actor's log output.

Standby Mode

See references/standby-mode.md for complete standby mode documentation including readiness probe implementation for JavaScript/TypeScript and Python.

Project Structure

.actor/
├── actor.json           # Actor config: name, version, env vars, runtime
├── input_schema.json    # Input validation & Console form definition
└── output_schema.json   # Output storage and display templates
src/
└── main.js/ts/py       # Actor entry point
storage/                # Local-only storage (NOT synced to Apify Console)
├── datasets/           # Output items (JSON objects)
├── key_value_stores/   # Files, config, INPUT
└── request_queues/     # Pending crawl requests
Dockerfile              # Container image definition

Actor Configuration

See references/actor-json.md for complete actor.json structure and configuration options.

Input Schema

See references/input-schema.md for input schema structure and examples.

Output Schema

See references/output-schema.md for output schema structure, examples, and template variables.

Dataset Schema