gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoring
Claude Code and agent skill by gemini-cli-extensions/data-agent-kit-starter-pack
- community
Checking that the SKILL.md file is still in the repository.
Guides the authoring and validation of Apache Airflow DAGs for Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Covers environment context discovery, Airflow 2 vs 3 compatibility, authoring best practices, and local/remote validation processes. Use when creating or extending an Airflow DAG. Don't use when authoring Python code unrelated to Airflow DAGs.
Install
npx vskill@latest install gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoringSKILL.md
The same SKILL.md was published earlier as gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoring.
GCP Managed Airflow DAG Authoring Guide
This skill guides you through authoring and validating Apache Airflow DAGs for Managed Service for Apache Airflow (MSAA; formerly Cloud Composer) environments.
Phase 1: Context Discovery
[!IMPORTANT]Before writing any DAG code, you MUST understand the constraints (e.g. versionof Airflow) and capabilities of your target environment if user is willing toprovide them.
1.1 Identify Target Environment & Access
Determine if you have direct access to the target Managed Airflow environment, local development environment or if you are working offline (only changing local files without validation).
- If environment access is available: Use
gcloudto inspect the
environment (see Section 1.3).
- If offline: Rely on user provided details.
1.2 Identify Development Environment
Determine if a local development environment is available.
- Check if
composer-devCLI is installed. - Check if a local Python environment with
airflowis available.
1.3 Inspect Target Environment (if available and requested)
Run the following commands to discover version constraints:
- Get Airflow/Image Version:
``bash gcloud composer environments describe <ENV_NAME> \ --location <REGION> \ --format="value(config.softwareConfig.imageVersion)" ``
- Get Installed Packages (Versions):
``bash gcloud composer environments describe <ENV_NAME> \ --location <REGION> \ --format="value(config.softwareConfig.pypiPackages)" ``
- Get DAGs GCS Bucket:
``bash gcloud composer environments describe <ENV_NAME> \ --location <REGION> \ --format="value(config.dagGcsPrefix)" ``
Phase 2: DAG Authoring Best Practices
2.1 General Airflow Best Practices
- Idempotency: Every task SHOULD be idempotent. Running it multiple times
with the same inputs (e.g., execution date) SHOULD produce the same result and not duplicate data.
- No Top-Level Code Execution: Do NOT execute database queries, external
API calls, or heavy computations at the top level of the DAG file (outside of tasks/operators). This code runs every few seconds during DAG parsing and will degrade performance.
- Explicit Catchup: Always set
catchup=Falsein the DAG definition
unless historical backfilling is explicitly required.
- Use Airflow Variables/Connections: Never hardcode credentials or
environment-specific configs. Use Variable.get() (with deserialize_json=True if applicable) and BaseHook.get_connection(). Access variables via Jinja templates (e.g., {{ var.value.my_var }}) to avoid database calls during DAG parsing.
2.2 Airflow 2 vs Airflow 3 Compatibility
Reference @skill:gcp-managed-airflow-migrations to navigate adjusting the code to specific target Airflow version.
Phase 3: Validation Process
[!IMPORTANT]You MUST validate DAGs before concluding your task.
3.1 Local Validation (Offline/Pre-deployment)
3.1.1 Static Analysis & Linting
Use ruff or pylint if available.
- If targeting Airflow 3, check with Airflow 3 rules if rulesets are
available.
3.1.2 Local Dev Environment (composer-dev)
If the user has composer-dev configured:
- Copy the DAG to the local directory with DAGs:
``bash cp path/to/dag.py $(composer-dev describe <LOCAL_ENV> --format="value(dags_directory)") ``
- Verify parsing:
``bash composer-dev run-airflow-cmd <LOCAL_ENV> dags list-import-errors ``
3.2: Target Environment Validation
Only perform these steps if you have GCP access and are authorized to deploy to a target environment.
3.2.1 Deploy to GCS
Upload the DAG to the target environment's GCS bucket:
3.2.2 Verify via Airflow CLI
Wait 1-2 minutes for the scheduler to parse the file, then run:
- Check for Import Errors:
``bash gcloud composer environments run <ENV_NAME> \ --location <REGION> \ dags list-import-errors ``
Pass Criteria: Output should be "No data found" or empty.
- Verify DAG is Listed:
``bash gcloud composer environments run <ENV_NAME> \ --location <REGION> \ dags list | grep <DAG_ID> ``
3.2.3 Monitor Cloud Logging
Check for runtime parsing errors in Cloud Logging:
Definition of Done
- DAG code adheres to Airflow version constraints of the target environment.
- DAG code follows best practices (no top-level execution, idempotent if
possible).
- DAG parses locally without import errors.
- (If environment is available) DAG is deployed to the target environment and
verified to have no import errors.
Security
- Security scanPassed, 93 of 100Pattern checks on the skill's files.
- Model reviewNot runA model review is scheduled for the next rescan.
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/gcp-managed-airflow-dag-authoring/SKILL.md
Popularity
- Repository stars
- 187
- Repository forks
- 57
- 7-day trend score
- 26
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoring)