gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoring

Claude Code and agent skill by gemini-cli-extensions/data-agent-kit-starter-pack

Trust tier 2
  • community
LLM review pending
Source check pending

Checking that the SKILL.md file is still in the repository.

Guides the authoring and validation of Apache Airflow DAGs for Managed Service for Apache Airflow (MSAA; formerly Cloud Composer). Covers environment context discovery, Airflow 2 vs 3 compatibility, authoring best practices, and local/remote validation processes. Use when creating or extending an Airflow DAG. Don't use when authoring Python code unrelated to Airflow DAGs.

Install

npx vskill@latest install gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoring

SKILL.md

The same SKILL.md was published earlier as gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoring.

GCP Managed Airflow DAG Authoring Guide

This skill guides you through authoring and validating Apache Airflow DAGs for Managed Service for Apache Airflow (MSAA; formerly Cloud Composer) environments.


Phase 1: Context Discovery

[!IMPORTANT]
Before writing any DAG code, you MUST understand the constraints (e.g. version
of Airflow) and capabilities of your target environment if user is willing to
provide them.
1.1 Identify Target Environment & Access

Determine if you have direct access to the target Managed Airflow environment, local development environment or if you are working offline (only changing local files without validation).

  • If environment access is available: Use gcloud to inspect the

environment (see Section 1.3).

  • If offline: Rely on user provided details.
1.2 Identify Development Environment

Determine if a local development environment is available.

  • Check if composer-dev CLI is installed.
  • Check if a local Python environment with airflow is available.
1.3 Inspect Target Environment (if available and requested)

Run the following commands to discover version constraints:

  1. Get Airflow/Image Version:

``bash gcloud composer environments describe <ENV_NAME> \ --location <REGION> \ --format="value(config.softwareConfig.imageVersion)" ``

  1. Get Installed Packages (Versions):

``bash gcloud composer environments describe <ENV_NAME> \ --location <REGION> \ --format="value(config.softwareConfig.pypiPackages)" ``

  1. Get DAGs GCS Bucket:

``bash gcloud composer environments describe <ENV_NAME> \ --location <REGION> \ --format="value(config.dagGcsPrefix)" ``


Phase 2: DAG Authoring Best Practices

2.1 General Airflow Best Practices
  • Idempotency: Every task SHOULD be idempotent. Running it multiple times

with the same inputs (e.g., execution date) SHOULD produce the same result and not duplicate data.

  • No Top-Level Code Execution: Do NOT execute database queries, external

API calls, or heavy computations at the top level of the DAG file (outside of tasks/operators). This code runs every few seconds during DAG parsing and will degrade performance.

  • Explicit Catchup: Always set catchup=False in the DAG definition

unless historical backfilling is explicitly required.

  • Use Airflow Variables/Connections: Never hardcode credentials or

environment-specific configs. Use Variable.get() (with deserialize_json=True if applicable) and BaseHook.get_connection(). Access variables via Jinja templates (e.g., {{ var.value.my_var }}) to avoid database calls during DAG parsing.

2.2 Airflow 2 vs Airflow 3 Compatibility

Reference @skill:gcp-managed-airflow-migrations to navigate adjusting the code to specific target Airflow version.


Phase 3: Validation Process

[!IMPORTANT]
You MUST validate DAGs before concluding your task.
3.1 Local Validation (Offline/Pre-deployment)
3.1.1 Static Analysis & Linting

Use ruff or pylint if available.

bash
ruff check path/to/dag.py
  • If targeting Airflow 3, check with Airflow 3 rules if rulesets are

available.

3.1.2 Local Dev Environment (composer-dev)

If the user has composer-dev configured:

  1. Copy the DAG to the local directory with DAGs:

``bash cp path/to/dag.py $(composer-dev describe <LOCAL_ENV> --format="value(dags_directory)") ``

  1. Verify parsing:

``bash composer-dev run-airflow-cmd <LOCAL_ENV> dags list-import-errors ``

3.2: Target Environment Validation

Only perform these steps if you have GCP access and are authorized to deploy to a target environment.

3.2.1 Deploy to GCS

Upload the DAG to the target environment's GCS bucket:

bash
gcloud storage cp path/to/dag.py gs://<TARGET_BUCKET>/dags/
3.2.2 Verify via Airflow CLI

Wait 1-2 minutes for the scheduler to parse the file, then run:

  1. Check for Import Errors:

``bash gcloud composer environments run <ENV_NAME> \ --location <REGION> \ dags list-import-errors ``

Pass Criteria: Output should be "No data found" or empty.

  1. Verify DAG is Listed:

``bash gcloud composer environments run <ENV_NAME> \ --location <REGION> \ dags list | grep <DAG_ID> ``

3.2.3 Monitor Cloud Logging

Check for runtime parsing errors in Cloud Logging:

query
resource.type="cloud_composer_environment"
resource.labels.environment_name="<ENV_NAME>"
log_id("airflow-scheduler")
severity>=ERROR
textPayload:"<DAG_FILE_NAME>"

Definition of Done

  • DAG code adheres to Airflow version constraints of the target environment.
  • DAG code follows best practices (no top-level execution, idempotent if

possible).

  • DAG parses locally without import errors.
  • (If environment is available) DAG is deployed to the target environment and

verified to have no import errors.

Security

  • Security scanPassed, 93 of 100Pattern checks on the skill's files.
  • Model reviewNot runA model review is scheduled for the next rescan.

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

Repository stars
187
Repository forks
57
7-day trend score
26

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoring/badge)](https://verified-skill.com/skills/gemini-cli-extensions/data-agent-kit-starter-pack/gcp-managed-airflow-dag-authoring)

Something wrong with this skill or its page? Tell us.