jsoehner/skills-backup/airflow-dag-patterns

Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

Use this skill when

• Creating data pipeline orchestration with Airflow
• Designing DAG structures and dependencies
• Implementing custom operators and sensors
• Testing Airflow DAGs locally
• Setting up Airflow in production
• Debugging failed DAG runs

Do not use this skill when

• You only need a simple cron job or shell script
• Airflow is not part of the tooling stack
• The task is unrelated to workflow orchestration

Instructions

1. Identify data sources, schedules, and dependencies.
2. Design idempotent tasks with clear ownership and retries.
3. Implement DAGs with observability and alerting hooks.
4. Validate in staging and document operational runbooks.
5. Trigger the local memory capture using the standardized capture script.

Refer to resources/implementation-playbook.md for detailed patterns, checklists, and templates.

Safety

• Avoid changing production DAG schedules without approval.
• Test backfills and retries carefully to prevent data duplication.

Resources

• resources/implementation-playbook.md for detailed patterns, checklists, and templates.

Anti-Patterns

• NEVER write Airflow tasks that are not idempotent (running them multiple times must yield the same result).
• NEVER hardcode credentials or secrets within DAG files; always pull from Airflow Connections or Variables.
• NEVER perform heavy database querying or long-running computations directly in the DAG parsing context.