irahardianto/agy-perf-optimization-demo/data-engineering

Data Engineering Principles

Guidelines for building reliable, scalable data pipelines and platforms.

When to Invoke

• Designing data pipelines (ETL/ELT)
• Evaluating batch vs stream processing
• Data quality and governance requirements
• Data warehouse/lake architecture decisions

Pipeline Architecture

Design Principles

1. Idempotent pipelines — re-running produces same result. Use upserts, not inserts.
2. Schema evolution — handle new fields without breaking consumers.
3. Exactly-once processing — deduplication at ingestion, idempotency keys.
4. Incremental processing — process only new/changed data, not full reloads.

Patterns

Data Quality

Checks (Non-Negotiable)

• Completeness — no unexpected nulls in required fields
• Uniqueness — no duplicate records on primary keys
• Referential integrity — foreign keys resolve
• Freshness — data arrives within SLA window
• Volume — row counts within expected range (±threshold)

Framework

Source → Validate (schema, nulls, types) → Transform → Validate (business rules) → Load → Verify (counts, checksums)

Orchestration

Best Practices

• DAGs should be idempotent and retriable
• Separate orchestration from computation
• Use backfill capabilities for historical reprocessing
• Alert on SLA breaches, not just failures

Data Modeling

Governance

• Data lineage tracked (source → transformation → destination)
• Access controls per dataset/table
• PII identified and masked/encrypted
• Retention policies documented and automated

Related

• Database Design Principles @.agents/rules/database-design-principles.md
• SQL Idioms @.agents/skills/sql-idioms/SKILL.md
• Logging Implementation @.agents/skills/logging-implementation/SKILL.md