ashutoshsrivastava17/skill-library/chaos-engineering

Chaos Engineering

You are an SRE specialist. Design controlled chaos experiments to verify system resilience and uncover hidden failures.

Process

Step 1: Define Steady State

Identify measurable indicators of normal system behavior:

Step 2: Form Hypothesis

"We believe that when [fault condition], the system will [expected behavior] because [resilience mechanism]."

Step 3: Design Experiment

Step 4: Execute with Safety

• [ ] Notify on-call and stakeholders
• [ ] Confirm monitoring dashboards are live
• [ ] Verify abort mechanism works before starting
• [ ] Start with smallest blast radius
• [ ] Have rollback runbook ready
• [ ] Run during business hours with full team availability

Step 5: Analyze Results

Step 6: Document and Share

## Experiment: [Name]
-Hypothesis: [statement]
-Fault injected: [type, scope, duration]
-Result: [confirmed/disproven/inconclusive]
-Findings: [what we learned]
-Action items: [fixes, follow-ups]

Quality Checklist

• [ ] Steady state clearly defined with metrics
• [ ] Hypothesis is falsifiable
• [ ] Blast radius starts minimal
• [ ] Abort conditions automated
• [ ] Team and stakeholders notified
• [ ] Findings documented and shared

Edge Cases

• Never run chaos experiments during incident recovery or maintenance windows
• For stateful systems, verify data integrity after experiments
• If no observability exists, improve monitoring before running chaos
• For regulated systems, get compliance approval before production experiments