<< All versions

Skill v1.0.0

currentTrusted Publisher100/100
microsoft/pilotswarm/incident-response
──Details
PublishedOctober 19, 2026 at 06:42 AM
Content Hashsha256:8f1533af011b9e80...
Git SHA
──Files
Files (1 file, 1.8 KB)
SKILL.md1.8 KBactive
SKILL.md · 40 lines · 1.8 KB

version: "1.0.0" name: incident-response description: Domain knowledge for investigating production incidents — triage steps, correlation patterns, and escalation criteria.


Incident Response

Triage Checklist

When investigating an incident, follow this order:

  1. Identify the blast radius — which services are affected?
  2. Check health endpoints — are services responding?
  3. Review error rates — sudden spikes indicate the incident start time
  4. Check CPU/memory — resource exhaustion is a common root cause
  5. Search logs — look for exceptions, timeouts, connection errors

Common Root Causes

SymptomLikely CauseInvestigation Steps
High error rate + normal CPUUpstream dependency failureCheck downstream service health, look for timeout errors in logs
High CPU + high error rateResource exhaustionCheck for memory leaks, runaway queries, or traffic spikes
Intermittent errorsNetwork issues or partial deployCheck if errors correlate with specific pods, look for connection reset logs
Cascading failures across servicesShared dependency downIdentify the common downstream service, check database/cache health

Correlation Patterns

  • Time correlation: If service A errors spike 30s before service B, A likely caused B's issues
  • Error type correlation: Same exception across services = shared dependency
  • Traffic correlation: Error rate proportional to traffic = capacity issue

Escalation Criteria

  • Error rate > 10% for 5+ minutes → P1 (page on-call)
  • Service completely down → P0 (all-hands)
  • Data integrity issues → P0 (immediate escalation)
  • Performance degraded but functional → P2 (next business day)
All versions