papadopouloskyriakos/agentic-chatops/k8s-triage

Kubernetes Alert Triage

When you see a message containing k8s-triage.sh in #infra-nl-prod or #infra-gr-prod, you MUST run it using the exec tool immediately. Do NOT ask questions.

Usage

./skills/k8s-triage/k8s-triage.sh "<alertname>" "<severity>" "<namespace>" "<summary>" "<node>" "<pod>" [--site nl|gr]

Site is auto-detected from node name prefix (grskg* → GR, otherwise NL). Use --site to override.

Environment Variables

CRITICAL RULES

1. ALWAYS use the `exec` tool to run the script. Do NOT describe what to do.
2. Do NOT make changes. All investigation is READ-ONLY (kubectl get/describe/logs only).
3. The script handles everything — YT dedup, issue creation, investigation, findings, escalation.

What's New (v2)

Issue Deduplication (Step 0)

Before creating a new YT issue, the script searches YouTrack for existing open issues with the same Alert Rule created within 24h. If found, it reuses that issue instead of creating a duplicate. Also searches for related issues (same node or namespace within 12h) and lists them in the findings.

Control Plane Deep Investigation

When the alert involves a control plane component (apiserver, etcd, controller-manager, scheduler), the script performs cross-component investigation:

• etcd pod status + logs (all members)
• apiserver pods + restart counts + error logs (all instances)
• controller-manager + scheduler status
• Control plane resource usage (kubectl top)

Smart Escalation (Step 5)

Escalation is triggered by ANY of these conditions:

• Critical severity (unchanged)
• Recurring alert (reusing an existing issue)
• Forced escalation (FORCE_ESCALATE=true, set by n8n for flapping alerts)
• Control plane warning (all control plane alerts escalate regardless of severity)

Integration Details

Register Callback

After creating/reusing the YouTrack issue, the script POSTs a register callback to n8n so the Prometheus Alert Receiver can track the issue:

POST /webhook/prometheus-alert (or /webhook/prometheus-alert-gr for GR site)
{
  "action": "register",
  "alertKey": "<alertname>:<namespace>",
  "issueId": "<IFRNLLEI01PRD-NNN or IFRGRSKG01PRD-NNN>"
}

YouTrack Custom Fields Set

The script sets 6 custom fields on newly created issues:

• Hostname — K8s node name (or "k8s-cluster" for cluster-wide alerts)
• Alert Rule — Prometheus alert name (e.g., KubePodCrashLooping)
• Severity — critical or warning
• Namespace — K8s namespace where the alert fired
• Pod — affected pod name (if applicable)
• Alert Source — prometheus (distinguishes from LibreNMS alerts)