papadopouloskyriakos/agentic-chatops/infra-triage

Infrastructure Alert Triage (Level 1 + Level 2)

When you see a message starting with [LibreNMS] ALERT in #infra-nl-prod or #infra-gr-prod, you MUST automatically run this full triage flow using the exec tool. Do NOT ask questions — just execute.

Usage

./skills/infra-triage/infra-triage.sh <hostname> "<rule_name>" <severity> [--site nl|gr]

Site is auto-detected from hostname prefix (grskg* → GR, otherwise NL). Use --site to override.

Environment Variables

What the Script Does

Step 0: Issue Deduplication

Searches YouTrack for existing open issues with the same hostname (created within 24h). If found, reuses instead of creating a duplicate. Reopens if in "To Verify". Also searches for related issues (same alert rule on different hosts within 12h).

Step 1: Create/Reuse YouTrack Issue

Creates issue in the site's YT project (IFRNLLEI01PRD for NL, IFRGRSKG01PRD for GR) or reuses existing. Registers callback with n8n for dedup tracking. Sets custom fields (Hostname, Alert Rule, Severity, Alert Source, VMID, PVE Host).

Step 2: Investigation

• Classify device via LibreNMS API (OS, type, hardware)
• Find host in PVE (IaC repo grep for hostname)
• Check LXC/VM status on Proxmox via SSH
• Check connectivity (ping from container)
• K8s diagnostics if hostname matches k8s-*
• Control plane deep investigation for K8s controller nodes (etcd pods+logs, apiserver pods, CP resource usage)
• Check Docker services
• Query LibreNMS API for device status

Step 3: Post Findings to YouTrack

Posts investigation results as YT comment, including related issues and recurring alert flag.

Step 4: Escalation

Always escalates to Claude Code unless SKIP_ESCALATION=true. Includes escalation reason (standard/flapping/recurring).

CRITICAL RULES

1. ALWAYS use the `exec` tool to run the script. Do NOT describe what to do — execute it.
2. Do NOT make changes. Level 2 is READ-ONLY. No pct start, pct stop, docker restart, config edits, etc.
3. Do NOT skip steps. Run ALL investigation steps even if the issue seems obvious.
4. ALWAYS escalate to Claude Code after posting findings (unless SKIP_ESCALATION).
5. After reporting output, add your CONFIDENCE score (see SOUL.md for format).