version: "1.0.1" name: infrastructure-kubernetes description: Kubernetes debugging methodology and scripts. Use for pod crashes, CrashLoopBackOff, OOMKilled, deployment issues, resource problems, or container failures.

Kubernetes Debugging

Core Principle: Events Before Logs

ALWAYS check pod events BEFORE logs. Events explain 80% of issues faster:

• OOMKilled -> Memory limit exceeded
• ImagePullBackOff -> Image not found or auth issue
• FailedScheduling -> No nodes with enough resources
• CrashLoopBackOff -> Container crashing repeatedly

Access Mode

Default: skip cluster discovery, go straight to direct mode. Do not run list_clusters.py "just in case" before every namespace query — it's only needed when you specifically must target a different, remote, gateway-registered cluster than the one this agent is already running in/against.

Direct mode (kubectl / kubeconfig) — use this unless told otherwise

Run scripts WITHOUT --cluster-id. They use the local kubeconfig (~/.kube/config) or in-cluster service account.

python .claude/skills/infrastructure-kubernetes/scripts/list_pods.py -n otel-demo
python .claude/skills/infrastructure-kubernetes/scripts/get_events.py <pod-name> -n otel-demo
python .claude/skills/infrastructure-kubernetes/scripts/get_logs.py <pod-name> -n otel-demo --tail 100

You can also run kubectl directly if the scripts don't cover your use case:

kubectl get pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl logs -n <namespace> <pod-name> --tail=100
kubectl describe pod -n <namespace> <pod-name>

Gateway mode (remote clusters via k8s-gateway) — only if you actually need a different cluster

Only works if the K8S_GATEWAY_URL env var is set on this agent AND the target cluster is registered with the gateway. Most environments do NOT have this configured — list_clusters.py will tell you plainly if gateway mode is unavailable; when it does, do not pass `--cluster-id` at all, and do not try it "to be safe" against a second context — every context resolves to the same identical result, so a second attempt is a wasted, redundant call, not a more thorough one.

python .claude/skills/infrastructure-kubernetes/scripts/list_clusters.py
python .claude/skills/infrastructure-kubernetes/scripts/list_pods.py -n production --cluster-id <CLUSTER_ID>

If you ignore this and pass --cluster-id anyway while gateway mode is off, the scripts print a [k8s-gateway] ... ignored warning to stderr as a last-resort safety net — but treat that as a signal you already made a mistake, not as normal output to work around.

Available Scripts

All scripts are in .claude/skills/infrastructure-kubernetes/scripts/

list_pods.py - List pods with status

python .claude/skills/infrastructure-kubernetes/scripts/list_pods.py -n <namespace> [--label <selector>] [--json] [--cluster-id <id>]

Only -n/--namespace, --label, --json, and --cluster-id are supported — there is no --output/--show-age/etc. Pod age is only included with --json; the plain-text table does not show it.

get_events.py - Get pod events (USE FIRST!)

python .claude/skills/infrastructure-kubernetes/scripts/get_events.py <pod-name> -n <namespace>

get_logs.py - Get pod logs

python .claude/skills/infrastructure-kubernetes/scripts/get_logs.py <pod-name> -n <namespace> [--tail N] [--container NAME]

describe_pod.py - Detailed pod info

python .claude/skills/infrastructure-kubernetes/scripts/describe_pod.py <pod-name> -n <namespace>

describe_deployment.py - Deployment status and rollout history

python .claude/skills/infrastructure-kubernetes/scripts/describe_deployment.py <deployment-name> -n <namespace>

list_namespaces.py - List all namespaces

python .claude/skills/infrastructure-kubernetes/scripts/list_namespaces.py

get_resources.py - Resource usage vs limits

python .claude/skills/infrastructure-kubernetes/scripts/get_resources.py <pod-name> -n <namespace>

describe_node.py - Node status, conditions, and resource usage

python .claude/skills/infrastructure-kubernetes/scripts/describe_node.py <node-name>
python .claude/skills/infrastructure-kubernetes/scripts/describe_node.py --all

Debugging Workflows

Pod Not Starting (Pending/CrashLoopBackOff)

1. list_pods.py - Check pod status
2. get_events.py - Look for scheduling/pull/crash events
3. describe_pod.py - Check conditions and container states
4. get_logs.py - Only if events don't explain

Pod Restarting (OOMKilled/Crashes)

1. get_events.py - Check for OOMKilled or error events
2. get_resources.py - Compare usage vs limits
3. get_logs.py - Check for errors before crash
4. describe_pod.py - Check restart count and state

Deployment Not Progressing

1. describe_deployment.py - Check replica counts and rollout history
2. list_pods.py - Find stuck pods
3. get_events.py - Check events on stuck pods

Node Resource Issues

1. describe_node.py --all - Check all nodes
2. list_pods.py - Check for Pending pods
3. get_events.py - Look for FailedScheduling

Common Issues & Solutions

Output Format

When reporting findings, use this structure:

## Kubernetes Analysis
 
**Pod**: <name>
**Namespace**: <namespace>
**Status**: <phase> (Restarts: N)
 
### Events
- [timestamp] <reason>: <message>
 
### Issues Found
1. [Issue description with evidence]
 
### Root Cause Hypothesis
[Based on events and logs]
 
### Recommended Action
[Specific remediation step]