nitinjain999/platform-skills/karpenter

Design, install, debug, review, plan capacity, audit, migrate, and upgrade Karpenter on EKS.

Verify current stable version before installing (Karpenter uses OCI, not a Helm repo):

crane ls public.ecr.aws/karpenter/karpenter | grep -E '^[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -5

Pin to the version from that output, e.g.:

helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter --version x.y.z

All guidance targets the karpenter.sh/v1 API. The v0.x Provisioner/AWSNodeTemplate API was removed in v1.0 — if you are on v0.x, use migrate mode first.

Interactive Wizard (fires when no arguments are provided)

When invoked with no arguments, ask before proceeding:

Q1 — Mode?

What do you need?
  1. generate — design NodePool and EC2NodeClass from requirements
  2. debug    — diagnose why nodes are not provisioning or pods are stuck Pending
  3. review   — production-readiness review of existing NodePool/EC2NodeClass
  4. audit    — reconstruct scale-out/scale-in history and explain why it happened
  5. plan     — predict what Karpenter would provision for a given workload before deploying
  6. migrate  — move from Cluster Autoscaler to Karpenter
  7. upgrade  — upgrade Karpenter version (including v0.x → v1.x CRD migration)
 
Enter 1–7 or mode name:

Q2 — Environment context (ask after mode, one question at a time):

1. EKS cluster version?  (e.g. 1.29, 1.30, 1.31)
2. Karpenter version currently installed (or target version for fresh install)?
3. Identity method for Karpenter controller?
     a) EKS Pod Identity  (recommended, requires EKS 1.24+)
     b) IRSA              (IAM Roles for Service Accounts)
4. Is this a private cluster (no public API endpoint)?  [yes/no]
5. Do you use Spot instances?  [yes / no / mixed]
6. Are you migrating from Cluster Autoscaler?  [yes/no]

Use the answers to set defaults for every section below. If the user pastes a manifest or error, infer as much as possible and skip questions already answered.

Mode: generate

Design a production-ready NodePool and EC2NodeClass from requirements.

Steps:

1. Collect (or infer from context):

• Workload profile: general-purpose / compute-optimised / memory-optimised / GPU / Spot-flex
• Instance family preferences (e.g. m, c, r, g4dn) and architecture (amd64, arm64/Graviton, or both)
• AMI family: AL2023 (default, recommended) | Bottlerocket | Windows2022
• Environment: dev/staging or production — this drives the AMI strategy
• Whether scale-to-zero is needed (requires limits planning)
• Disruption tolerance: can workloads survive consolidation? Are there PDBs?
• GitOps method: Flux / Argo CD / direct kubectl

1. Generate EC2NodeClass with:

• amiSelectorTerms: use alias: al2023@latest for dev/staging; use a pinned AMI ID (id: ami-xxxx) for production. Floating @latest in production means an untested AMI can land on fleet nodes during any scheduled SSM parameter update. Ask the user which environment this is for before choosing.

> Ask user: "What is your EKS cluster name?" — substitute into karpenter.sh/discovery: <cluster-name> before applying.

• subnetSelectorTerms and securityGroupSelectorTerms using karpenter.sh/discovery: <cluster-name> tags
• instanceProfile or role matching the Karpenter node IAM role
• blockDeviceMappings with encrypted EBS and IMDSv2 enforced via metadataOptions
• tags including at minimum karpenter.sh/discovery, Environment, and billing tags

1. Generate NodePool with:

• spec.template.spec.requirements covering at least 3 instance families and both On-Demand/Spot capacity types (or explicit single type)
• minValues on instance family or size requirements to guarantee Spot diversity
• limits on CPU and memory — never omit, uncapped NodePools can runaway
• weight if this is one of multiple NodePools (higher weight = preferred)
• disruption block: consolidationPolicy, consolidateAfter, budgets
• expireAfter set to 720h (30 days) unless the user has a reason to extend

1. Generate companion resources:

• IRSA or Pod Identity IAM policy (minimum permissions — see references)
• PodDisruptionBudget for stateful workloads that will land on Karpenter nodes
• Flux Kustomization dependency or Argo CD syncWave annotation if GitOps is in use

1. Show validation:

``bash kubectl apply --dry-run=server -f ec2nodeclass.yaml kubectl apply --dry-run=server -f nodepool.yaml kubectl describe nodepool <name> # check Conditions: Ready=True ``

Reference: references/karpenter.md → NodePool design, EC2NodeClass, IAM

Rollback: kubectl delete nodepool <name> && kubectl delete ec2nodeclass <name> — Karpenter immediately stops provisioning nodes from these templates. Existing nodes remain until drained by the scheduler or TTL.

Mode: debug

Diagnose why pods are stuck Pending or nodes are not provisioning.

Steps:

1. Collect evidence first — do not suggest fixes before seeing output:

```bash # Pending pods and their scheduling failure reason kubectl get pods -A --field-selector=status.phase=Pending kubectl describe pod <name> -n <ns> # look for Events: FailedScheduling

# NodeClaim lifecycle kubectl get nodeclaim -A kubectl describe nodeclaim <name> # look for Conditions block

# NodePool status kubectl get nodepool -A kubectl describe nodepool <name> # look for: Ready condition, limits

# Karpenter controller logs (most diagnostic signal is here) kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter \ --since=30m --tail=200

# Node registration failures kubectl get nodes kubectl describe node <new-node-name> # look for NotReady or taint issues ```

1. Work through this decision tree in order:

1. For IAM errors — test the exact API calls Karpenter makes:

``bash # Test from a pod in the karpenter namespace using the controller SA kubectl run -it --rm debug --image=amazon/aws-cli \ --serviceaccount=karpenter -n karpenter -- \ aws ec2 describe-instance-types --region eu-north-1 ``

1. State the most likely root cause with the exact field or annotation to fix. Show the corrected spec section. Show the command to verify the fix worked.

Reference: references/karpenter.md → Troubleshooting

Rollback: If you edited NodePool/EC2NodeClass during debug: kubectl apply -f <original-backup>.yaml to restore. Always take a backup before editing: kubectl get nodepool <name> -o yaml > nodepool-backup.yaml.

Mode: review

Production-readiness review of NodePool and EC2NodeClass manifests.

Review in this exact priority order:

Correctness

• Does EC2NodeClass.spec.instanceProfile or .spec.role reference a valid node IAM role?
• Are subnetSelectorTerms and securityGroupSelectorTerms using stable selectors (tags, not names which can change)?
• Does NodePool.spec.template.spec.nodeClassRef.name match the EC2NodeClass metadata name?
• Are requirements satisfiable — does the cluster have subnets in the selected AZs with enough IP space?
• Is limits set? An uncapped NodePool will provision nodes without bound on runaway workloads.

Cost

• Are at least 3 instance families listed for Spot (narrow families = frequent InsufficientCapacityError)?
• Is minValues set on instance family or size requirements to enforce Spot diversity?
• Is Graviton (arm64) included where workloads support it (~20% cheaper on EC2)?
• Is consolidationPolicy: WhenEmptyOrUnderutilized enabled? Idle nodes are silent cost leaks.
• Is expireAfter set? Nodes that run indefinitely accumulate AMI drift and resist consolidation.

Reliability

• Are PDBs in place for workloads that will land on these nodes?
• Does disruption.budgets protect critical workloads from simultaneous eviction?
• Is terminationGracePeriod set on the NodePool (default 48h — may be too long or too short)?
• Does startupTaints prevent workloads scheduling onto nodes before DaemonSets are ready?

Security

• Is amiSelectorTerms using alias: al2023@latest or a pinned, scanned AMI ID? (Floating latest in production means untested AMIs can land on nodes.)
• Is EBS encrypted? (blockDeviceMappings[].ebs.encrypted: true)
• Is IMDSv2 enforced? (metadataOptions.httpTokens: required, httpPutResponseHopLimit: 1)
• Do EC2 instance tags include karpenter.sh/discovery: <cluster-name> — and is this tag protected by an SCP/tag policy against modification?
• Does the node IAM role follow least privilege? (No ec2:*, no iam:*)

GitOps safety

• Does the Flux/Argo Kustomization dependsOn the Karpenter Helm release so CRDs exist before NodePool?
• Is the IAM role ARN injected via postBuild.substitute or Argo CD ApplicationSet parameter, not hardcoded?

Separate findings into:

• Critical — must fix before deploying to production
• Improvement — should fix (cost, resilience)
• Note — informational (consider Graviton, document disruption tolerance)

Reference: references/karpenter.md → NodePool design, Security, Cost strategy

Mode: audit

Reconstruct what scaled out or in, when, and why — for post-incident analysis or cost review.

Steps:

1. Collect the timeline from three layers:

```bash # Layer 1: NodeClaim lifecycle events (node creation and termination) kubectl get events -A \ --field-selector involvedObject.kind=NodeClaim \ --sort-by='.lastTimestamp' | tail -50

# Layer 2: Disruption reason on terminated nodes (if node still exists) kubectl get node <name> \ -o jsonpath='{.annotations.karpenter\.sh/disruption-reason}'

# Layer 3: Karpenter controller logs filtered to provisioning and disruption kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter \ --since=3h | grep -E "provisioned|disrupted|consolidat|drift|interrupt|expir"

# Layer 4: EC2 instance history (for Spot interruptions, verify in CloudWatch) aws ec2 describe-spot-instance-requests \ --filters "Name=tag:karpenter.sh/nodepool,Values=<nodepool-name>" \ --query 'SpotInstanceRequests[*].{ID:InstanceId,State:State,Fault:Fault}' \ --region eu-north-1 ```

1. Map each event to a cause: