gke-labs/kube-agents/gke-reliability

GKE Reliability Skill

This skill provides workflows for configuring your GKE cluster and workloads for high availability and reliability.

You are read-only: each change below is a proposal you report with your findings, never a command you run.

Workflows

1. Verify Cluster High Availability

Check if the cluster is regional or has multi-zonal node pools.

Command:

gcloud container clusters describe <cluster-name> --region <region> --format="json(location, locations)"

If location is a region (e.g., us-central1), the control plane is regional. If locations has multiple entries, nodes are spread across multiple zones.

2. Configure Pod Disruption Budgets (PDB)

PDBs ensure that a minimum number of pods are available during voluntary disruptions (like node upgrades).

Check existing PDBs:

kubectl get pdb -n <namespace>

Assess drain exposure (a missing PDB is not, on its own, a workload that "won't survive a drain"):

kubectl get pods -n <namespace> -l <selector> -o wide

• State the observed placement: replica count and pods per node (3 pods on 3 nodes).
• Say a single drain evicts only the pods on the drained node; replicas elsewhere keep running.
• Frame a missing PDB as unprotected during voluntary disruptions such as node-pool upgrades and autoscaler scale-downs: back-to-back drains can evict the next replica before its replacement is Ready.
• Flag co-location separately: with no pod anti-affinity or topology spread, a reschedule can put replicas on one node, and one drain then takes them all.

Example Manifest:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  maxUnavailable: 1 # never minAvailable: it can block every drain
  selector:
    matchLabels:
      app: my-app

3. Configure Health Probes

Ensure all production containers have Liveness, Readiness, and optionally Startup probes.

• Readiness Probe: Determines when a container is ready to start accepting traffic.
• Liveness Probe: Determines when to restart a container.
• Startup Probe: Disables liveness and readiness checks until the app has started up.

Check workload probes:

kubectl get deployment <app-name> -n <namespace> -o yaml | grep -E "livenessProbe|readinessProbe|startupProbe"

4. Graceful Shutdown

Ensure applications handle SIGTERM signals gracefully and have an appropriate terminationGracePeriodSeconds set (default is 30s).

5. Topology Spread Constraints

Ensure pods are spread across zones or nodes to avoid correlated failures.

Example Manifest excerpt:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule # or ScheduleAnyway
      labelSelector:
        matchLabels:
          app: my-app

6. Maintenance Windows and Exclusions

Configure when GKE can perform automated upgrades to avoid peak hours.

Maintenance window: propose a recurring window with a start time, an end time and a FREQ=DAILY recurrence (Config Connector ContainerCluster field spec.maintenancePolicy.recurringWindow).

Best Practices

1. Regional Clusters: Always use regional clusters for production workloads to survive zone failures.
2. Probes for All Containers: Every container in a production pod should have at least a readiness probe.
3. PDBs for Critical Apps: Use PDBs to prevent downtime during automated node upgrades.
4. Zone Spreading: Always use topologySpreadConstraints to ensure pods are distributed across zones, even in regional clusters.
5. Schedule Maintenance: Set maintenance windows to ensure upgrades happen during low-traffic periods.