kyma-project/cli/kyma-troubleshoot

Skill: kyma-troubleshoot

Diagnose problems on a Kyma-enabled Kubernetes cluster with the open-source Kyma CLI (v3) and kubectl, following a read-only-first ladder that narrows from cluster health, to the affected component, to individual resources — before proposing any fix.

Kyma CLI v3 is a Go-based, Apache-2.0 tool from the open-source Kyma project (github.com/kyma-project/cli). It is a full rewrite of v2 — do not assume v2 syntax.

Scope and guardrails

This skill covers only the open-source Kyma project. When using it:

• ✅ Use open-source Kyma (kyma-project/*), open-source modules, the Kyma CLI,

and standard kubectl.

• ❌ Do not encode SAP-managed (SKR) specifics, BTP cockpit flows, internal

endpoints or URLs, SKR configuration, or any SAP-internal knowledge.

• ⚠️ Diagnosis is read-only. Never delete or unmanage a module as a

troubleshooting step without showing the user the exact impact and getting explicit confirmation (see Safety).

Prerequisites

• The kyma CLI installed and on PATH (v3). Confirm with kyma version.
• kubectl installed and its current context pointing at the affected cluster.

Verify your kubectl context before starting — all diagnose commands fail silently with a connection error if no context is set:

kubectl config current-context     # confirm which cluster you are targeting
kubectl cluster-info               # verify connectivity
# If wrong or missing, switch context:
kubectl config get-contexts
kubectl config use-context <name>
# Or pass it per-command: kyma alpha diagnose cluster --kubeconfig <path>

Before running a command whose exact flags you are unsure of, inspect help output rather than guessing — the installed version is the source of truth:

kyma alpha diagnose cluster --help
kyma help <command>

Instructions

Work top-down. Do not skip to a fix before you have observed the actual state.

1. Establish overall state (read-only)

Run the cluster diagnose command first — it is the single most informative first step:

kyma alpha diagnose cluster              # full cluster snapshot (YAML output)
kyma alpha diagnose cluster -f json      # same, JSON format — pipe to jq for filtering
kyma alpha diagnose cluster --verbose    # include collection-error details (cluster subcommand only)
# Add --skip-extensions to suppress the harmless extensions warning that clutters output

kyma alpha diagnose cluster output has four sections:

`metadata` is only populated on Gardener/SAP-managed clusters. On plain clusters

(k3d, kind, EKS, GKE, etc.) all metadata fields will be empty — this is expected, not an error.

Then get the module list to see every module and its state:

kyma module list                 # which modules exist and their Ready/Processing/Error/etc. state
kyma module list --show-errors   # same, with inline error details per module (KLM-managed clusters only)

If `kyma module list` returns "not managed by KLM" (plain cluster without Lifecycle Manager),

fall back to kubectl to discover what's installed:

```bash

kubectl api-resources --verbs=list | grep kyma-project.io # which module CRDs are present

kubectl get pods -n kyma-system | grep operator # which operator pods are running

```

`kymaModulesErrors: []` in diagnose cluster output is ambiguous — an empty list means

both "all installed modules are healthy" AND "no modules are installed at all".

Always cross-check with kyma module list (or the kubectl fallback above) to distinguish the two.

`kyma alpha diagnose cluster` is always the first command. It surfaces cluster-wide

warning events, node pressure, and all module errors in a single call — before you

narrow by symptom.

2. Narrow by symptom

Pick the diagnostic that matches what the user reports:

2a. Networking, gateway, mTLS, sidecar issues

# First-pass mesh analysis — catches misconfigured AuthorizationPolicy,
# PeerAuthentication, DestinationRule, Gateway, and injection issues:
istioctl analyze [-n <namespace> | --all-namespaces]
 
# Then run the Kyma-level Istio diagnostic:
kyma alpha diagnose istio [-A|--all-namespaces] [--level warning|error|info] [-n <namespace>] [--timeout 30s]

Confirm Istio is installed first — kyma alpha diagnose istio exits silently with

zero bytes when the Istio module is absent, which is indistinguishable from "no issues found".

Run kubectl get istio -n kyma-system before running the diagnose command.

- "no resources found" → CRD is present, but no Istio CR was created (partial install); check operator pod logs

- "the server doesn't have a resource type 'istio'" → CRD was never applied; enable the Istio module first

istioctl analyze is your quickest signal — it names the problematic resource and namespace and calls out common IST codes (e.g. IST0133 for an overly broad AuthorizationPolicy).

`istioctl` not on PATH? Skip istioctl analyze and go directly to

kyma alpha diagnose istio -n <namespace> --level warning as a standalone

alternative — it covers the same injection and policy checks without requiring

istioctl to be installed.

2b. App or module errors

`--strict` silently drops ALL non-JSON lines, including plain-text error messages

from containers that don't emit structured logs. If you see fewer logs than expected,

drop --strict to see all output.

# Aggregates JSON-formatted error-level container logs.
# --since and --lines are independent; combine to apply both limits:
kyma alpha diagnose logs --module <module> --since 1h [--timeout 30s] [--strict]
kyma alpha diagnose logs --module <module> --lines 200 [--timeout 30s] [--strict]
# --lines defaults to 200 when omitted

kyma alpha diagnose logs accepts --module repeatedly to aggregate across several modules at once. Append --format json --output <file> to save results. Add --strict to suppress non-JSON log lines and see only structured error entries.

`--since` and `--lines` are independent flags — use either or both.

--since bounds by time window; --lines caps line count; combine them to apply both limits simultaneously.

Empty output behaviour differs by subcommand:

- kyma alpha diagnose logs returns logs: {} when a module is absent — non-empty output, but empty map.

- kyma alpha diagnose istio exits silently (zero bytes) when Istio is not installed.

In both cases, check kyma module list to confirm the module is present before concluding there is no problem.

3. Drill down with kubectl

Once you know the affected component, inspect its resources directly.

First, discover the module's CR kind and operator pod:

# Find the CRD name for a module (e.g. "istio", "serverless")
kubectl api-resources --verbs=list | grep kyma-project.io
 
# Find the operator pod for a module
kubectl get pods -n kyma-system | grep <module>
# or by label:
kubectl get pods -n kyma-system -l app.kubernetes.io/name=<module>-operator

Then inspect:

kubectl get <module-cr> -n kyma-system -o yaml        # inspect CR .status conditions
kubectl describe <resource> -n <namespace>            # events + conditions
kubectl get events -n <namespace> --sort-by=.lastTimestamp
kubectl logs <operator-pod> -n kyma-system            # module operator logs

Focus on the CR's .status.conditions — Kyma modules report their health there. Module CRs are in the kyma-system namespace. Common module CR names:

Verify the CRD is installed first — if kubectl get <module-cr> returns

"no resource type", the module's CRD was never applied (e.g. install failed mid-way).

Run kubectl api-resources --verbs=list | grep kyma-project.io to see which CRDs exist

before assuming a CR is simply missing.

To extract just the conditions without the full YAML:

kubectl get <module-cr> -n kyma-system -o jsonpath='{.status.conditions[*]}' | jq .
# Example for serverless:
kubectl get serverless -n kyma-system -o jsonpath='{.status.conditions[*]}' | jq .

For Istio/service-mesh issues, also run:

# Sidecar version overview — shows proxy version vs control plane version:
istioctl proxy-status
 
# Check istiod and api-gateway logs for post-change or post-upgrade errors:
kubectl logs -n istio-system deployments/istiod
kubectl logs -n kyma-system deployments/api-gateway

For network connectivity issues:

# Check for NetworkPolicies that may be blocking traffic:
kubectl get networkpolicies -A
 
# List Istio resources that affect connectivity:
kubectl get authorizationpolicies,peerauthentications,destinationrules,gateways -A

4. Match against common failure patterns

General module failures

Istio / networking failures