<< All versions

Skill v1.0.0

currentLLM-judged scan90/100
konflux-ci/integration-service/debugging-running-instance
──Details
PublishedOctober 24, 2026 at 08:08 AM
Content Hashsha256:d2d1590204860a65...
Git SHAbc403ee70079
──Files
Files (1 file, 4.3 KB)
SKILL.md4.3 KBactive
SKILL.md · 88 lines · 4.3 KB

version: "1.0.0" name: debugging-running-instance description: Use when debugging integration-service running on a cluster. Covers pod logs, health probes, metrics, webhooks, network policies, environment variables, snapshot GC, and common failure modes.

Debugging a Running Instance

Overview

The integration-service runs as a Deployment in the integration-service namespace. It exposes health probes, Prometheus metrics, and optional webhooks. A separate CronJob handles snapshot garbage collection.

When to Use

  • Controller not reconciling resources as expected
  • Webhooks rejecting or silently ignoring mutations/validations
  • Metrics not being scraped
  • Status reports not appearing on PRs
  • Snapshot GC not cleaning up old snapshots

Quick Reference

WhatHow
Controller logskubectl logs -n integration-service deploy/controller-manager -f
Health checkkubectl get -n integration-service deploy/controller-manager -o jsonpath='{.status.conditions}'
Readiness probeHTTP GET /readyz on port 8081
Liveness probeHTTP GET /healthz on port 8081
MetricsPort 8080 (HTTPS, self-signed cert), scraped at /metrics
Webhook port9443 (when enabled)
Leader electionLease-based, 15s lease duration, 10s renew deadline, 2s retry period
Pod labelcontrol-plane: controller-manager

Environment Variables

Controller

VariableDefaultEffect if wrong
CONSOLE_URLnonePR status comments show CONSOLE_URL_NOT_AVAILABLE
CONSOLE_URL_TASKLOGnoneTask log links show CONSOLE_URL_TASKLOG_NOT_AVAILABLE
CONSOLE_NAMEnoneConsole display name missing from PR comments
PIPELINE_TIMEOUTnoneInvalid duration is logged as error and skipped
TASKS_TIMEOUTnoneInvalid duration is logged as error and skipped
FINALLY_TIMEOUTnoneInvalid duration is logged as error and skipped
INTEGRATION_NSintegration-serviceWrong namespace for PAC secret lookup
PAC_SECRETpipelines-as-code-secretCan't authenticate to git providers

Network Policies

Namespaces must be explicitly labeled for traffic to reach the controller:

TrafficRequired label
Metrics scrapingmetrics: enabled on source namespace
Webhook callswebhook: enabled on source namespace

Missing labels = silently blocked traffic. Check with: kubectl get ns <name> --show-labels

Webhook Behavior

WebhookTypeFailure PolicyEffect
IntegrationTestScenarioMutatingIgnoreDefaults applied silently on failure
IntegrationTestScenarioValidatingFailRejects invalid ITS (bad names, conflicting resolver params)
ComponentGroupValidatingFailBlocks invalid ComponentGroups
NudgeConfigValidatingFailRejects invalid NudgeConfig (cycles, missing Components)
SnapshotMutatingIgnoreDefaults may be skipped if the webhook is unavailable
SnapshotValidatingFailRejects invalid Snapshots and blocks Snapshot operations if the webhook is unavailable

Debugging Checklist

  1. Controller not starting? Check env var values and RBAC — timeout parsing errors are logged, not fatal
  2. Not reconciling? Check leader election lease, RBAC, and that CRDs are installed
  3. PR status missing? Check CONSOLE_URL env var and PAC secret availability
  4. Webhooks not firing? Check namespace labels (webhook: enabled) and cert-manager status
  5. Metrics missing? Check namespace labels (metrics: enabled) and ServiceMonitor exists
  6. Snapshots piling up? Check snapshotgc CronJob logs — runs every 6h with separate service account

Common Mistakes

ProblemFix
CONSOLE_URL_NOT_AVAILABLE in PR commentsSet CONSOLE_URL env var with {{NAMESPACE}} and {{PIPELINE_RUN_NAME}} placeholders
Controller crash loopCheck env var values — timeout vars must be valid Go durations (e.g., 2h, 30m)
Snapshot operations fail while the webhook is unavailableSnapshot validation uses failurePolicy: Fail — check the webhook pod, service, and certificate health
Metrics endpoint unreachableVerify namespace label metrics: enabled and that ServiceMonitor target matches
All versions