Skill v1.0.1
currentAutomated scan100/100+7 new
name: detecting-container-drift-at-runtime description: Detect unauthorized modifications to running containers by monitoring for binary execution drift, file system changes, and configuration deviations from the original container image. domain: cybersecurity subdomain: container-security tags:
- container-drift
- runtime-security
- immutable-containers
- falco
- kubernetes
- container-security
- drift-detection
- microsoft-defender
version: '1.0' author: mahipal license: Apache-2.0 nist_csf:
- PR.PS-01
- PR.IR-01
- ID.AM-08
- DE.CM-01
mitre_attack:
- T1610
- T1611
- T1609
- T1525
Detecting Container Drift at Runtime
Overview
Container drift occurs when running containers deviate from their original image state through unauthorized file modifications, unexpected binary execution, configuration changes, or package installations. Since containers should be treated as immutable infrastructure, any drift is a potential indicator of compromise. Detection techniques leverage the DIE (Detect, Isolate, Evict) model -- an immutable workload should not change during runtime, so any observed change is potentially evidence of malicious activity.
When to Use
- When investigating security incidents that require detecting container drift at runtime
- When building detection rules or threat hunting queries for this domain
- When SOC analysts need structured procedures for this analysis type
- When validating security monitoring coverage for related attack techniques
Prerequisites
- Kubernetes cluster v1.24+ with runtime security tooling
- Falco or Sysdig for runtime drift detection
- Container image registry with image manifests available
- Familiarity with Linux filesystem layers and OverlayFS
Core Concepts
Types of Container Drift
- Binary drift: Execution of binaries not present in the original image (downloaded malware, compiled tools)
- File drift: Creation, modification, or deletion of files in the container filesystem
- Configuration drift: Changes to environment variables, mounted secrets, or runtime parameters
- Package drift: Installation of new packages via apt, yum, pip, or npm at runtime
- Network drift: New listening ports or outbound connections not expected for the workload
Detection Methods
Image-Based Comparison: Compare the running container's filesystem against its source image to identify added, modified, or removed files.
Behavioral Monitoring: Use eBPF or kernel-level monitoring to detect process execution, file access, and network activity that deviates from expected behavior.
Digest Verification: Continuously verify that running container image digests match the approved deployment manifests.
Implementation with Falco
Detecting New Binary Execution
- rule: Drift Detected (Container Image Modified Binary)desc: Detect execution of a binary not present in the original container imagecondition: >spawned_process andcontainer andnot proc.pname in (container_entrypoint) andproc.is_exe_upper_layer = trueoutput: >Drift detected: new binary executed in container(user=%user.name command=%proc.cmdline container=%container.nameimage=%container.image.repository:%container.image.tagexe_path=%proc.exepath)priority: WARNINGtags: [container, drift]- rule: Container Shell Spawneddesc: Detect interactive shell in a container that should be immutablecondition: >spawned_process andcontainer andproc.name in (bash, sh, dash, zsh, csh, ksh) andnot proc.pname in (container_entrypoint)output: >Shell spawned in container (user=%user.name shell=%proc.namecontainer=%container.name image=%container.image.repository)priority: WARNINGtags: [container, drift, shell]
Detecting Package Manager Usage
- rule: Package Manager Execution in Containerdesc: Detect use of package managers indicating driftcondition: >spawned_process andcontainer andproc.name in (apt, apt-get, yum, dnf, apk, pip, pip3, npm, gem, cargo)output: >Package manager executed in container (user=%user.namecommand=%proc.cmdline container=%container.nameimage=%container.image.repository)priority: ERRORtags: [container, drift, package-manager]
Detecting File System Modifications
- rule: Container File System Writedesc: Detect writes to container upper layer filesystemcondition: >open_write andcontainer andfd.typechar = 'f' andnot fd.name startswith /tmp andnot fd.name startswith /var/log andnot fd.name startswith /procoutput: >File write in container (user=%user.name file=%fd.namecontainer=%container.name)priority: NOTICEtags: [container, drift, filesystem]
Implementation with Kubernetes Enforcement
Read-Only Root Filesystem
Prevent drift by making container filesystems immutable:
apiVersion: apps/v1kind: Deploymentmetadata:name: immutable-appspec:template:spec:containers:- name: appimage: app:v1.0@sha256:abc123...securityContext:readOnlyRootFilesystem: trueallowPrivilegeEscalation: falserunAsNonRoot: truevolumeMounts:- name: tmpmountPath: /tmp- name: cachemountPath: /var/cachevolumes:- name: tmpemptyDir:sizeLimit: 100Mi- name: cacheemptyDir:sizeLimit: 50Mi
Pod Security Standards Enforcement
apiVersion: v1kind: Namespacemetadata:name: productionlabels:pod-security.kubernetes.io/enforce: restrictedpod-security.kubernetes.io/audit: restrictedpod-security.kubernetes.io/warn: restricted
Image Digest Verification
Continuous Digest Monitoring
#!/bin/bash# Compare running container digests against approved manifestNAMESPACE="production"kubectl get pods -n "$NAMESPACE" -o json | jq -r '.items[] |.spec.containers[] |"\(.image) \(.imageID)"' | while read IMAGE IMAGE_ID; doAPPROVED_DIGEST=$(kubectl get deploy -n "$NAMESPACE" -o json | \jq -r ".items[].spec.template.spec.containers[] | select(.image==\"$IMAGE\") | .image")if [[ "$IMAGE" != *"@sha256:"* ]]; thenecho "[WARN] Container using mutable tag: $IMAGE"fidone
Microsoft Defender for Containers Integration
For Azure Kubernetes environments, Microsoft Defender provides built-in binary drift detection:
{"alertType": "K8S.NODE_ImageBinaryDrift","severity": "Medium","description": "Binary executed that was not part of the original container image","remediationSteps": ["Investigate the binary origin and purpose","Check if the container was compromised","Rebuild the container from a clean image","Enable readOnlyRootFilesystem"]}
Drift Response Playbook
- Detect: Alert fires on drift event (Falco, Defender, Sysdig)
- Validate: Confirm the drift is not from an approved process (init containers, config reloads)
- Isolate: Apply a deny-all NetworkPolicy to the affected pod
- Investigate: Capture container filesystem diff and process list
- Evict: Delete the drifted pod (ReplicaSet will recreate from clean image)
- Remediate: Fix the root cause (patch vulnerability, update image, tighten RBAC)