name: incident-response description: Handle security incidents with IR playbooks and procedures. category: security risk: critical source: https://github.com/BagelHole/DevOps-Security-Agent-Skills source_repo: BagelHole/DevOps-Security-Agent-Skills source_type: community date_added: '2026-09-20' license: MIT license_source: https://github.com/BagelHole/DevOps-Security-Agent-Skills/blob/main/LICENSE compatibility: Requires the relevant security tooling (scanners, vault CLIs) and an authorized scope for any active assessment. Docs-only; helper scripts and templates not bundled. metadata: author: devops-skills version: '1.0'

Incident Response

Handle security incidents effectively with structured response procedures.

Prerequisites

• IR team roster with on-call rotation and escalation paths
• Secure communication channel (separate from production systems)
• Forensic workstation with analysis tools installed
• Evidence storage with chain-of-custody controls
• Legal counsel contact information
• Pre-authorized incident response actions documented

Incident Response Phases

phases:
  1_preparation:
    - IR team roster and 24/7 contact info
    - Tools and privileged access ready
    - Playbooks documented and tested
    - Evidence collection kit prepared
    - Communication templates drafted
 
  2_detection:
    - Alert triage and validation
    - Initial assessment and scoping
    - Severity classification
    - Incident ticket creation
 
  3_containment:
    - Short-term containment (stop bleeding)
    - Evidence preservation (before changes)
    - System isolation (network/host level)
    - Credential rotation if needed
 
  4_eradication:
    - Root cause analysis
    - Remove threat actor access
    - Patch exploited vulnerabilities
    - Clean compromised systems
 
  5_recovery:
    - System restoration from clean backups
    - Enhanced monitoring deployment
    - Phased return to production
    - Business continuity verification
 
  6_lessons_learned:
    - Post-incident review (within 72 hours)
    - Timeline reconstruction
    - Documentation update
    - Process and detection improvements

Severity Classification

The targets below (and the 72-hour retrospective target) are examples to agree with the organization, not notification deadlines or guarantees. Classify phishing by actual access/data/business impact, not automatically Medium.

Evidence collection and containment

Live collection changes a host and may run attacker-controlled binaries. Use a trusted forensic workstation/toolkit, approved scope and a private encrypted evidence volume; record collector identity, UTC time/clock offset, source, tool/version, commands, errors and custody transfers. Hashes detect later byte changes but do not prove an honest source, completeness or chain of custody. Active containment may take priority over preservation to prevent ongoing harm; record that decision.

Bounded Linux collection example

On an approved Linux test host with GNU tools, a minimal metadata collection is:

set -euo pipefail
umask 077
# EVIDENCE_PARENT must already be an approved private evidence volume.
: "${EVIDENCE_PARENT:?Set approved evidence volume}"
EVIDENCE_DIR=$(mktemp -d "$EVIDENCE_PARENT/collection.XXXXXXXX")
date -u +%Y-%m-%dT%H:%M:%SZ > "$EVIDENCE_DIR/collection-time.txt"
uname -a > "$EVIDENCE_DIR/uname.txt"
ps -eo pid,ppid,user,comm > "$EVIDENCE_DIR/process-names.txt"
# Write the final log before hashing it; never append to hashed evidence afterward.
for name in collection-time.txt uname.txt process-names.txt; do
  (cd "$EVIDENCE_DIR" && sha256sum "$name")
done > "$EVIDENCE_DIR/checksums.sha256"
(cd "$EVIDENCE_DIR" && sha256sum -c checksums.sha256)

This omits memory, full command lines, credential databases, logs, containers and remote clusters deliberately: each requires explicit data scope, retention/access controls and an independently tracked outcome. A missing tool/error is partial collection, not a clean host. Avoid unlimited whole-filesystem walks or collecting /etc/shadow by default. Bound disk/time and do not hide permission failures.

Memory acquisition

Use a reviewed kernel-compatible acquisition tool only under the IR plan: loading LiME modifies kernel state and may crash a system. /proc/kcore is not always available and copying its enormous virtual address space is not a reliable generic memory dump. Record hashes, kernel/tool version, errors and approved size/time bounds; no successful memory capture is implied by a printed message.

AWS evidence and isolation runbook

1. Confirm account, region, instance and every attached volume/ENI from a read-only

inventory. Preserve that inventory and request IDs in private evidence storage.

1. Snapshot the approved volume set consistently (multi-volume/application consistency

needs planning), record IDs and wait for completed/error states. Creation is a billable write; an accepted request is not a completed forensic image.

1. Export the relevant log data and query/time coverage. describe-flow-logs exports

configuration, not packets or historical flow records. CloudTrail event history lookup is not complete data-event or organization-wide incident coverage.

1. Separately approve containment and preserve original SG/ENI configurations plus

rollback/recovery path. A newly created SG has default outbound allow rules: attaching it alone does not isolate egress. Security groups are stateful and existing connections may persist; use the organization’s tested quarantine controls across every ENI/route/connection path, preserving authorized evidence access.

1. Verify observed isolation from controlled test points and monitor before declaring

containment. Store evidence in an existing restricted encrypted retention-controlled destination; never derive arbitrary public bucket names or reuse predictable /tmp evidence files.

Forensics Commands Reference

# --- Disk forensics ---
# Create forensic image of a disk
dd if=/dev/sda of=/evidence/disk.img bs=4M status=progress
sha256sum /evidence/disk.img > /evidence/disk.img.sha256
 
# Mount forensic image read-only
# Mount a working copy with filesystem-specific no-journal-replay options
# and a write blocker where applicable; ro alone may permit journal recovery.
# Do not mount an untrusted image on the production host.
 
# Find recently modified files
find /mnt/forensic -type f -mtime -3 -ls | sort -k11
 
# Find files by owner
find /mnt/forensic -user www-data -type f -newer /tmp/reference-time -ls
 
# --- Log analysis ---
# Search auth logs for brute force
grep "Failed password" /var/log/auth.log | awk '{print $11}' | sort | uniq -c | sort -rn | head -20
 
# Search for privilege escalation
grep -E "(sudo|su\[)" /var/log/auth.log | grep -v "session opened"
 
# Search web logs for attack patterns
grep -iE "(union.*select|<script|\.\.\/|%00)" /var/log/nginx/access.log
 
# Timeline analysis with find
find / -newermt "2025-01-15 00:00" ! -newermt "2025-01-16 00:00" -ls 2>/dev/null | sort -k9
 
# --- Network forensics ---
# Capture network traffic
tcpdump -i eth0 -w /evidence/capture.pcap -c 100000
 
# Analyze pcap for suspicious connections
tcpdump -r /evidence/capture.pcap -nn 'dst port 4444 or dst port 8888 or dst port 1337'
 
# Check for DNS tunneling
tcpdump -r /evidence/capture.pcap -nn 'udp port 53' | awk '{print $NF}' | sort | uniq -c | sort -rn | head -20
 
# --- Malware analysis ---
# Check file for known malware hashes
sha256sum suspicious_file
# Query only a hash if disclosure is approved; never upload confidential samples automatically.
 
# Strings analysis
strings suspicious_file | grep -iE "(http|ftp|ssh|password|key|token)"
 
# Check for packed/obfuscated binaries
file suspicious_file
readelf -h suspicious_file 2>/dev/null

Communication Templates

Initial Notification (Internal)

## Security Incident Notification
 
**Incident ID:** INC-YYYY-NNNN
**Severity:** [Critical/High/Medium/Low]
**Status:** Active - Investigating
**Time Detected:** YYYY-MM-DD HH:MM UTC
**Reported By:** [Name/System]
 
### Summary
[1-2 sentence description of what was detected]
 
### Impact Assessment
-**Systems affected:** [list]
-**Data at risk:** [type and scope]
-**Users impacted:** [count/scope]
-**Business impact:** [description]
 
### Current Actions
-[ ] Evidence preservation in progress
-[ ] Containment measures being applied
-[ ] IR team assembled
 
### Next Update
Expected at: YYYY-MM-DD HH:MM UTC
 
### Incident Commander
[Name] - [Contact info]

Stakeholder Update

## Incident Update - INC-YYYY-NNNN
 
**Update #:** N
**Time:** YYYY-MM-DD HH:MM UTC
**Severity:** [unchanged/upgraded/downgraded]
**Status:** [Investigating/Contained/Eradicating/Recovering/Resolved]
 
### Progress Since Last Update
-[Bullet points of actions taken]
 
### Current Understanding
-**Root cause:** [Known/Under investigation]
-**Scope:** [Expanded/Unchanged/Reduced]
-**Threat actor:** [If applicable]
 
### Active Containment Measures
-[List of measures in place]
 
### Next Steps
-[Planned actions with ETA]
 
### Decisions Needed
-[If any decisions required from leadership]

External Breach Notification (if required)

The external notice is a draft in the reference; legal/communications owners must verify facts, jurisdiction-specific deadlines and recipients before sending.

Contents

• Notice of Data Security Incident
• IR Playbook: Compromised Credentials
• Troubleshooting
• Best Practices
• Related Skills

When to Use This Skill

Use this skill when:

• Responding to an active security incident (breach, malware, unauthorized access)
• Building incident response playbooks and runbooks
• Conducting IR tabletop exercises and drills
• Setting up evidence collection and forensic capabilities
• Establishing communication protocols for security events
• Performing post-incident reviews and process improvements

Limitations

• Apply guidance only within authorized scope; test destructive steps in non-production first.
• Docs-only import: upstream scripts and templates not bundled.

Example

# Read-only first: inventory before any active step.
command -v sha256sum # availability only; not evidence of collection success

Adapted from BagelHole/DevOps-Security-Agent-Skills (MIT); frontmatter, When to Use/Limitations, and safety boundaries added for upstream compliance. Docs-only import: helper scripts and templates not bundled.