zscaler/zscaler-mcp-server/investigate-alerts

ZDX: Investigate Alerts

Keywords

alerts, incidents, zdx alerts, active alerts, historical alerts, affected devices, alert investigation, ongoing alert, ISP issue, blackout, degradation, alert scope, impacted users, alert history

Overview

Investigate active and historical ZDX alerts to determine their scope, affected users, root cause, and whether they represent isolated incidents or broader patterns. This skill provides a systematic approach to alert triage and investigation.

Use this skill when: An administrator wants to review current or past alerts, understand alert impact, investigate ISP or network incidents, or determine if a reported issue correlates with a known alert.

ZDX Copilot alignment: This skill covers the Troubleshooting category -- "Show me the number of ongoing alerts", "Show me the Incidents in the last 48 hours", "What can I do when I see a blackout incident with an ISP?"

Data Presentation Requirements

All tables are rendered by the HTML template, not by hand. Do not author <table>, <thead>, <tbody>, <tr>, <th>, or <td> markup in your reply or in the report. You produce a JSON payload (see Data Payload Contract) and the template at ./templates/report.html.template turns it into the styled, sortable, exportable tables shown in ./example/report.example.html.

After each table, provide:

1. Detailed analysis explaining the alert severity, scope, and correlation between alerts
2. Root cause identification based on metric correlation and affected scope
3. Next steps / resolution with specific remediation actions, escalation paths, and monitoring recommendations

Use color-coded rows by alert priority:

• Red: High priority alerts (many affected devices, critical applications, long duration)
• Yellow: Medium priority alerts (moderate impact, localized scope)
• Green: Low priority or resolved alerts

⚠ HTML OUTPUT — READ THIS BEFORE PRODUCING ANY HTML

There is exactly one acceptable way to produce the HTML output:

1. Read the template from disk — do NOT inline a copy in your response. The template lives next to this SKILL.md inside the skill's package, at:

``text ./templates/report.html.template ``

The ./ prefix is intentional: this path is relative to the skill folder (the directory containing this SKILL.md), never an absolute path. Most agents that load skills from an uploaded .zip extract the package into a working directory and expose its contents via that relative path — read the file by joining the skill's own root directory with ./templates/report.html.template. Do not rewrite this to an absolute path that points at the author's machine.

1. Build a single JSON object (__ZDX_DATA__ payload) shaped exactly as documented in the Data Payload Contract section below. Aggregate the responses from the ZDX MCP tool calls (Steps 1–6 of the Workflow) into that object.

1. Replace the literal token __ZDX_DATA__ (which appears once, inside <script type="application/json" id="zdx-data">__ZDX_DATA__</script>) with the JSON object. Do not edit any other part of the template.

1. Write the result to disk as alert_investigation_report_<YYYYMMDD-HHMMSS>.html next to the .docx, and give the user a computer:// link to it.

This template already provides: Zscaler header with logo · sticky top bar · scope summary bar · KPI cards with severity-coded top borders · per-table search + filter chips · sortable color-coded tables · per-table CSV export · light/dark theme toggle · top-right language dropdown (EN / ES / PT / FR / JA) · printable PDF view · localStorage prefs · Analysis / Root Cause / Remediation block.

If you find yourself writing `<html>`, `<style>`, or `<table>` in a code-block destined for the user, stop. Read the template instead.

A populated reference rendering ships with this skill at ./example/report.example.html (relative to the skill folder). Open it in a browser to preview the exact layout and depth expected.

Data Payload Contract

The full __ZDX_DATA__ payload is one JSON object. Every field below is required unless marked optional. This skill emits two tables: the active alert summary and the per-alert metric correlation.

{
  "generated_at": "<ISO 8601 timestamp>",
  "scope_en": "Free-form description in English",
  "scope_es": "...in Spanish (optional, falls back to scope_en)",
  "scope_pt": "...in Portuguese (optional)",
  "scope_fr": "...in French (optional)",
  "scope_ja": "...in Japanese (optional)",
  "kpis": {
    "total": "<int>",
    "high": "<int>",
    "medium": "<int>",
    "low": "<int>",
    "mostAffectedApp": "<app name or '—'>",
    "impactedDevices": "<int>"
  },
  "tables": {
    "alerts": [
      {
        "severity": "critical | warning | good",
        "priority": "High | Medium | Low",
        "name": "<alert title>",
        "application": "<app name>",
        "duration": "<e.g. '6 hours'>",
        "affectedDevices": "<int>",
        "locations": "<comma-separated location names>",
        "bottleneck": "<e.g. 'PFT: 12.4s'>",
        "status": "Active | Resolved"
      }
    ],
    "correlation": [
      {
        "severity": "critical | warning | good",
        "alert": "<alert title>",
        "pft": "<value or '—'>",
        "dns": "<value or '—'>",
        "availability": "<value or '—'>",
        "rootCauseIndicator": "Server-side / CDN | Local DNS resolver | Application server instability | ISP path | …"
      }
    ]
  },
  "analysis": {
    "summary": "...",
    "rootCause": "...",
    "remediation": [
      { "priority": "Immediate | Investigate | Monitor | Communicate", "action": "..." }
    ]
  }
}

Map each row's severity from priority: High → critical, Medium → warning, Low → good.

Output Artifacts — MANDATORY

You MUST generate BOTH files below. Both are REQUIRED output for every alert investigation.

1. Word Document (.docx) — REQUIRED

Write a Word document to disk named alert_investigation_report_<YYYYMMDD-HHMMSS>.docx containing:

• Executive summary with active alert count, severity breakdown, and scope
• Active alerts table (priority, alert name, application, duration, affected devices, locations, bottleneck)
• Metric correlation per alert (PFT, DNS, availability, root cause indicator)
• Historical pattern analysis (recurring alerts, frequency, time patterns)
• Per-alert root cause analysis with supporting evidence
• Prioritized remediation actions and escalation paths

2. Interactive HTML Web Page (.html) — REQUIRED

Generated by the template-substitution flow described in the HTML OUTPUT section above. Filename: alert_investigation_report_<YYYYMMDD-HHMMSS>.html. Do not hand-author HTML or CSS — the template ships everything the report needs.

Workflow

Step 1: List Active Alerts

Retrieve all currently active alerts.

zdx_list_alerts()

For a broader time window (e.g., last 48 hours):

zdx_list_alerts(since=48)

Filter by location or department for scoped investigation:

zdx_list_alerts(
  location_id=["<location_id>"],
  since=24
)

zdx_list_alerts(
  department_id=["<department_id>"],
  since=24
)

Triage by severity and scope:

• How many alerts are active?
• Which applications are affected?
• How long have they been active?
• Are multiple alerts pointing to the same root cause?

Step 2: Get Alert Details

For each alert that warrants investigation:

zdx_get_alert(alert_id="<alert_id>")

Key information to extract:

• Alert type (performance degradation, availability, etc.)
• Triggered application and threshold
• Impacted departments and locations
• Geolocation data
• Alert trigger conditions
• Start time and duration

Step 3: Determine Impact Scope

List all devices affected by the alert:

zdx_list_alert_affected_devices(alert_id="<alert_id>")

Filter by location to understand geographic scope:

zdx_list_alert_affected_devices(
  alert_id="<alert_id>",
  location_id=["<location_id>"]
)

Filter by department for business impact:

zdx_list_alert_affected_devices(
  alert_id="<alert_id>",
  department_id=["<department_id>"]
)

Scope determination:

• Single user: Likely a device or local network issue
• Single location: Office-specific issue (local ISP, WiFi, LAN)
• Multiple locations, same geo: Regional ISP or cloud region issue
• Global: Application server issue or major ISP backbone problem

Step 4: Correlate with Application Metrics

Cross-reference the alert with application performance data to confirm the bottleneck.

zdx_get_application_score_trend(
  app_id="<affected_app_id>",
  since=24
)

zdx_get_application_metric(
  app_id="<affected_app_id>",
  metric_name="pft"
)

zdx_get_application_metric(
  app_id="<affected_app_id>",
  metric_name="dns"
)

zdx_get_application_metric(
  app_id="<affected_app_id>",
  metric_name="availability"
)

Correlation mapping:

• Alert + high PFT + normal DNS → Application or CDN issue
• Alert + high DNS → DNS infrastructure problem
• Alert + low availability → Network or application unreachable
• Alert + all metrics degraded → Major infrastructure event

Step 5: Check Historical Patterns

Look for recurring patterns in past alerts.

zdx_list_historical_alerts(since=336)

Filter by location or department:

zdx_list_historical_alerts(
  location_id=["<location_id>"],
  since=168
)

Pattern analysis:

• Same alert recurring daily at specific times? → Scheduled load or maintenance window
• Same alert recurring weekly? → Periodic resource constraints
• Same location repeatedly affected? → Infrastructure issue at that site
• Same application repeatedly affected? → Application stability concerns

Step 6: Deep Trace for Network Path Analysis

If the alert suggests a network issue, check if deep traces exist for affected devices.

zdx_list_devices(location_id=["<affected_location_id>"])

For a specific affected device:

zdx_list_device_deep_traces(device_id="<device_id>")

If a trace exists, analyze the full diagnostics data:

zdx_get_device_deep_trace(device_id="<device_id>", trace_id="<trace_id>")
zdx_get_deeptrace_webprobe_metrics(device_id="<device_id>", trace_id="<trace_id>")
zdx_get_deeptrace_cloudpath(device_id="<device_id>", trace_id="<trace_id>")
zdx_get_deeptrace_cloudpath_metrics(device_id="<device_id>", trace_id="<trace_id>")
zdx_get_deeptrace_health_metrics(device_id="<device_id>", trace_id="<trace_id>")
zdx_list_deeptrace_top_processes(device_id="<device_id>", trace_id="<trace_id>")
zdx_get_deeptrace_events(device_id="<device_id>", trace_id="<trace_id>")

If no trace exists and the alert is recurring, discover probe IDs and start a proactive deep trace to capture evidence during the next occurrence (requires write tools):

zdx_get_web_probes(device_id="<device_id>", app_id="<affected_app_id>")
zdx_list_cloudpath_probes(device_id="<device_id>", app_id="<affected_app_id>")
zdx_start_deeptrace(device_id="<device_id>", session_name="Alert-Investigation-<date>", app_id=<app_id>, web_probe_id=<id>, cloudpath_probe_id=<id>, session_length_minutes=30, probe_device=True)

Deep traces reveal hop-by-hop latency and packet loss, pinpointing the exact network segment causing the issue. Web probe metrics isolate DNS vs TCP vs SSL bottlenecks, and event timelines correlate configuration changes with metric degradation.

For comprehensive deep trace analysis, see the Diagnose Deep Trace skill.

Step 7: Present Alert Investigation Report