zscaler/zscaler-mcp-server/investigate-multi-app-outage

ZDX: Investigate Multi-Application Outage at a Location

Keywords

multi-app outage, location outage, office outage, common network path, shared bottleneck, cross-app correlation, hop analysis, ISP issue, cloud path, multiple applications affected, regional outage, site-wide issue, salesforce slow, servicenow slow, cross-tenant impact, shared infrastructure

Overview

Diagnose a scenario where users at one specific office or location are unable to reach multiple SaaS applications simultaneously. The skill correlates active ZDX alerts, scopes impact by location, intersects the affected-device lists across the impacted apps, and then compares the per-application network path (cloud path) on a common affected device to identify the shared upstream segment that is the actual root cause.

Use this skill when: An admin receives a ZDX alert (or user complaint) that names a single office and two or more applications — e.g., "Columbus users can't reach Salesforce and ServiceNow" — and the desired output is a list of affected devices plus an evidence-backed identification of the shared network path issue.

ZDX Copilot alignment: Troubleshooting + Optimization — converts a coarse "users at office X can't reach apps Y/Z" complaint into a hop-level root cause and a prioritized remediation list.

⚠ HTML OUTPUT — READ THIS BEFORE PRODUCING ANY HTML

You MUST NOT hand-write or copy/paste an HTML page for this skill. There is exactly one acceptable way to produce the HTML output:

1. Read this file from disk — do NOT inline a copy in your response. The template lives next to this SKILL.md inside the skill's package, at:

``text ./templates/report.html.template ``

The ./ prefix is intentional: this path is relative to the skill folder (the directory containing this SKILL.md), never an absolute path. Most agents that load skills from an uploaded .zip extract the package into a working directory and expose its contents via that relative path — read the file by joining the skill's own root directory with ./templates/report.html.template. Do not rewrite this to an absolute path that points at the author's machine.

1. Build a single JSON object (__ZDX_DATA__ payload) shaped exactly as documented in the Data Payload Contract section below. Aggregate the responses from the ZDX MCP tool calls (Steps 1–7 of the Workflow) into that object.

1. Replace the literal token __ZDX_DATA__ (which appears once, inside <script type="application/json" id="zdx-data">__ZDX_DATA__</script>) with the JSON object. Do not edit any other part of the template.

1. Write the result to disk as multi_app_outage_<location>_<YYYYMMDD-HHMMSS>.html next to the .docx, and give the user a computer:// link to it.

This template already provides: Zscaler header with logo · sticky top bar · ACTIVE INCIDENT pill · scope summary bar · color-coded incident banner · 6 KPI cards with severity-coded top borders and subtitles · per-table search + filter chips · sortable color-coded tables · per-table CSV export · light/dark theme toggle · top-right language dropdown (EN / ES / PT / FR / JA) · printable PDF view · localStorage prefs · Analysis / Root Cause / Remediation block.

If you find yourself writing `<html>`, `<style>`, or `<table>` in a code-block destined for the user, stop. Read the template instead.

A populated reference rendering ships with this skill at ./example/report.example.html (relative to the skill folder). Open it in a browser to preview the exact layout and depth expected.

Data Payload Contract

The full __ZDX_DATA__ payload is one JSON object. Every field below is required unless marked optional.

{
  "generated_at": "2026-05-18T12:00:00Z",
  "scope": {
    "office":    "Columbus Office (ID: 73260557)",
    "apps":      "Salesforce Lightning (ID 18) & ServiceNow (ID 5)",
    "window":    "Last 24h",
    "generated": "2026-05-18T12:00:00Z"
  },
  "incident": {
    "severity": "critical",
    "title":    "Critical — Multi-App Availability Loss at Columbus Office",
    "body":     "2–4 sentences of plain-language explanation: which apps, which alerts (by ID), what the score and PFT/DNS numbers actually mean, what the recent-3-datapoint pattern shows. Don't be terse — the admin reads this first."
  },
  "kpis": {
    "office":          { "value": "Columbus",          "sub": "Location ID 73260557", "severity": "info" },
    "appsAffected":    { "value": 2,                   "sub": "Salesforce Lightning · ServiceNow", "severity": "critical" },
    "affectedDevices": { "value": 5,                   "sub": "Across both alerts (combined)", "severity": "critical" },
    "avgScore":        { "value": "14 / 100",          "sub": "Critical threshold < 34", "severity": "critical" },
    "sharedHop":       { "value": "Hop 3 — Local ISP", "sub": "203.0.113.4 — 4.2–4.8% loss", "severity": "critical" },
    "deepTrace":       { "value": "⚠ No Data",         "sub": "New traces required", "severity": "warning" }
  },
  "columnOverrides": {
    "hops": {
      "app1Latency": "<App1> Latency",
      "app1Loss":    "<App1> Loss",
      "app2Latency": "<App2> Latency",
      "app2Loss":    "<App2> Loss"
    }
  },
  "tables": {
    "devices": [
      {
        "severity":    "Critical",
        "scope":       "ServiceNow Only",
        "device":      "PC-Aniru-71",
        "user":        "Anirudh Singh\nanirudh.singh@thezerotrustexchange.com",
        "os":          "Windows 11 Pro",
        "deviceId":    "86937887",
        "appsFailing": "ServiceNow"
      }
    ],
    "appMetrics": [
      {
        "severity":      "critical",
        "application":   "Salesforce Lightning",
        "appId":         18,
        "zdxScoreAvg":   15.5,
        "zdxScoreMin":   0,
        "pftAvg":        "7,541 ms",
        "pftMax":        "11,855 ms",
        "dnsAvg":        "7,022 ms",
        "probeFailures": "3 / 24 probes",
        "status":        "Critical"
      }
    ],
    "hops": [
      {
        "severity":    "critical",
        "verdict":     "SHARED",
        "hop":         3,
        "address":     "203.0.113.4 (Local ISP)",
        "app1Latency": "248ms", "app1Loss": "4.2%",
        "app2Latency": "255ms", "app2Loss": "4.8%"
      }
    ]
  },
  "analysis": {
    "summary":     "3–5 sentence narrative explaining what the numbers say across the two apps, why the matching metric profile is the strongest correlation signal, and what was ruled out by comparison.",
    "rootCause":   "1–3 sentence statement of the shared bottleneck with the supporting evidence — name the exact hop or segment and quote the latency/loss numbers that pin it.",
    "remediation": [
      "Immediate (within 1 hour): … (concrete action with target)",
      "Capture evidence (within 2 hours): start two deep-trace sessions on device <id> …",
      "Investigate (within 4 hours): … (specific systems to check)",
      "Monitor: schedule a recurring 15-minute deep trace every 4 hours against one affected device per app …",
      "Communicate: notify the affected user list that the issue is identified as ISP-side and Zscaler / apps are not at fault."
    ]
  }
}

Field rules

• `generated_at` — ISO-8601 UTC timestamp.
• `scope` — REQUIRED object with office, apps, window, generated. The keys map to translated labels (Office, Apps, Window, Generated) automatically. Include the location ID and app IDs inline in the values, as shown.
• `incident` — REQUIRED object. severity is "critical" or "warning". The body is plain text (no HTML); be specific — quote alert IDs, score ranges, exact metric values, and what the most recent probe attempts returned.
• `kpis` — REQUIRED. Every KPI value must be {value, sub, severity} (not just a primitive). severity controls the top-border color and the number color: "critical" | "warning" | "good" | "info" | "neutral".
• `columnOverrides.hops` — REQUIRED. Rename the four generic app1* / app2* columns to the actual application names for this run so the table reads "Salesforce Latency", "ServiceNow Loss", etc.
• `tables.devices` — one row per affected device. severity is "Critical" (in the intersection — failing every impacted app) or "Warning" (failing only some). scope is a free-text label like "ServiceNow Only", "Salesforce Only", or the union string.
• `tables.appMetrics` — one row per impacted application. Numeric or string-with-unit values are both fine. status becomes a colored pill (Critical / Warning / Good).
• `tables.hops` — one row per hop. verdict is "SHARED" | "Degraded" | "OK". Set severity to "critical" on SHARED rows so the row background turns red.
• `analysis` — REQUIRED. Be analytical, not robotic. Tie numbers to conclusions. Match the depth shown in the example.
• Every row in every table must include a severity field ("critical" | "warning" | "good" | "info" | "neutral") — drives the colored row background.

Output Artifacts — MANDATORY

You MUST generate BOTH files below. Both files are REQUIRED output for every multi-app outage diagnosis.

1. Word Document (.docx)

Write a Word document to disk named multi_app_outage_<location>_<date>.docx containing:

• Executive summary: location, applications affected, count of impacted devices, shared bottleneck identified
• Affected devices table (device, user, OS, ZCC version, last seen, list of apps the device is failing on)
• Per-application metric table at the affected location (PFT, DNS, availability) with deltas vs. organization baseline
• Cloud-path hop comparison table (one row per hop, latency/loss per app side-by-side, flag for shared bottleneck)
• Root cause analysis with supporting evidence — explicitly state which hop or segment is shared
• Healthy-reference comparison (a different office hitting the same apps cleanly, or different apps from the same office hitting cleanly)
• Prioritized remediation actions (immediate, investigate, escalate, monitor)

2. Interactive HTML Web Page (.html)

Produced via the template-fill workflow at the top of this file. No inline HTML.

Workflow

This is a multi-tool correlation skill. Treat the steps below as gates — do not skip ahead; each step's output is required to scope the next. Capture everything you collect into the __ZDX_DATA__ payload as you go.

Step 1: Resolve Location & Application IDs

zdx_list_locations(search="Columbus")

Capture the location's numeric id. If multiple locations match (e.g., "Columbus, OH" and "Columbus, GA"), ask the admin which one before proceeding.

zdx_list_applications(
  location_id=["<location_id>"],
  since=24,
  query="[?contains(name, 'Salesforce') || contains(name, 'ServiceNow')].{id: id, name: name, score: score}"
)

Resolve the numeric id for each affected application from the response. The score here is the score at this location — note it for the kpis.avgScore field (low score confirms the alert is genuine).

If an app name is ambiguous (e.g., "Salesforce" matches both "Salesforce" and "Salesforce Marketing Cloud"), confirm with the admin which one is affected.

Step 2: Confirm With Active Alerts at the Location

zdx_list_alerts(location_id=["<location_id>"], since=24)

Filter to alerts whose application / app_id matches one of the IDs from Step 1.

For each matching alert:

zdx_get_alert(alert_id="<alert_id>")

Capture: start_time, duration, alert trigger / threshold, impacted department(s), and the bottleneck (e.g., pft, dns, availability). Quote the alert IDs verbatim into incident.body — they are part of the evidence.

If NO alerts exist for either app at this location, skip to Step 3 anyway — metrics + device lists are enough.

Step 3: Identify Affected Devices Per Alert and Intersect