p00318821-eng/skills-plugins-hooks-agents/spark-operations-cli

Name: spark-operations-cli
Author: p00318821-eng

Claude Code and agent skill by p00318821-eng/skills-plugins-hooks-agents

Trust tier 2

community

By p00318821-eng
Coding
p00318821-eng/skills-plugins-hooks-agents
skills/spark-operations-cli/SKILL.md
Updated 9 Nov 2026
Version 1.0.01 version recorded

Source check pending

Checking that the SKILL.md file is still in the repository.

Diagnose failed Spark jobs, unhealthy Livy sessions, and performance bottlenecks in Microsoft Fabric via read-only CLI triage. Use when the user wants to: (1) diagnose why a Spark job, notebook run, or Lakehouse job failed, (2) triage stuck or dead Livy sessions, (3) identify OOM, shuffle spill, or data skew, (4) retrieve driver and executor logs or Spark Advisor findings, (5) copy event logs and start a local Spark History Server, (6) diagnose all Spark activities within a failed pipeline run. Triggers: "diagnose my failed notebook", "why did my spark job fail", "triage spark failure", "diagnose pipeline run failure", "why did my pipeline fail", "livy session stuck in starting", "spark executor OOM", "check spark advisor findings", "shuffle spill diagnosis", "why did my lakehouse job fail", "diagnose lakehouse table load", "data skew diagnosis", "open spark history server locally", "analyze spark failure logs", "spark job triage".

Install

npx vskill@latest install p00318821-eng/skills-plugins-hooks-agents/spark-operations-cli

SKILL.md

Update Check — ONCE PER SESSION (mandatory)
The first time this skill is used in a session, run the check-updates skill before proceeding.
- GitHub Copilot CLI / VS Code: invoke the check-updates skill.
- Claude Code / Cowork / Cursor / Windsurf / Codex: compare local vs remote package.json version.
- Skip if the check was already performed earlier in this session.

CRITICAL NOTES
1. To find the workspace details (including its ID) from workspace name: list all workspaces and, then, use JMESPath filtering
2. To find the item details (including its ID) from workspace ID, item type, and item name: list all items of that type in that workspace and, then, use JMESPath filtering
3. Skill disambiguation: spark-operations-cli is for read-only triage and diagnosis of existing jobs and sessions. For creating notebooks, running new jobs, or Spark development, use spark-authoring-cli. For interactive PySpark analysis and Livy session creation, use spark-consumption-cli.

Spark Operations — CLI Skill

This skill provides diagnostics for Microsoft Fabric Spark job failures, Livy session health, and performance bottlenecks using Fabric REST APIs and CLI tools (az rest). All diagnostic operations are read-only; session cleanup (e.g., stopping zombie sessions) requires explicit user confirmation. For Spark development and notebook authoring, use spark-authoring-cli. For interactive PySpark analysis, use spark-consumption-cli.

The TOC is grouped by purpose. Start at Diagnostic Workflows when triaging an active failure; the earlier sections are foundational references.

1. Fabric Foundations (concepts)

Task	Reference	Notes
Fabric Topology & Key Concepts	COMMON-CORE.md § Fabric Topology & Key Concepts
Environment URLs	COMMON-CORE.md § Environment URLs
Authentication & Token Acquisition	COMMON-CORE.md § Authentication & Token Acquisition	Wrong audience = 401; read before any auth issue
Core Control-Plane REST APIs	COMMON-CORE.md § Core Control-Plane REST APIs
Pagination	COMMON-CORE.md § Pagination
Long-Running Operations (LRO)	COMMON-CORE.md § Long-Running Operations (LRO)
Rate Limiting & Throttling	COMMON-CORE.md § Rate Limiting & Throttling
Job Execution	COMMON-CORE.md § Job Execution
Capacity Management	COMMON-CORE.md § Capacity Management
Gotchas & Troubleshooting	COMMON-CORE.md § Gotchas & Troubleshooting
Best Practices	COMMON-CORE.md § Best Practices

2. CLI Setup & Authentication

Task	Reference	Notes
Tool Selection Rationale	COMMON-CLI.md § Tool Selection Rationale
Finding Workspaces and Items in Fabric	COMMON-CLI.md § Finding Workspaces and Items in Fabric	Mandatory — READ link first [needed for finding workspace id by its name or item id by its name, item type, and workspace id]
Authentication Recipes	COMMON-CLI.md § Authentication Recipes	`az login` flows and token acquisition
Fabric Control-Plane API via `az rest`	COMMON-CLI.md § Fabric Control-Plane API via az rest	Always pass `--resource https://api.fabric.microsoft.com` or `az rest` fails
Pagination Pattern	COMMON-CLI.md § Pagination Pattern
Long-Running Operations (LRO) Pattern	COMMON-CLI.md § Long-Running Operations (LRO) Pattern
Gotchas & Troubleshooting (CLI-Specific)	COMMON-CLI.md § Gotchas & Troubleshooting (CLI-Specific)	`az rest` audience, shell escaping, token expiry
Quick Reference: `az rest` Template	COMMON-CLI.md § Quick Reference: az rest Template
Quick Reference: Token Audience / CLI Tool Matrix	COMMON-CLI.md § Quick Reference: Token Audience ↔ CLI Tool Matrix	Which `--resource` + tool for each service

3. Spark Sessions, Notebooks & Jobs (background)

Task	Reference	Notes
Livy Session Management	SPARK-CONSUMPTION-CORE.md § Livy Session Management	Session creation, states, lifecycle, termination
Interactive Data Exploration	SPARK-CONSUMPTION-CORE.md § Interactive Data Exploration	Statement execution, output retrieval, data discovery
Notebook Execution & Job Management	SPARK-AUTHORING-CORE.md § Notebook Execution & Job Management

4. Spark Monitoring APIs (primary triage surface)

Task	Reference	Notes
Spark Monitoring API Overview	SPARK-MONITORING-CORE.md § Overview	GA monitoring APIs — no active session required
Workspace & Item Session Listing	SPARK-MONITORING-CORE.md § Workspace and Item-Level Session Listing	List Spark apps across workspace with filtering
Spark Advisor API	SPARK-MONITORING-CORE.md § Spark Advisor API	Key — automated skew detection, task errors, recommendations
Open-Source Spark History Server APIs	SPARK-MONITORING-CORE.md § Open-Source Spark History Server APIs	Jobs, stages, executors, SQL queries via REST
Driver and Executor Log APIs	SPARK-MONITORING-CORE.md § Driver and Executor Log APIs	Direct log retrieval without active session
Livy Log API	SPARK-MONITORING-CORE.md § Livy Log API	Session-level log with byte-offset pagination
Resource Usage API	SPARK-MONITORING-CORE.md § Resource Usage API	vCore timeline, idle/running cores, efficiency metrics
Monitoring Diagnostic Workflow	SPARK-MONITORING-CORE.md § Diagnostic Workflow Using Monitoring APIs	Step-by-step triage using monitoring APIs

5. Diagnostic Workflows (start here for active triage)

Task	Reference	Notes
Automated Diagnostic Workflow (full)	automated-diagnostic-workflow.md	Steps 1–7: resolve → route by state → failure/perf/resource/health → report. Includes Step 1b expired-data fallback and report templates
Diagnostic Tiers	diagnostic-workflow.md § Diagnostic Tiers	Tier 1 (online REST) vs Tier 2 (local SHS)
Key Diagnostic Patterns	diagnostic-workflow.md § Key Diagnostic Patterns	Symptom → first check → likely cause lookup
Severity Thresholds	diagnostic-workflow.md § Severity Thresholds	Metric thresholds for classifying findings
Manual CLI Recipes	diagnostic-workflow.md § Manual CLI Recipes	Ad-hoc diagnostic commands for manual use
Pipeline Run Diagnosis	pipeline-diagnosis.md	Diagnose all Spark activities within a pipeline run (Steps P1–P6)

6. Job Failure Diagnostics

Task	Reference	Notes
Failure Triage Workflow	job-diagnostics.md § Failure Triage Workflow	Step-by-step decision tree for diagnosing failures
Job Failure Classification	job-diagnostics.md § Failure Classification	OOM, shuffle, timeout, dependency, configuration errors
Reading Spark Logs via REST	job-diagnostics.md § Reading Spark Logs via REST	Driver/executor log retrieval from Livy
Job Instance History	job-diagnostics.md § Job Instance History	Query recent runs, compare durations, detect regressions

7. Livy Session Health

Task	Reference	Notes
Session Health Assessment	session-health.md § Livy Session Lifecycle	Session states, transitions, expected durations
Idle and Zombie Session Detection	session-health.md § Idle and Zombie Session Detection	Find and clean up leaked sessions
Session Resource Monitoring	session-health.md § Session Resource Monitoring	Memory and executor usage via Livy
Session Recovery Patterns	session-health.md § Session Recovery Patterns	Restart strategies and session replacement

8. Performance Diagnostics

Task	Reference	Notes
Performance Anti-Patterns	performance-patterns.md § Anti-Patterns	Spill, shuffle, skew, small files, collect misuse
Stage and Task Analysis	performance-patterns.md § Stage and Task Analysis	Reading Spark UI metrics via REST
Optimization Recipes	performance-patterns.md § Optimization Recipes	Partition tuning, broadcast joins, caching
Capacity and Resource Diagnostics	performance-patterns.md § Capacity and Resource Diagnostics	CU consumption, throttling detection

9. Offline / Deep-Dive Tools

Task	Reference	Notes
JobInsight Event Log Copy	jobinsight-api.md § LogUtils.copyEventLog	Copy event logs from Fabric to OneLake for offline analysis
Local Spark History Server	spark-history-server.md § Overview	Start local SHS for full Spark UI (DAG, tasks, SQL plans)

Must/Prefer/Avoid

MUST DO

This is the start of the file. Read the full SKILL.md on GitHub.

Security

Security scanPassed, 100 of 100Pattern checks on the skill's files.
Model reviewPassed

See the full security report

Quality evaluation

Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

SKILL.md in the repositoryCheckingskills/spark-operations-cli/SKILL.md

Popularity

7-day trend score: 5

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

Claude Code
Cursor
GitHub Copilot
Windsurf
Codex
Gemini CLI
Cline
Amp
Roo Code
Goose
Continue
Kilo Code
Junie
Trae
OpenHands
Qwen Code
AI Agent
Tabnine Chat
Sourcegraph Cody
Amazon Q
Firebase Genkit
Sweep
Devin
Phind
Mentat
Claude Desktop
Aider
GPT Pilot
Plandex
bolt.new
v0
Replit Agent
Anse
TypingMind
BetterChatGPT
LibreChat
Warp AI
Pieces
Fig AI

Embed badge

[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/p00318821-eng/skills-plugins-hooks-agents/spark-operations-cli/badge)](https://verified-skill.com/skills/p00318821-eng/skills-plugins-hooks-agents/spark-operations-cli)

Something wrong with this skill or its page? Tell us.