gemini-cli-extensions/data-agent-kit-starter-pack/gcp-spark

Claude Code and agent skill by gemini-cli-extensions/data-agent-kit-starter-pack

Trust tier 2
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Authors, optimizes, and runs PySpark and Spark code on Google Cloud Managed Spark (Dataproc Serverless, clusters, and Spark Connect). Use when: - Authoring or running Spark/PySpark notebooks in any kernel (Local Kernel with spark-remote templates, Remote Spark / Remote Kernel, ManagedSparkSession, Spark Connect). - Querying, transforming, or writing data with PySpark or Spark SQL, including Apache / BigLake Iceberg, BigQuery, GCS, Spanner, Cloud SQL, AlloyDB, Amazon S3, or Azure Blob Storage / ADLS Gen2. - Building PySpark ETL pipelines, machine learning workloads, or batch jobs. - Managing Dataproc clusters, serverless batches, or interactive sessions. - Optimizing PySpark or Spark SQL queries for performance, memory, and OOM prevention. Do NOT use when: - Auditing multi-region fleet inventory, DCU/cost consumption, idle/zombie clusters, or workload observability (use @skill:gcp-spark-observability). - Troubleshooting failed Spark jobs or Dataproc batches (use @skill:gcp-spark-troubleshooting).

Install

npx vskill@latest install gemini-cli-extensions/data-agent-kit-starter-pack/gcp-spark

SKILL.md

The same SKILL.md was published earlier as gemini-cli-extensions/data-agent-kit-starter-pack/gcp-spark.

Managed Spark on Google Cloud

[!IMPORTANT]
You MUST follow the Task Execution Workflow when writing spark code.

Notebook Generation Environment Rules

[!CAUTION]
When the task is to generate a Spark notebook (as opposed to executing
one), you MUST NOT modify your own local execution environment (i.e. do
not run these commands with your shell or other tools). Specifically:
- NEVER create a virtual environment (python -m venv, virtualenv,
conda create, uv venv, poetry init/install, etc.).
- NEVER install packages (pip install, %pip install,
uv pip install, conda install, poetry add).
- NEVER run environment discovery loops such as activating a venv,
pip list, or probing for ipykernel / google-cloud-spark-connect as a
precondition for writing the notebook.
Assume the user will select the appropriate kernel and dependencies after the
notebook is generated.

Task Execution Workflow

  1. Understand user request:
  • New Spark notebooks target the Local Python kernel and connect to

Managed Spark Serverless with ManagedSparkSession (Spark Connect); do NOT ask the user which kernel to use. Only when the prompt explicitly asks for a Remote Spark kernel or to run against a Dataproc cluster, skip session initialization and use the pre-initialized spark variable. Use the Runtime Profile (formerly Session Template) named in the prompt; otherwise ask the user which one to use (scan for existing ones with gcloud dataproc session-templates list --location=<REGION>). Local Python notebooks MUST start with these two cells (plain constants; no os.environ, no Session() objects):

``python PROJECT_ID = "<PROJECT_ID>" REGION = "<REGION>" RUNTIME_PROFILE = "<RUNTIME_PROFILE_ID>" ``

```python # %pip install -U 'google-cloud-spark-connect[client,interactive]~=2.2' from google.cloud.managed_spark_connect import ManagedSparkSession

spark = ( ManagedSparkSession.builder .projectId(PROJECT_ID) .location(REGION) .runtimeProfile(RUNTIME_PROFILE) .getOrCreate() ) ```

  1. Understand schemas: ALWAYS use @skill:discover-assets skill or

references/schema_direct_inspection.md to understand input and output schemas. Include the schema in your thought process BEFORE generating any code. Do NOT guess column names. Cap GCP data asset discovery attempts at 3 retries max. Unless explicitly specified, assume that the assets are located in the same project. Avoid scanning for assets across other projects as it can take a long time. If an expected dataset or table does not exist, use @skill:discover-assets to discover all similar tables in the namespace or project.

MISSING SOURCE RULE: If the request does not name a table, GCS path or catalog (or a required database instance/connection or s3a:// / abfss:// / wasbs:// / wasb:// URI), you MUST run at most ONE @skill:discover-assets search. If it does not return a single clear match, you MUST list the candidates, ask the user which source to use, and STOP. You MUST NOT infer the source by listing buckets, reading existing notebooks or scripts, or running SHOW NAMESPACES / SHOW TABLES.

MINOR TYPO RULE: If there is a minor singular/plural or 1–2 character typo in a table name (e.g. employees vs employee, or loans vs loan), immediately fix the table name in code and proceed without asking for user confirmation, noting the correction in your response.

STRICT HALT RULE: If the discovered table names differ from the requested table by more than a minor typo (e.g. completely different words, prefixes, or suffixes), you must IMMEDIATELY report the missing table and a neutral list of all available alternatives in the same namespace to the user without making any recommendations. You MUST ask the user which alternative to use and then STOP EXECUTING your turn. Do NOT write any Spark code or notebooks. Do NOT proceed with code generation, do NOT add fallback logic to code, and do NOT automatically substitute any alternative table (even if its schema seems to match) without explicit user permission.

  1. Verify source accessibility: For Google Cloud Storage (gs://) paths,

verify access/existence using gcloud storage ls gs://<path-to-dataset> (skip gcloud storage ls for external cloud URIs such as s3a://, abfss://, wasbs://, or wasb://, which are accessed directly by Spark's Hadoop FileSystem connectors). If accessing or reading a GCS path fails with a storage error e.g., permission errors like 403 Forbidden/Forbidden/PermissionDenied, or location errors like 404 Not Found/NotFound/FileNotFoundException you should report the error immediately. Either (1) ask the user what to do next, or (2) if asked to execute a notebook, save the notebook with the error output and recommend next steps to resolve the issue. Do NOT scan all buckets for alternative fallback datasets when encountering GCS errors. NEVER create a new GCS bucket (gcloud storage buckets create) without explicit user confirmation; if a required bucket does not exist, STOP and ask the user.

  1. Generate spark code:
  • Output Format: ALWAYS generate code in **Python Notebooks

(.ipynb)** format unless a standalone script or batch/cluster job is explicitly requested: generate a Python script (.py) when a PySpark script/job is requested, or a Spark SQL script (.sql) when a Spark SQL script or spark-sql job/batch is requested (refer to references/gcloud_dataproc.md for spark-sql syntax and submission).

  • Spark Session Initialization:

> [!IMPORTANT] Initializing a Spark session on Google Cloud can take 2-3 > minutes. You MUST inform the user about the potential delay.

> [!CAUTION] You MUST NOT create a local Spark session. The > following are BANNED: - SparkSession.builder.master("local") - > pyspark.sql.SparkSession.builder.getOrCreate() in notebooks or local > Spark Connect scripts - Any try/except fallbacks that revert to a > local SparkSession - Silently switching execution modes when an > explicitly requested mode fails (e.g. falling back from Spark Connect > / interactive session to gcloud dataproc batches submit or cluster > creation without asking the user). > > - Remote Kernel Notebooks (`.ipynb`): spark is > pre-initialized; use it directly without calling > ManagedSparkSession or SparkSession.builder.getOrCreate(). > - Local Python Notebooks (`.ipynb`) & Spark Connect scripts: You > MUST use ManagedSparkSession from > google-cloud-spark-connect to connect to Managed Spark > Serverless. > - Dataproc Batch & Cluster jobs (`.py` submitted via > `gcloud dataproc batches submit pyspark` or > `gcloud dataproc jobs submit pyspark`): Use standard > SparkSession.builder.appName("<APP_NAME>").getOrCreate() (never > .master("local")).

Refer to references/gcloud_dataproc.md for detailed configuration. Minimal initialization for Local Python notebooks / Spark Connect:

```python from google.cloud.managed_spark_connect import ManagedSparkSession

spark = ( ManagedSparkSession.builder.projectId("<PROJECT_ID>") .location("<REGION>") .getOrCreate() ) ```

  • Production Logging: In standalone production PySpark batch jobs and

scripts (.py), you MUST use the standard Python logging module instead of print() statements for job lifecycle, progress, and record counts. Do NOT use the logging module in notebooks (.ipynb); use print(), .show(), and standard cell outputs instead. Configure logging for .py scripts with timestamps:

```python import logging

logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s", ) ```

  • Security & Dynamic Variables: You MUST NOT hardcode plaintext

credentials, passwords, or tokens (use Secret Manager — refer to references/jdbc_cloudsql.md when retrieving credentials from Secret Manager). You MUST NOT hardcode environment-specific variables; parameterize project IDs, dataset IDs, table names, and bucket paths via os.environ.get(...) or arguments.

  • Missing or Inaccessible Custom JAR Rule (Do Not Improvise):
  • If a required custom gs://.../*.jar URI (e.g., JDBC driver or

Socket Factory Uber JAR) is not provided by the user, or if Spark session/job execution fails because the JAR does not exist or is inaccessible in GCS, you MUST STOP and ask the user for the gs:// JAR URI(s).

  • NEVER guess JAR paths, scan arbitrary GCS buckets for .jar

files, download JARs from Maven Central (curl/wget) to upload to GCS, or substitute spark.jars.packages Maven coordinates.

  • Read and Write data: ALWAYS load only the connector reference

file(s) needed for the prompt:

This is the start of the file. Read the full SKILL.md on GitHub.

Security

  • Security scanPassed, 100 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

Repository stars
187
Repository forks
57
7-day trend score
26

Stars and forks are for the whole GitHub repository, not this skill alone. Refreshed 29 Jul 2026, 3 months ago.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/gemini-cli-extensions/data-agent-kit-starter-pack/gcp-spark/badge)](https://verified-skill.com/skills/gemini-cli-extensions/data-agent-kit-starter-pack/gcp-spark)

Something wrong with this skill or its page? Tell us.