jswortz/my-skills/dssib-vertex-pipelines

Claude Code and agent skill by jswortz/my-skills

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

-- name: dssib-vertex-pipelines description: Guide for deploying Dynamic Swarm (DSSIB) Fan-Out agents on Vertex AI Pipelines with ML Metadata integrat...

Install

npx vskill@latest install jswortz/my-skills/dssib-vertex-pipelines

SKILL.md

The same SKILL.md was published earlier as jswortz/my-skills/dssib-vertex-pipelines.

-- name: dssib-vertex-pipelines description: Guide for deploying Dynamic Swarm (DSSIB) Fan-Out agents on Vertex AI Pipelines with ML Metadata integration.


Deploying DSSIB Agents on Vertex AI Pipelines

Use this skill when deploying or architecting a multi-agent fan-out orchestration system (like NovaStorm) on Google Cloud. Instead of using Cloud Run for long-running asynchronous background agents, Vertex AI Pipelines is the preferred native orchestrator.

Core Concepts

  1. Vertex AI Pipelines (`kfp`): Use Kubeflow pipeline definitions to orchestrate the ReAct loop. It handles the parallelization gracefully.
  2. Parallel Fan-out: Rather than spawning async asyncio tasks in an HTTP container, use kfp.dsl.ParallelFor to achieve robust parallelism for the analytical agents querying BigQuery in parallel.
  3. ML Metadata (MLMD): The most important benefit is native lineage tracking. Use kfp.dsl.Metrics and kfp.dsl.Artifact to track all generated outputs (insights, visual charts, mutated agent scripts/skills) into Vertex ML Metadata.
  4. Dataplex Integration: The ML Metadata naturally feeds back into the Dataplex Universal Catalog, ensuring that the insights generated by the swarm are deeply integrated into the data governance layer.
  5. Scheduled Execution: Use the PipelineJobSchedule.create API to orchestrate hourly or daily pipeline runs, streamlining the architecture by removing the need for external Cloud Scheduler to Cloud Run HTTP triggers.
  6. GEPA (Generative Evolutionary Prompt Augmentation): Use dynamic LLM temperature-dilated mutation to evolve analytical agents' "Skill DNA" over extended simulations (e.g., 50 epochs), hybridizing crossover skills.
  7. Secure Secret Injection: Never pass API keys or credentials as plain text pipeline arguments. Always mount them dynamically inside the component via Secret Manager at runtime.

Implementation Example Pattern

python
from kfp import dsl
from google.cloud import aiplatform
@dsl.component
def bq_insight_generator(target: str) -> str:
# BQ local agent and Reasoning Engine logic
return f"Insight for {target}"
@dsl.pipeline(name="dssib-fanout-pipeline", description="Dynamic Swarm Orchestration")
def fanout_pipeline(table_targets: list):
with dsl.ParallelFor(table_targets) as target:
# Fan out insights dynamically across all target tables
insight = bq_insight_generator(target=target)

Advantages Over Cloud Run + HTTP Triggers

  • Automatic UI visualization of the 10+ parallel fan-out steps in Vertex AI.
  • Native metadata logging (tracking LLM prompts, token counts, and output insights as pipeline artifacts).
  • First-class Google Cloud integrations with BigQuery, Dataplex, and TimesFM models.

Security

  • Security scanPassed, 95 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

Repository stars
2
Repository forks
2
7-day trend score
10

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/jswortz/my-skills/dssib-vertex-pipelines/badge)](https://verified-skill.com/skills/jswortz/my-skills/dssib-vertex-pipelines)

Something wrong with this skill or its page? Tell us.