devsh-orchestrator - Multi-Agent Orchestration Skill
Purpose: Enable head agents (like Claude Code CLI running locally or in cloud workspaces) to orchestrate multiple sub-agents running in cloud sandboxes. Supports parallel task execution, dependency management, real-time status updates, and inter-agent coordination.
Use Cases
- Parallel Development: Spawn multiple agents to work on different parts of a codebase simultaneously
- Task Distribution: Break down complex tasks and assign to specialized agents
- Review Coordination: Have one agent write code while another reviews
- Test Automation: Run tests in parallel across different environments
- Head Agent Mode: Run as a cloud workspace that coordinates sub-agents
Quick Start
bash
# Spawn a sub-agent to work on a specific task
devsh orchestrate spawn --agent claude/haiku-4.5 --repo owner/repo "Fix the auth bug in login.ts"
# Spawn as a cloud workspace (head agent) for coordinating sub-agents
devsh orchestrate spawn --cloud-workspace --agent claude/opus-4.6 "Coordinate feature implementation"
# Check status of spawned agents (with real-time updates)
devsh orchestrate status <orch-task-id> --watch
# Get aggregated results from all sub-agents
devsh orchestrate results <orchestration-id>
# Wait for an orchestration task to complete
devsh orchestrate wait <orch-task-id>
# Send a message to a running agent (uses task-run-id)
devsh orchestrate message <task-run-id> "Please also update the tests" --type request
# Cancel an orchestration task
devsh orchestrate cancel <orch-task-id>
Commands
Spawn Agent
Spawn a new sub-agent in a sandbox to work on a task.
bash
devsh orchestrate spawn [flags] "prompt"
# Flags:
# --agent <name> Agent to use (default: claude/haiku-4.5)
# --repo <owner/repo> GitHub repository to clone
# --branch <name> Branch to checkout (default: main)
# --pr-title <title> Pull request title
# --priority <n> Task priority (0 = highest, default: 5)
# --depends-on <id> Task ID this depends on (can repeat)
# --use-env-jwt Use CMUX_TASK_RUN_JWT for auth (for sub-agent spawning)
# --cloud-workspace Spawn as orchestration head (cloud workspace)
# --json Output result as JSON
Examples:
bash
# Simple spawn
devsh orchestrate spawn --agent claude/haiku-4.5 --repo owner/repo "Add input validation to the API"
# Spawn as orchestration head (cloud workspace)
devsh orchestrate spawn --cloud-workspace --agent claude/opus-4.6 "Coordinate feature implementation across multiple agents"
# Spawn with dependencies
devsh orchestrate spawn \
--agent codex/gpt-5.1-codex-mini \
--depends-on ns7abc123 \
"Write tests for the changes made in the previous task"
# Spawn from within a head agent (uses JWT auth)
devsh orchestrate spawn --use-env-jwt --agent claude/haiku-4.5 "Sub-task from coordinator"
Note: Each spawned agent works on a new branch ({prefix}{slug}-{id}, prefix defaults to dev/), not the base branch. After completion, use gh pr create or enable Auto-PR in settings to open pull requests. See the devsh skill documentation for details on branch and PR behavior.
Get Orchestration Task Status
Get detailed status of a specific orchestration task.
bash
devsh orchestrate status <orch-task-id> [flags]
# Flags:
# --watch, -w Continuously monitor status (exits on terminal state)
# --interval <seconds> Polling interval for watch mode (default: 3)
# --json Output as JSON
Watch Mode:
Watch mode provides real-time status updates and automatically exits when the task reaches a terminal state (completed, failed, or cancelled).
bash
# Monitor task with live updates
devsh orchestrate status k97xcv2... --watch
# Custom polling interval
devsh orchestrate status k97xcv2... --watch --interval 5
Get Orchestration Results
Get aggregated results from all sub-agents in an orchestration.
bash
devsh orchestrate results <orchestration-id> [flags]
# Flags:
# --use-env-jwt Use CMUX_TASK_RUN_JWT for auth (for head agents)
# --json Output as JSON
Examples:
bash
# Get results for an orchestration
devsh orchestrate results orch_abc123
# Get results as JSON (for parsing)
devsh orchestrate results orch_abc123 --json
# Get results from within a head agent
devsh orchestrate results orch_abc123 --use-env-jwt
List Agents
List all spawned sub-agents and their status.
bash
devsh orchestrate list [flags]
# Flags:
# --status <state> Filter by status (pending, running, completed, failed)
# --json Output as JSON
Wait for Orchestration Task
Wait for an orchestration task to complete (or timeout).
bash
devsh orchestrate wait <orch-task-id> [flags]
# Flags:
# --timeout <duration> Timeout duration (default: 5m)
# --json Output as JSON
Send Message
Send a message to a running agent via the mailbox.
bash
devsh orchestrate message <task-run-id> "message" [flags]
# Flags:
# --type <type> Message type: handoff, request, status (required)
Cancel Orchestration Task
Cancel an orchestration task.
bash
devsh orchestrate cancel <orch-task-id>
Head Agent Mode
Cloud workspaces spawned with --cloud-workspace act as orchestration head agents that coordinate multiple sub-agents. Head agents receive:
- Special Environment Variables:
CMUX_IS_ORCHESTRATION_HEAD=1 - Identifies this as a head agentCMUX_ORCHESTRATION_ID - Unique orchestration session ID
- Head Agent Instructions: A file at
/root/lifecycle/memory/orchestration/HEAD_AGENT_INSTRUCTIONS.md with coordination guidance
- Bi-directional Sync: Access to
pull_orchestration_updates MCP tool for syncing local state with server
Head Agent Responsibilities
- Plan the Work: Break down the overall task into discrete sub-tasks
- Spawn Sub-Agents: Use
devsh orchestrate spawn to create sub-agents - Monitor Progress: Use
devsh orchestrate status --watch to track completion - Collect Results: Use
devsh orchestrate results to aggregate outputs - Coordinate: Handle dependencies and sequencing between tasks
Bi-directional Sync
Head agents can sync their local PLAN.json with the server state using MCP:
javascript
// Pull latest state from server
pull_orchestration_updates() // Returns aggregated state from all sub-agents
This updates local PLAN.json with:
- Current status of all sub-agent tasks
- Unread messages from the mailbox
- Aggregated completion counts
Real-Time Updates (SSE)
The orchestration system supports Server-Sent Events for real-time updates:
SSE Endpoint
GET /api/orchestrate/events/<orchestration-id>
Events sent:
connected - Connection establishedtask_status - Task status changedtask_completed - Task completed (with result)heartbeat - Keep-alive every 30 seconds
Using SSE in Watch Mode
The --watch flag automatically uses SSE when available for minimal latency:
bash
devsh orchestrate status <id> --watch
Orchestration Patterns
1. Sequential Pipeline
Run agents in sequence where each depends on the previous.
bash
# Step 1: Implement feature
RUN1=$(devsh orchestrate spawn --json --agent claude/sonnet-4.5 "Implement user authentication" | jq -r '.orchestrationTaskId')
# Step 2: Write tests (depends on step 1)
RUN2=$(devsh orchestrate spawn --json --depends-on $RUN1 --agent codex/gpt-5.1-codex-mini "Write tests for auth" | jq -r '.orchestrationTaskId')
# Step 3: Review (depends on step 2)
devsh orchestrate spawn --depends-on $RUN2 --agent claude/opus-4.5 "Review the implementation and tests"
# Wait for final task
devsh orchestrate status $RUN3 --watch
2. Parallel Fan-Out
Spawn multiple agents to work on independent tasks simultaneously.
bash
# Spawn multiple agents in parallel
devsh orchestrate spawn --agent claude/haiku-4.5 "Fix bug in auth.ts" &
devsh orchestrate spawn --agent claude/haiku-4.5 "Fix bug in api.ts" &
devsh orchestrate spawn --agent claude/haiku-4.5 "Fix bug in db.ts" &
wait
# Wait for all to complete using watch mode
for id in $(devsh orchestrate list --status running --json | jq -r '.[].orchestrationTaskId'); do
devsh orchestrate status $id --watch &
done
wait
3. Leader-Worker Pattern
One cloud workspace coordinates, workers execute.
bash
# Spawn a head agent (cloud workspace) to coordinate
devsh orchestrate spawn --cloud-workspace --agent claude/opus-4.6 \
"Analyze the codebase and coordinate implementing user roles across multiple agents"
Inside the cloud workspace, the head agent can:
bash
# Spawn workers for specific tasks
devsh orchestrate spawn --use-env-jwt --agent claude/haiku-4.5 "Implement role model"
devsh orchestrate spawn --use-env-jwt --agent claude/haiku-4.5 "Add role-based middleware"
devsh orchestrate spawn --use-env-jwt --agent codex/gpt-5.1-codex-mini "Write role tests"
# Monitor all tasks
devsh orchestrate status $TASK_ID --watch
# Collect results
devsh orchestrate results $CMUX_ORCHESTRATION_ID --use-env-jwt
4. Review Loop
Implementation and review workflow.
bash
# Implement
RUN1=$(devsh orchestrate spawn --json --agent claude/sonnet-4.5 "Implement feature X" | jq -r '.orchestrationTaskId')
# Wait for implementation
devsh orchestrate status $RUN1 --watch
# Review
devsh orchestrate spawn --agent claude/opus-4.5 \
"Review the changes from task $RUN1. Check for bugs, security issues, and code quality."
Memory Structure
Orchestration data is stored at /root/lifecycle/memory/orchestration/:
orchestration/
├── PLAN.json # Current orchestration plan
├── AGENTS.json # Spawned sub-agents registry
├── EVENTS.jsonl # Event log for debugging
└── HEAD_AGENT_INSTRUCTIONS.md # Coordination guidance (head agents only)
PLAN.json
Stores the orchestration plan for the current workflow.
json
{
"version": 1,
"orchestrationId": "orch_abc123",
"createdAt": "2025-02-23T12:00:00Z",
"status": "running",
"headAgent": "claude/opus-4.6",
"isOrchestrationHead": true,
"description": "Implement user authentication feature",
"tasks": [
{
"id": "task_001",
"prompt": "Implement auth endpoints",
"agentName": "claude/sonnet-4.5",
"status": "completed",
"orchestrationTaskId": "ns7abc123",
"result": "Successfully implemented auth endpoints"
},
{
"id": "task_002",
"prompt": "Write auth tests",
"agentName": "codex/gpt-5.1-codex-mini",
"status": "running",
"dependsOn": ["task_001"],
"orchestrationTaskId": "ns7def456"
}
]
}
AGENTS.json
Tracks all spawned agents for this orchestration session.
json
{
"version": 1,
"orchestrationId": "orch_abc123",
"agents": [
{
"orchestrationTaskId": "ns7abc123",
"taskRunId": "tr_xyz789",
"agentName": "claude/sonnet-4.5",
"status": "completed",
"spawnedAt": "2025-02-23T12:00:00Z",
"completedAt": "2025-02-23T12:15:00Z",
"prompt": "Implement auth endpoints",
"sandboxId": "morphvm_abc123"
}
]
}
EVENTS.jsonl
Append-only log of orchestration events.
jsonl
{"timestamp":"2025-02-23T12:00:00Z","event":"agent_spawned","agentName":"claude/sonnet-4.5","orchestrationTaskId":"ns7abc123"}
{"timestamp":"2025-02-23T12:15:00Z","event":"agent_completed","orchestrationTaskId":"ns7abc123","status":"completed"}
{"timestamp":"2025-02-23T12:15:01Z","event":"message_sent","from":"head","to":"ns7def456","type":"handoff"}
Environment Variables