baphuongna/pi-crew/child-pi-spawning

child-pi-spawning

Child Pi workers are subprocesses spawned by task-runner.ts via runChildPi() in child-pi.ts. Understanding the spawn flow, lifecycle events, and failure modes is essential for debugging worker crashes and "worker blinks" issues.

Spawn Flow

task-runner.ts (runTeamTask)
  → runChildPi({ cwd, task, agent, model, skillPaths, signal, onLifecycleEvent })
    → child-pi.ts (runChildPi main function)
      → buildPiWorkerArgs() → getPiSpawnCommand() → spawn(command, args, options)
        → ChildProcess spawned
        → activeChildProcesses.set(pid, child)
        → input.onLifecycleEvent({ type: "spawned", pid, ts })
        → stdout.on("data") → ChildPiLineObserver
        → stderr.on("data")
        → child.on("error") → onLifecycleEvent("spawn_error")
        → child.on("exit") → onLifecycleEvent("exit")
        → child.on("close") → onLifecycleEvent("close"), settle(result)

Key components

• ChildPiLineObserver: Parses JSON events and stdout lines from child Pi's output stream
• Response timeout: 5-minute timer resets on every stdout/stderr chunk; on timeout → SIGTERM
• Final drain: After last assistant event, waits finalDrainMs (default 2s) then SIGTERM
• Hard kill: After hardKillMs (default 2s) from SIGTERM, SIGKILL
• Active process tracking: activeChildProcesses Map for global cleanup

Lifecycle Events

ChildPiLifecycleEvent interface — emitted via onLifecycleEvent callback:

interface ChildPiLifecycleEvent {
  type: "spawned" | "spawn_error" | "response_timeout" | "final_drain" | "hard_kill" | "exit" | "close";
  pid?: number;
  exitCode?: number | null;
  error?: string;
  ts: string;
}

Event sequence for normal completion:

1. spawned      pid=12345            ← child.pid assigned
2. [stdout events: message, tool_execution_start, tool_execution_end, message_end...]
3. final_drain  pid=12345            ← last assistant event received, SIGTERM sent
4. exit         exitCode=0           ← process exited
5. close        exitCode=0           ← stdio fully closed

Event sequence for crash:

1. spawned      pid=12345
2. spawn_error   error="..."         ← OR →
3. exit         exitCode=1
4. close        exitCode=1

Event sequence for timeout:

1. spawned      pid=12345
2. [no stdout for 5 min]
3. response_timeout error="No output for 300000ms"
4. final_drain  pid=12345
5. hard_kill    pid=12345            ← SIGKILL after hardKillMs
6. exit         exitCode=null
7. close        exitCode=null

onLifecycleEvent Callback Pattern

The callback bridges child-pi events → events.jsonl:

// task-runner.ts
onLifecycleEvent: (event: ChildPiLifecycleEvent) => {
  appendEvent(manifest.eventsPath, {
    type: `worker.${event.type}`,
    runId: manifest.runId,
    taskId: task.id,
    message: event.error ?? `Worker ${event.type}`,
    data: { pid: event.pid, exitCode: event.exitCode, error: event.error },
  });
}

Why a callback instead of direct logging: child-pi.ts has no access to manifest/eventsPath. The callback lets the caller (task-runner) decide how to log.

Scaffold Mode

When: executeWorkers = false or runtime.kind === 'scaffold'

Behavior: No child process spawned. runChildPi is never called. The task:

1. Writes the prompt to disk as an artifact
2. Immediately completes with a scaffold result artifact
3. No worker.spawned event — the agent appears and completes instantly

Display implication: In widget, scaffold agents appear and complete within 1 frame. This is normal behavior, not a bug.

Detection: runtimeKind === "child-process" triggers child spawning; "scaffold" or "live-session" skip it.

Child Args and Environment

Args built by buildPiWorkerArgs() (pi-args.ts)

pi
  --role <role>
  --task-id <taskId>
  --run-id <runId>
  --cwd <cwd>
  [--session]
  [--model <model>]
  [--thinking <level>]           # off/minimal/low/medium/high/xhigh
  [--max-depth <n>]              # from limits.maxTaskDepth (default 2)
  [--skill-dir <path>]           # one per skill directory
  [--transcript <path>]           # output transcript
  --task
  <task-prompt-text>

Environment variables

PI_EXECUTION_MODE=child           # marks child process context
PI_TEAMS_WORKER=1                # enables team-worker features
PI_CREW_PARENT_PID=<pid>         # parent process PID (added by child-pi.ts)
<redacted secrets>               # API keys filtered by sanitizeEnvSecrets()

GetPiSpawnCommand

Resolves the pi binary path and builds the final command/args. On Windows, uses pi.cmd or pi.exe.

Common Spawn Failures

Exit Code Mapping

Note: final_drain followed by exitCode=0 means the worker completed its output before being killed. The 0 exit code preserves the result.

PID Tracking

• PID recorded in manifest.async.pid at spawn (via checkpointTask)
• PID checked by hasStaleAsyncProcess() (process-status.ts) to detect dead processes
• PID used by killProcessPid() (child-pi.ts) for termination
• PID in childHardKillTimers Map for timer cleanup on exit

Enforcement — Child Pi Spawning Gate

Before debugging worker spawn issues, verify:

• [ ] Spawn error type identified (binary not found, permission denied, API key missing)
• [ ] PID tracking status confirmed (recorded in manifest, process alive/dead)
• [ ] Lifecycle event sequence matches expected pattern (spawned → output → exit)
• [ ] Timeout source determined (5-min response timeout, hard kill, final drain)
• [ ] Scaffold mode correctly identified (no worker.spawned event expected)

If ANY answer is NO → Stop. Re-examine events.jsonl and manifest before proceeding.

Anti-patterns

• Blocking on spawn: spawn() is async — never await it synchronously. Use the Promise-based API.
• Not handling exit: Always handle child.on("exit") and child.on("close"). Without handlers, zombie processes accumulate.
• Ignoring lifecycle events: Without onLifecycleEvent handling, worker crashes leave no traceable evidence.
• Not cleaning up timers: Hard-kill timers, response-timeout timers, and final-drain timers must be cleared on all exit paths.
• Passing secrets in args: Child args are visible in process list. Use env vars (with redaction) instead.
• Not handling `spawn_error`: Errors on spawn (binary not found, permission denied) must be caught and logged.

Source patterns

• src/runtime/child-pi.ts — runChildPi, ChildPiLifecycleEvent, activeChildProcesses, killProcessPid
• src/runtime/task-runner.ts — executeTask loop, onLifecycleEvent callback, runtimeKind
• src/runtime/pi-args.ts — buildPiWorkerArgs, applyThinkingSuffix
• src/runtime/runtime-resolver.ts — resolveCrewRuntime, isLiveSessionRuntimeAvailable, scaffold detection
• src/runtime/model-resolver.ts — model fallback chain
• src/utils/env-filter.ts — sanitizeEnvSecrets
• src/config/defaults.ts — responseTimeoutMs, finalDrainMs, hardKillMs

Verification

cd pi-crew
# Test scaffold mode (no worker spawn)
PI_TEAMS_MOCK_CHILD_PI=json-success node --experimental-strip-types -e "
import { runChildPi } from './src/runtime/child-pi.ts';
const r = await runChildPi({ cwd: '.', task: 'test', agent: {name:'test'}, mock: 'success' });
console.log('exitCode:', r.exitCode);
"
npx tsc --noEmit
node --experimental-strip-types --test test/unit/task-runner.test.ts test/unit/child-pi.test.ts 2>/dev/null || echo "Tests may need specific files"
npm test