version: "1.0.1" name: terrarium-user-guide description: Guide for running agent tasks with Terrarium. Use this skill when the user wants to configure a job, choose agents, run tasks via CLI or Python API, understand results and metrics, or troubleshoot execution issues. Activate whenever someone mentions running tasks, configuring agents, job TOML files, trial results, evaluation scores, or anything related to using Terrarium.

Terrarium User Guide

Terrarium is a Python framework for building living environments and running LLM agents through multi-stage, multi-turn tasks. Task scripts (pure Python) orchestrate capabilities like email, databases, calendars, and workspaces, while collecting structured results and metrics.

Prerequisites

uv sync                  # install dependencies

For sandbox-based agents (claude_code, openclaw, codex, hermes), build the Docker image:

docker build -t terrarium/claude-code -f docker/claude-code.Dockerfile docker/
docker build -t terrarium/openclaw -f docker/openclaw.Dockerfile docker/
docker build -t terrarium/codex -f docker/codex.Dockerfile docker/
docker build -t terrarium/hermes -f docker/hermes.Dockerfile docker/

Set environment variables in .env at the project root:

ANTHROPIC_API_KEY=sk-...
ANTHROPIC_BASE_URL=https://api.anthropic.com  # optional
OPENAI_API_KEY=sk-...
OPENAI_BASE_URL=https://api.openai.com        # optional

Two Ways to Run

Option 1: CLI

terrarium run -c job.toml

The CLI shows a live execution log, per-trial results with pass/fail checks, and aggregated metrics.

Option 2: Python API

Run a batch job:

import asyncio
from terrarium import Job, JobConfig, AgentConfig
config = JobConfig(
    agents=[AgentConfig(name="claude_code")],
    datasets=["demo/demo_dataset"],
    n_attempts=1,
)
result = asyncio.run(Job(config).run())
for tr in result.trial_results:
    print(f"{tr.trial_name}: {tr.checker_result.score}")

Run a single trial:

from terrarium import Trial, TrialConfig, TaskConfig, AgentConfig
config = TrialConfig(
    task=TaskConfig(path="demo/demo_dataset/branch_and_loop"),
    agent=AgentConfig(name="claude_code"),
)
result = await Trial(config).run()
print(result.checker_result.score)

Job Configuration (TOML)

A complete job config:

job_name = "my_eval"
datasets = ["demo/demo_dataset", "benchmarks/tau2"]
tasks = ["path/to/standalone_task"]     # optional, for tasks not in a dataset
n_attempts = 3                          # repeat each agent x task combo
n_concurrent_trials = 4                 # parallel execution slots
 
[[agents]]
name = "claude_code"
model_name = "claude-sonnet-4-6"        # optional, overrides default model
 
[[agents]]
name = "openclaw"
model_name = "terrarium/anthropic/claude-sonnet-4.6"
 
[retry]
max_retries = 1
min_wait_sec = 1.0
max_wait_sec = 60.0
wait_multiplier = 2.0

Equivalent Python:

JobConfig(
    job_name="my_eval",
    agents=[
        AgentConfig(name="claude_code", model_name="claude-sonnet-4-6"),
        AgentConfig(name="openclaw", model_name="terrarium/anthropic/claude-sonnet-4.6"),
    ],
    datasets=["demo/demo_dataset", "benchmarks/tau2"],
    tasks=["path/to/standalone_task"],
    n_attempts=3,
    n_concurrent_trials=4,
    retry=RetryConfig(max_retries=1),
)

Key fields

Execution expansion

The job engine expands agents x (datasets + tasks) x n_attempts into individual trials. For example, 2 agents, 1 dataset with 3 tasks, 2 attempts = 12 trials.

Agents

Built-in agents

Agent configuration

claude_code — runs Claude Code CLI in Docker (terrarium/claude-code). Requires ANTHROPIC_API_KEY in .env.

[[agents]]
name = "claude_code"
model_name = "claude-sonnet-4-6"
kwargs = { max_turns = 50 }          # optional

openclaw — runs OpenClaw CLI in Docker (terrarium/openclaw). Requires a models config JSON file.

[[agents]]
name = "openclaw"
model_name = "terrarium/anthropic/claude-sonnet-4.6"
kwargs = { models_config_path = "./models.json" }

codex — runs OpenAI Codex CLI in Docker (terrarium/codex). Requires OPENAI_API_KEY in .env.

[[agents]]
name = "codex"
model_name = "gpt-5.4"                       # or "openai/gpt-5" via OpenRouter

hermes — runs NousResearch hermes-agent CLI in Docker (terrarium/hermes). Requires a model config YAML file.

[[agents]]
name = "hermes"
model_name = "claude-sonnet-4-6"
kwargs = { model_config_path = "./hermes_model.yaml" }

mini — in-process agent using litellm. No Docker needed. Supports tool registration and custom system prompts from task code.

[[agents]]
name = "mini"
model_name = "gpt-4o-mini"
kwargs = { temperature = 0.0, max_tool_calls = 100 }

Custom agent — import any class implementing BaseAgent:

[[agents]]
name = "my_agent"
import_path = "my_package.agent:MyAgent"

model_name is passed as the model kwarg to the agent constructor. All other kwargs are passed through directly.

Sandbox Providers

Capabilities and containerized agents run inside sandboxes provisioned by a SandboxProvider. The provider is selected per-job through sandbox_provider and propagates to every Trial the Job expands.

Built-in providers

Provider configuration

docker — default backend, talks to the local Docker daemon. No configuration needed for Docker Desktop / a local daemon.

[sandbox_provider]
name = "docker"
kwargs = { external_network = "shared-net" }  # optional

k8s — runs sandboxes as Pods in a Kubernetes cluster. Requires a kubeconfig the driver can use and an existing namespace.

[sandbox_provider]
name = "k8s"
kwargs = { namespace = "terrarium" }

Custom provider — implement any class extending SandboxProvider:

[sandbox_provider]
import_path = "my_package.providers:MyProvider"

kwargs is forwarded directly to the provider constructor.

Dataset Configuration

A dataset is a directory with dataset.toml and subdirectories each containing a task:

my_dataset/
  dataset.toml
  task_a/
    task.toml
    task.py
  task_b/
    task.toml
    task.py

The dataset.toml configures metrics:

[metadata]
name = "my_benchmark"
 
[metrics]
types = ["mean", "max", "pass@5"]   # default: ["mean"]

Tasks are discovered recursively — nested directories like domain/subdomain/task/ work.

Available metrics

Output Structure

Results are persisted as JSON:

outputs/
  {job_name}/
    config.json        # JobConfig
    result.json        # JobResult (all trials + aggregated stats)
    {trial_name}/
      config.json      # TrialConfig
      result.json      # TrialResult (trajectory, checks, timing)

Trial naming

Trial directories follow the pattern: {agent}__{model}__{source}__{task}[__attemptN]

For example: claude_code__claude-sonnet-4-6__demo__branch_and_loop__attempt0

Result viewer

Open demo/viewer.html in a browser and drag in any result.json (job-level or trial-level) to explore:

• Job overview with stats and score distribution
• Per-trial details with full trajectory (tool calls, thinking blocks)
• Timing analysis and checker results

Understanding Results

Scores

Each task defines checkers — boolean predicates that verify expected outcomes. The trial score is n_passed / n_total (0.0 to 1.0).

JobResult

A JobResult is the top-level output of a job run:

TrialResult

Each trial produces a TrialResult:

JobStats

Trial results are grouped by {agent}__{model}__{source}. Each group gets an AgentDatasetStats:

The top-level JobStats also has n_trials and n_errors summed across all groups.