frontier-harness-eval/eval/frontierharness-eval

FrontierHarness Eval for a Third-Party Harness

Score a harness that is not in the published FrontierHarness v1.0 set, on the same tasks, runtime, and cost accounting, so the result can be placed next to the twelve baseline configurations in results/eval-data.json.

Workspace setup

npx skills add frontier-harness-eval/eval --skill frontierharness-eval installs this skill and its scripts, but not the repository's task definitions or baseline results. Prepare those files before following the evaluation workflow.

Set FH to the absolute path of the scripts/ directory beside the SKILL.md you loaded. Use that installed copy even after changing the working directory; the install location varies by agent and by project/global scope. For example, replace this path with the actual location:

FH="/absolute/path/to/frontierharness-eval/scripts"

If the current directory already contains benchmark.json, results/eval-data.json, and tasks/, use it as the benchmark workspace. An existing repository checkout or a workspace created by npx @frontierharness/eval both qualify. Otherwise reuse a known checkout, or create one with Git:

git clone https://github.com/frontier-harness-eval/eval.git frontierharness-eval
cd frontierharness-eval

If that destination already exists, inspect it and reuse it if complete, or choose a new directory; do not overwrite existing work. Run the commands below from the benchmark workspace so data paths and runs/ resolve there. Keep FH pointing to the installed skill's scripts, not the repository's unrelated top-level scripts/.

Prerequisites

Use the official Runta skills for Runta setup and command guidance. Install them in the same agent and scope as this skill if needed:

npx skills add https://runta.com/docs --skill runta-installer runta-cli

Read runta-installer when the Runta CLI needs installation or authentication setup; this evaluation requires the local CLI, and its provisioning script installs the runtime's SDK dependencies. Read runta-cli when checking runtime commands or troubleshooting Runta operations. If these skills are unavailable, use the linked Runta docs and the checks below. Keep this skill's benchmark requirements: prepare a clean runtime without an agent preset, freeze one golden checkpoint, and use a fresh restore per task with the specified resources.

Confirm all of these before touching a runtime:

runta --version                 # brew install runta-dev/tap/runta  (or npm i -g @runta/runta-cli)
runta checkpoint ls             # any API call proves the CLI is authenticated
jq --version && node --version  # jq for trial parsing, node >= 18 for the report scripts

Collect from the user before starting: harness name and version, the GitHub repo and commit for the harness under evaluation, and the task subset.

The model is not a variable; the provider is. FrontierHarness holds the model constant at Kimi K3 so the harness is the only thing that differs. Which provider serves it is up to the user, selected with --provider. The scripts warn if the model is not Kimi K3, and refuse an unknown provider name.

Ask which provider the user has a key for, and use --provider fireworks if they have no preference. A different provider must be documented and validated with a matched control; matching the model name alone does not prove equivalent serving behavior. Costs use the frozen benchmark token prices in reference.md, regardless of the provider's actual billing rates. Only change the model if the user explicitly wants a non-comparable run, and say so in the report.

Runtime quota and concurrency

This tenant's shared maximum is 32 vCPUs and 64 GB RAM across all runtimes, not per instance. Leave headroom below both limits; do not create enough instances to fill the quota.

• Before creating or restoring a runtime, inspect runta ps -a and account for

existing CPU and memory allocations plus the proposed runtime. Include build runtimes, trials retained for recovery, other evaluations, and unrelated workloads. If available capacity is unclear, resolve it before provisioning.

• Run trials sequentially by default. Only increase concurrency when the combined

allocations leave headroom under both limits, and recheck before each allocation. Preserve the benchmark's per-runtime resources; reduce concurrency instead of shrinking trial runtimes to fit.

• Remove this evaluation's build runtime after the checkpoint and checks succeed.

Remove completed trial runtimes only after their evidence is verified locally. Recover retained attempts before cleanup, and leave unrelated runtimes alone. If capacity is insufficient or RESOURCE_EXHAUSTED occurs, stop provisioning, wait for capacity or safely clean up this evaluation's finished runtimes, then recheck usage before retrying.

Workflow

Copy this checklist into your working notes and keep it updated:

- [ ] 1. Clean runtime created
- [ ] 2. Harness repo cloned at a pinned commit
- [ ] 3. Benchmark stack installed and frozen as a golden checkpoint
- [ ] 4. Trials run from fresh restores, trajectories saved
- [ ] 5. Comparison diagram generated
- [ ] 6. Report built and shared

Steps 1 through 3 are one command (provision-golden-checkpoint.sh), but read the per-step notes below because the fidelity rules live there.

1-3. Clean runtime, repo, and golden checkpoint

Authenticate the CLI with runta login, or set RUNTA_TOKEN if you prefer an explicit token. The provider key only has to be exported the first time: it is stored as a tenant secret, and the API never hands the value back, so a later re-cut of the checkpoint reuses the stored secret instead of demanding the plaintext again.

export FIREWORKS_API_KEY=...   # or the key for whichever --provider you pick
 
bash "$FH/provision-golden-checkpoint.sh" \
  --runtime fh-build \
  --checkpoint fh-golden-myharness-v1 \
  --harness my-harness \
  --provider fireworks \
  --repo https://github.com/acme/my-harness \
  --commit 9f2c1ab \
  --cpus 4 --memory 8192 --disk-size-gib 50 --keep-runtime \
  --install-script ./install-my-harness.sh

What the script does, and why each part matters:

• Clean runtime. runta run with no --agent preset, so no vendor harness is

pre-installed and nothing competes with the harness under test. Disk defaults to 50 GiB: building a harness from source plus a task image can overflow the 16 GiB Runtime Image default. Keep it at 50 GiB so every trial restores with the same capacity.

• Repo pinned by commit. The harness is cloned to /work/harness at --commit.

A branch name is not reproducible; always pin a SHA.

• Benchmark stack. Installs uv, Harbor 0.22.0 for Terminal-Bench, datacurve-pier==0.3.1 plus the deep-swe corpus at commit `435ee89ec2f2e2289f33b0da4f992f0b7b7266b9` (the published v1.1 label is not a Git tag; see the corpus caveat), and runta-sdk[harbor].
• Credential as a secret stub. The provider key named by --secret-name (defaulted

from --provider) is stored with runta secret set and injected by the egress proxy, so the real key never lands inside the runtime or inside a checkpoint. The script also allowlists the provider host plus the benchmark registry, source download hosts, and package registries required by verifiers. This runtime policy also permits agent package downloads, subject to Harbor/Pier isolation. The shared host list lives in scripts/providers.sh; see trial network access. Installation and image pulls happen before this restriction; egress setup errors stop preparation instead of silently falling back to a broken policy. Verify with runta exec fh-build -- sh -lc 'test "$FIREWORKS_API_KEY" = runta-secret-stub'.

• Small checkpoint, per-trial image pulls. Formal task images are pulled after

each fresh restore. Pre-pulling the full corpus has produced roughly 22 GiB of images and stalled checkpoint creation; --prepull-tasks remains an explicit opt-in with a warning. The only task ever executed before the checkpoint is terminal-bench-sample@2.0 with Harbor's oracle agent. Never execute a formal task before the checkpoint — that is warm-cache bias and it invalidates the comparison.

• Container runtime workaround. Restores the known Runta wrapper symlink

/usr/local/sbin/runc to /usr/bin/runc to avoid the reported injected-init hang during Pier verifier builds. Unknown binaries are left intact. Use --keep-runta-runc to disable the workaround; the manifest records the choice and resolved binary path. Harbor retains its explicit CA overlay.

• Manifest. /work/manifest.json records tool versions, the harness and corpus

commits, model, topology, image policy, and resolved runc path. Each trial's restore.log records its image pull output, including the registry digest. Keep both the manifest and logs to audit a later run.

• Golden checkpoint. runta checkpoint create freezes filesystem and process

state. The script waits up to --checkpoint-timeout (default 900 seconds) for ready and retains the build runtime on failure. Each trial restores from it, then pulls its own image before launching the harness.

Before moving on, confirm the checkpoint is ready:

runta checkpoint ls

The example uses --keep-runtime so the build runtime remains available for the stub and manifest checks. After confirming the checkpoint is ready, remove the build runtime with runta rm fh-build.

For an agent absent from Harbor or Pier, register it in both runner registries through --install-script and pass its registered name as --harness. Service-based harnesses are supported: set --harness-topology runtime-service or external-service, document the service version and resources, and ensure task state is reset between restores. The report discloses the topology difference from the container CLI baselines.

4. Run trials and save trajectories

Before a full sweep, preflight the harness's actual tool calls and both the agent's and verifier's dependency paths under the intended task isolation. A reachable model endpoint alone is insufficient. For offline source caches, native tool validation, and distinguishing verifier infrastructure failures from wrong answers, read isolated evaluation repairs. Preserve original tests, prompts, timeouts, and raw results; record repairs in a new run variant.

Each task gets its own fresh restore. The runtime is deleted only after complete evidence is verified locally and the trial record is written. Never reuse a runtime across tasks.

bash "$FH/run-trials.sh" \
  --checkpoint fh-golden-myharness-v1 \
  --harness my-harness \
  --provider fireworks \
  --run-id 2026-09-02-myharness \
  --out runs

Pass the same --provider here as at provisioning time. The checkpoint has that provider's key name baked in as a stub, so a mismatch leaves the harness without a credential.