netlify/axis/using-axis

Using AXIS

AXIS (Agent Experience Index Score) is a synthetic testing framework for AI agents. This skill is the operator's guide: how to invoke the CLI, where AXIS writes files, and how to read the scoring output.

For authoring scenarios and axis.config.json, see the configure-axis skill.

When to use this skill

Trigger phrases include "run AXIS", "run the ask suite", "what does --profile do", "compare runs", "explain this score", "which scenario regressed", "set a baseline", "where does AXIS put its reports", "what does this dimension mean".

Refer to the framework's output as the AXIS Result. The acronym is Agent Experience Index Score.

CLI commands

The binary is axis (or npx @netlify/axis without a global install).

axis init

Scaffold axis.config.json and a sample scenario, then install AXIS skills via npx skills.

axis run

Execute every scenario against every configured agent in isolated workspaces, score the results, and write a report.

--failed can be combined with --profile (though not with -s/-a), but retry a report under the profile that produced it: profiles can rename agents and narrow the suite, so pairs from a different suite match nothing and the run exits non-zero having discovered no jobs. A repeated pair is retried in full, not run by run, so the retry report holds a complete sample.

--runs must be odd (1, 3, 5, up to 19): an even sample has no middle run, so the median falls between two runs, and at runs: 2 the representative selection degenerates to "always run 1". Even values error out rather than rounding.

--runs trades cost for score stability. Each extra run is a full agent execution plus its four judge calls, so --runs 3 costs roughly 3x the tokens and 3x the wall clock. Use it for baselines and CI gates, not for every exploratory run. See "Repeated runs" below for how the reported number is chosen.

If the config defines profiles, axis run runs the default suite and axis run --profile <name> swaps in that overlay: usually a different agent matrix and a different slice of scenarios. An unknown profile name is an error, not a fallback to the default suite, so a typo in CI cannot quietly run the wrong suite. -s and -a narrow within the active suite; neither can reach a scenario the suite excluded. The active profile is recorded in the report manifest as profile.

axis reports

View past runs. Three call shapes:

• axis reports lists every report in .axis/reports/ (most recent first; -n to cap the list).
• axis reports <reportId|latest> shows the manifest for one run, scored summary per scenario.
• axis reports <reportId|latest> <scenarioKey> opens the per-run detail (transcript, audits, criteria). Use -a <agent> to filter when multiple agents ran the same scenario.

Flags: --json for machine output, --html to open the rendered report in a browser.

axis baseline

Manage saved baselines for regression detection. A baseline is a snapshot of one report's scenario scores under a name (default main).

• axis baseline set [name] saves the latest report as a baseline.
• axis baseline list lists saved baselines.
• axis baseline show [name] prints the saved entries.
• axis baseline compare [name] diffs the latest report against the baseline.
• axis baseline delete [name] removes a baseline.

axis run --compare-baseline combines a fresh run with an immediate diff in one step.

Expected directories

Everything AXIS writes lives under .axis/ at the config directory (typically the project root). Source-of-truth files (config, scenarios, skills) are NOT under .axis/.

project/
├── axis.config.{json,js,ts}        ← config (you author this)
├── scenarios/                      ← scenarios (you author these)
│   └── hello-world.json
├── skills/                         ← optional local skills referenced by config
│   └── <skill-name>/SKILL.md
├── adapters/                       ← optional custom adapter modules
│   └── <name>.{ts,js}
└── .axis/                          ← AXIS-managed, safe to gitignore
    ├── reports/
    │   └── <reportId>/             ← e.g. 2026-06-12-205816
    │       ├── report.json         ← run manifest (summary, results[])
    │       ├── report.html         ← rendered HTML report
    │       └── scenarios/
    │           └── <scenarioKey>/
    │               ├── <agent>.json           ← per-run detail (pair that ran once)
    │               ├── <agent>.raw.ndjson     ← raw transcript
    │               ├── <agent>.sparse-index.txt
    │               └── <agent>/
    │                   ├── artifacts/         ← files captured by scenario.artifacts globs
    │                   └── run-<i>/           ← only when the pair ran more than once
    │                       ├── result.json    ← that run's detail
    │                       ├── raw.ndjson
    │                       ├── sparse-index.txt
    │                       └── artifacts/
    ├── baselines/
    │   └── <name>.json             ← saved baselines (default name: main)
    ├── remotes/                    ← cloned scenarios from remote git URLs
    ├── repos/                      ← repos cloned by `copy` actions (per repo + ref)
    └── skills-cache/               ← cloned remote skills

When asked "where is X", the answer is almost always here. Do not search the project tree blindly; jump to the path.

Scoring framework

Every run is scored on four independent dimensions, each 0-100, combined into a weighted composite.

Override weights in axis.config.json under settings.scoring_weights.

Important distinctions

• Environment and Service evaluate execution reliability only. Did ls, cat, bash, fetch, MCP calls succeed cleanly? They do NOT judge whether the output was useful or task-fit.
• The Agent dimension is decision quality across every interaction. Every tool call is an agent choice, even a plain ls. The agent judge audits every interaction, not just calls tagged as agent-y.
• Speed is always heuristic (threshold buckets per category), never LLM-evaluated. Every other dimension uses an LLM judge.

Agent sub-dimensions

The agent dimension weights its own sub-dimensions:

Env and Service sub-dimensions are simpler: success 0.7, speed 0.3, rest 0.

Composite formula

axisScore = goal * w_goal + environment * w_env + service * w_svc + agent * w_agent

Calibration

All dimensions use log-normal CDF mapping with median 0.5 and sigma 0.4:

Practical consequence: even flawless runs cap around 95-99. Treat 95+ as a top-band result. A clean 100 is structurally near-impossible across all four dimensions.

Reading a report

.axis/reports/<reportId>/report.json has:

• version, reportId, timestamp, durationMs
• summary: { total, completed, failed, averageAxisScore }
• results[], one entry per (scenario, agent) pair

Each result entry contains:

• scenarioKey, scenarioName, agentName
• durationMs, exitCode, tokenUsage: { input, output, cacheReadInput }
• score.axisScore (0-100 composite)
• score.goalAchievement.{score, criteria[]} where each criterion has check, weight, score, rationale
• score.environment.{score, dimensions, audits[]}
• score.service.{score, dimensions, audits[]}
• score.agent.{score, dimensions, audits[]} (audits every interaction, not just agent-tagged)

Each dimensions object has { success, speed, weight, relevance, necessity } mapped to 0-100.

Repeated runs

A pair configured with runs > 1 still occupies one results[] entry. Its score, durationMs, tokenUsage, and file describe the representative run: the real run whose composite sits nearest the median composite. Because run counts are odd, that run's score _is_ the median, so the headline equals the median of the runs listed in runs[]. AXIS never averages composites into a headline, because an average has no transcript behind it to explain.

Three extra fields appear: