gonka-ai/gonka/mlnode-validate

MLNode PoC v2 Validation

The caller decides what to test. This skill never picks a default MLNode and never picks a default model. Both must be passed explicitly.

This file is the contract. The skill is implemented by two scripts under mlnode/packages/benchmarks/scripts/poc_validation/. Do not read or edit those scripts as part of running this skill -- run validate.py and report what it prints.

Two file types -- keep them straight

The skill works with two distinct artifact kinds. Do not conflate:

• Golden reference (committed in repo): one JSON per supported

model under mlnode/packages/benchmarks/scripts/poc_validation/artifacts/<sanitized model>.json. Pre-computed by the project: contains the honest PoC vectors, canonical PoC params, consensus-default additional_args, and the consensus-default stat_test block. Read-only at validate time.

• Per-run report (produced on the caller's machine, NOT

committed): one experiment directory per validate.py invocation under mlnode/packages/benchmarks/data/experiments/<exp_name>_<ts>/. Layout matches mlnode/packages/benchmarks/scripts/README.md -- one experiment = one server + one model deployment. Each directory contains validate_config.json (inputs), validate_report.json (full structured result), and validate_report.txt (short text summary).

When this skill says "reference", it means the golden file. When it says "report", it means the per-run output.

Available golden references

The repo ships these references under mlnode/packages/benchmarks/scripts/poc_validation/artifacts/. Each one is keyed by model id; the auto-lookup <sanitized model>.json picks one filename per model. Variants beyond the default require an explicit --reference <path>.

For Qwen3-235B the same model id has two references, and they exercise different code paths (tp-size + MoE backend). When validating qwen235b, run validate.py twice -- once with the default lookup, once with the deepgemm variant -- and report both verdicts. Example:

# Run 1: default 32-nonce reference (FlashInfer, tp=4)
python3 mlnode/packages/benchmarks/scripts/poc_validation/validate.py \
    --mlnode-url "$MLNODE_URL" \
    --model     Qwen/Qwen3-235B-A22B-Instruct-2507-FP8
 
# Run 2: extended 2000-nonce reference (DeepGEMM, tp=2)
python3 mlnode/packages/benchmarks/scripts/poc_validation/validate.py \
    --mlnode-url "$MLNODE_URL" \
    --model     Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
    --reference mlnode/packages/benchmarks/scripts/poc_validation/artifacts/qwen-qwen3-235b-a22b-instruct-2507-fp8-deepgemm.json

Each run lands in its own experiment directory, so reports do not collide. If the caller is short on time and explicitly asks for a quick check, the default-lookup run alone is acceptable; otherwise both should be run.

Required inputs

The caller MUST supply both:

• MLNODE_URL -- base URL of the MLNode under test (e.g.

http://1.2.3.4:8080). No default.

• MODEL -- target HuggingFace model id, exactly as the MLNode

would receive it (e.g. Qwen/Qwen3-235B-A22B-Instruct-2507-FP8, Qwen/Qwen3-0.6B). The full org/repo form. No default.

If either is missing, ask the caller for it before running anything. A short reference like "qwen 0.6B" is NOT enough -- ask for the full HF id. Do not pick a model from the available artifacts on the caller's behalf.

Where every other parameter comes from

Beyond MLNODE_URL and MODEL, the caller does not need to pass anything. Every other parameter has a documented source:

• seq_len, k_dim, block_hash, public_key, node_id,

node_count -- all read from the golden reference for MODEL. The reference pins the exact PoC inputs its vectors were computed under.

• vLLM additional_args -- the consensus-default deploy args for

this model are saved in the reference's additional_args field and used as-is. The caller can pass extra flags only if they ask for them: --tp-size and --max-model-len add-or-update the reference baseline (the same flag's existing value is replaced, otherwise appended); --extra-arg <token> appends arbitrary tokens.

• stat_test.dist_threshold / p_mismatch / fraud_threshold --

resolved per-key with provenance: server default (0.02 / 0.001 / 0.01) is the floor; the reference's stat_test block (which mirrors the on-chain consensus params for this model) overrides; CLI flags (--threshold, --p-mismatch, --fraud-threshold) are the top. All three values are always sent to the server and shown in the report with their source.

Optional inputs

The caller can override deploy / validation / sampling defaults. Only pass these flags when the caller asks for them:

• Stat-test parameters. These three flags fill the stat_test block

the MLNode runs (per-nonce L2 mismatch test + binomial fraud test). --threshold IS the dist_threshold field of that block, just named more concisely on the CLI. Defaults come from the artifact; any key not provided is filled by the MLNode server defaults.

• --threshold <float> (e.g. 0.2) -- the dist_threshold field.

L2 distance above which a single nonce counts as a per-nonce mismatch. Default: artifact's dist_threshold.

• --p-mismatch <float> (e.g. 0.001) -- per-nonce probability of

a benign mismatch under honest computation. Default: artifact's stat_test.p_mismatch if present, else server default.

• --fraud-threshold <float> (e.g. 0.01) -- probability_honest

cutoff at which the server flags fraud. Default: artifact's stat_test.fraud_threshold if present, else server default.

• Deploy overrides (only pass when the caller explicitly asks; the

artifact's additional_args already encode the consensus-default deploy config for this model):

• --dtype <auto|float16|bfloat16|fp8> -- vLLM dtype.
• --tp-size <int> -- add-or-update --tensor-parallel-size.
• --max-model-len <int> -- add-or-update --max-model-len.
• --extra-arg <token> -- pass once per token to append additional

vLLM args. The caller owns avoiding conflicts with the artifact baseline.

• Phase skips. By default all four phases run; each flag drops one phase:
• --skip-download -- the model is known to be cached already.
• --skip-deploy -- vLLM is known to already be serving MODEL.
• --skip-throughput -- skip the throughput measurement (faster runs).
• --skip-validate -- skip the vector-validation step (deploy / measure only).
• Sampling: --warmup-seconds <int>, --measure-seconds <int>,

--sample-interval <int>, --batch-size <int>.

• Timeouts (seconds): --download-timeout <float>,

--deploy-timeout <float>, --validation-timeout <float>.

• Output location:
• --exp-dir <path> -- write into this experiment directory.
• --exp-name <name> -- prefix used to auto-create the experiment

dir under data/experiments/<exp-name>_<ts>/ (default: mlnode_validate_<sanitized model>).

• --reference <path> -- override the golden reference path (for

testing custom references). Default: looked up from MODEL. --artifact is accepted as a legacy alias.

How to run

Bare minimum (recommended starting form):

python3 mlnode/packages/benchmarks/scripts/poc_validation/validate.py \
    --mlnode-url "$MLNODE_URL" \
    --model     "$MODEL"

Add overrides only when the caller asks for them. The deploy config the caller sees is the artifact's additional_args plus any of --dtype, --tp-size, --max-model-len, --extra-arg they passed; the caller does not need to know what's in the artifact's baseline.

What the script does, in order

The script prints [i/4] headers (1-indexed) as it progresses. The agent should relay these headers to the caller while the run is in flight so the caller can see which phase is slow.

1. [1/4] download -- ensures the requested HF repo is cached on the

MLNode.

• POST /api/v1/models/status {hf_repo} -- returns one of

DOWNLOADED | DOWNLOADING | NOT_FOUND | PARTIAL.

• If not DOWNLOADED, POST /api/v1/models/download {hf_repo} to

start the download, then poll /models/status until DOWNLOADED.

• Skipped with --skip-download. Hard-fails if the status flips

back to NOT_FOUND mid-poll, or the timeout expires.

1. [2/4] deploy -- starts vLLM if it is not already running.

• POST /api/v1/inference/up/async {model, dtype, additional_args}.
• Polls GET /api/v1/inference/up/status until is_running == true

or status becomes failed/cancelled. Server-side elapsed_seconds is printed on each poll so the caller sees progress.

• Skipped with --skip-deploy; if so, the script verifies vLLM is

already running and aborts otherwise.

1. [3/4] throughput -- measures full-system PoC throughput.

• POST /api/v1/inference/pow/init/generate {block_hash, ..., params}

(the artifact's params); the proxy fans out to every healthy vLLM replica with a different group_id so they process disjoint nonces. The first response reports backends and n_groups.

• After --warmup-seconds, samples

GET /api/v1/inference/pow/status every --sample-interval for --measure-seconds. Reports per-replica nonces_per_second and the sum across replicas (the system-level throughput for this model). Then POST /api/v1/inference/pow/stop.

• Skipped with --skip-throughput.

1. [4/4] validate -- verifies pre-computed honest vectors.

• POST /api/v1/inference/pow/generate with wait=true,

nonces=[...], validation.artifacts=<artifact>, and the full stat_test block (dist_threshold, p_mismatch, fraud_threshold). The MLNode recomputes the same nonces, compares them to the supplied vectors under L2 (per-nonce mismatch test), then runs a binomial fraud test using p_mismatch and fraud_threshold. Returns {n_total, n_mismatch, mismatch_nonces, p_value, fraud_detected}.

• The pass criterion is the binomial test alone: `fraud_detected ==

false. A non-zero n_mismatch with fraud_detected == false means the MLNode produced a few mismatches but they fall within the statistical tolerance the test allows; this is still a PASS, and the script labels it PASS (with mismatches within stat-test tolerance)`.

• Skipped with --skip-validate.

After the four phases, the script writes three files into the experiment directory (mlnode/packages/benchmarks/data/experiments/<exp_name>_<ts>/, matching the layout in mlnode/packages/benchmarks/scripts/README.md):

• validate_config.json -- the resolved inputs only (mlnode URL,

model, reference path + meta, deploy config, PoC params, stat_test with provenance, raw CLI args). Lets a future reader reproduce the run without re-deriving any defaults.

• validate_report.json -- the full structured report (config +

per-phase results + verdict). This is the audit trail.

• validate_report.txt -- short human-readable summary; first line

after the banner is verdict: <PASS|FAIL|...> for at-a-glance reading.

How to report back to the caller