xingwudao/open-xquant/compare-strategy-versions

Phase Path Containment Preflight

Before any phase artifact read, write, directory creation, command, or handoff, run this preflight completely and block on any failure:

1. Read .open-xquant/workspace.yaml. Resolve version_root from

paths.versions_dir, using versions only when that key is absent. Require version_root to be a safe workspace-relative path: reject an absolute path or any .. path segment, resolve it canonically with symlinks, and require the result to stay inside the workspace.

1. Read current.json. For normal phase work, set expected_version_id to

current.json.active_version; it must exist. Only a contract that explicitly owns cross-version inspection may instead set expected_version_id from the referenced version id for that historical read. This comparator owns that exception only while validating each candidate; it never permits active-version work to consume another version.

1. Set the intended version directory to

<version_root>/<expected_version_id>/ and resolve it canonically. Require the intended version directory to remain inside the canonical version root and workspace; otherwise treat it as a symlink escape. Read version_manifest.json only from that exact directory. The manifest version_id must equal expected_version_id.

1. Before using each required phase_paths value, require a non-empty

workspace-relative string. Reject an absolute path and any .. path segment. Resolve <workspace>/<phase_path> canonically, including existing symlink ancestors even when the leaf will be created, and require the target to be the intended version directory or a descendant of it. A symlink escape outside that directory is invalid.

1. On any identity or path failure, stop before phase artifact reads, writes,

directory creation, commands, or handoffs. Do not normalize an unsafe path into acceptance and do not fall back to a default phase path.

Block examples when expected_version_id is v001 include strategy_store/v001/../v002/04_spec_build, strategy_store/v002/04_spec_build, /tmp/04_spec_build, and strategy_store/v001/escape/04_spec_build when escape is a symlink whose target is outside the intended version directory. An allowed custom nested phase path is strategy_store/v001/custom/phases/04_spec_build when its canonical target remains under the intended version directory.

For a new-version bootstrap, only manage-strategy-version may proceed before the new manifest exists or the new id becomes active. It must apply the same workspace-relative, traversal, canonical-containment, and symlink checks to every constructed phase path before directory creation, then write a matching manifest before publishing current.json last.

Strategy Version Comparator

Round 26: comparison schema 3; the exact closed request fields are schema_version, mode, selection_id, selection_request_id, selection_policy, candidate_set, and comparison_population. Historical refresh is write -> review -> lineage -> prepare new selection -> comparison -> resume with fresh selection, candidate-set hash, and comparison ID.

This skill extends compare-experiments for version-governed workspaces. It compares completed run packages without writing into either run directory.

Workspace Path Resolution

Read .open-xquant/workspace.yaml before reading or writing comparison artifacts. Resolve each key independently from paths: use paths.experiment_registry, paths.governance_dir, paths.comparisons_dir, paths.comparison_registry, and paths.final_dir. Use experiments.jsonl, governance, comparisons, comparisons/comparisons.jsonl, or final, respectively, only when that key is absent; never derive one key from another configured key. Each configured value must be a safe workspace-relative path. Reject absolute paths, traversal outside the workspace, and symlink escapes whose resolved target leaves the workspace.

Call the resolved paths <experiment_registry>, <governance_dir>, <comparisons_dir>, <comparison_registry>, and <final_dir> below.

Candidate Run Containment

Treat every registry row as an untrusted reference. For each candidate:

1. Set expected_version_id to candidate.version_id under the explicit

cross-version exception above. Load only <version_root>/<candidate.version_id>/version_manifest.json; the manifest version_id must equal candidate.version_id.

1. Read that manifest's exact phase_paths.09_backtests. Validate its safe

workspace-relative form and canonical containment under the candidate's intended version directory. Do not fall back to a default backtest path.

1. Require candidate.run_path to be a safe workspace-relative path with no

.. segment. Resolve it canonically, including symlinks, and prove it is exactly one direct run directory inside that backtest phase:

resolved_run.parent == resolved_backtest_phase
resolved_run.name == candidate.run_id

The resolved path must be a directory. Reject a stale default such as legacy_root/v001/09_backtests/run_001 when the configured root is research_store; reject a cross-version path such as research_store/v002/09_backtests/run_001 for candidate v001; reject a nested path such as research_store/v001/09_backtests/nested/run_001; and reject research_store/v001/09_backtests/escape/run_001 when escape is a symlink outside the candidate's backtest phase. Registry identity and lineage audit eligibility do not replace these checks.

When the comparator is invoked for final-selection evidence, the handoff must include the exact hash-bound schema-version-2 candidate_set.json reference and its exact schema-version-2 selection-policy reference. Re-read and validate that candidate set before candidate evidence consumption and immediately before comparison-manifest publication. Comparison population must be a subset of the hash-bound candidate set and preserve candidate order; never discover, substitute, or append a registry candidate outside that set. The final selector applies the complete two-candidate equality or larger-set union-and-connected-graph coverage rule across all comparison references.

Final-Selection Comparison Handoff

The normal final-selection path is prepare_selection -> candidate_set_ready -> build_selection_comparison -> comparison_ready -> resume_selection. This is one coordinated request and uses the same selection_id, same exact selection-policy reference, and same exact candidate-set reference throughout. The exact schema-version-2 request below is retained for historical recognition; current production uses the Round 25 schema-version-3 contract:

{
  "schema_version": 2,
  "mode": "build_selection_comparison",
  "selection_id": "selection_20260712_180000",
  "selection_policy": {
    "path": "<final_dir>/selection_20260712_180000/selection_policy.json",
    "sha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000"
  },
  "candidate_set": {
    "path": "<final_dir>/selection_20260712_180000/candidate_set.json",
    "sha256": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
  },
  "comparison_population": [
    {
      "version_id": "v001",
      "run_id": "runA"
    },
    {
      "version_id": "v002",
      "run_id": "run_20260712_173012"
    }
  ]
}

{
  "schema_version": 3,
  "mode": "build_selection_comparison",
  "selection_id": "selection_20260712_190000",
  "selection_request_id": "selection-request-20260712-1",
  "selection_policy": {"path": "<final_dir>/selection_20260712_190000/selection_policy.json", "sha256": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},
  "candidate_set": {"path": "<final_dir>/selection_20260712_190000/candidate_set.json", "sha256": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"},
  "comparison_population": [{"version_id": "v001", "run_id": "runA"}, {"version_id": "v002", "run_id": "runB"}]
}

Resolve the candidate-set path as the exact direct regular file under <final_dir>/<selection_id>/, re-read its exact bytes, recompute its full hash, validate its complete schema and transitive evidence, and require comparison_population to equal its ordered projection. Do not accept a path-only ref, allocate a selection id, mutate the candidate set, or discover a replacement population. On successful manifest publication return exactly:

Require at least two unique candidates. Re-read the schema-version-2 policy artifact and require its exact selection id and {path, sha256} reference to equal the request and candidate-set binding. Reject stale or cross-selection policy.

{
  "schema_version": 2,
  "status": "comparison_ready",
  "selection_id": "selection_20260712_180000",
  "selection_policy": {
    "path": "<final_dir>/selection_20260712_180000/selection_policy.json",
    "sha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000"
  },
  "candidate_set": {
    "path": "<final_dir>/selection_20260712_180000/candidate_set.json",
    "sha256": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
  },
  "comparison_ref": {
    "path": "<comparisons_dir>/selection_20260712_180000/cmp_v001_runA_vs_v002_runB/comparison_manifest.json",
    "sha256": "sha256:2222222222222222222222222222222222222222222222222222222222222222"
  },
  "blocking_findings": []
}

Every historical schema-version-2 comparator result has exactly schema_version, status, selection_id, selection_policy, candidate_set, comparison_ref, and blocking_findings. status is comparison_ready, blocked, or fail; comparison_ref is null for blocked/fail and blocking_findings is then non-empty. The coordinator collects one or more ready refs and invokes resume_selection without a new user request. Comparator failure never changes the selection id, candidate set, decision, or prior pointer.

Immutable Selection Comparison Output

For build_selection_comparison, the only output root is <comparisons_dir>/<selection_id>/<comparison_id>/, an immutable selection-scoped directory. A normal manifest path is <comparisons_dir>/selection_20260712_180000/cmp_v001_runA_vs_v002_runB/comparison_manifest.json. Create the directory exclusively before any output write and reject an existing output directory. Never overwrite, delete, merge, or repair comparison evidence, especially evidence reachable from a prior current_final.json. Hash exact final bytes into the schema-version-2 comparison manifest. A remediable retry uses a fresh comparison_id under the same selection_id and keeps the same policy/candidate-set binding; restart_selection allocates a new selection and comparison scope.

Comparison Types

Within-version comparison:

• same version_id
• same confirmed SPEC unless the comparison is explicitly a robustness variant
• use oxq backtest compare-runs as a strict comparability gate
• failures usually block winner-style conclusions

Cross-version comparison:

• different version_id values
• different spec_hash is expected, not automatically fatal
• produce a spec diff and explain likely impact
• do not name a winner from metrics alone

Inputs

• Two or more candidate run packages.
• <experiment_registry> with version_id, run_id, run_path, and decision

fields.

• Optional existing <comparisons_dir>/<comparison_id>/ artifacts only for a

standalone, non-selection comparison. build_selection_comparison rejects every existing output directory instead of treating it as optional input.

• <governance_dir>/lineage_audit_*.json for candidate lineage eligibility.
• Optional user-confirmed comparison scope.

Workflow