buildoak/fieldwork-skills/trueskill-rank

TrueSkill Rank

Rank any collection of text items by quality using TrueSkill + LLM-as-judge.

Setup

pip install trueskill

• Python 3.11+ required
• agent-mux for parallel dispatch (optional -- falls back to direct OpenAI API if OPENAI_API_KEY is set)
• Claude Code: copy this skill folder into .claude/skills/trueskill-rank/
• Codex CLI: append this SKILL.md content to your project's root AGENTS.md

For the full installation walkthrough (prerequisites, verification, API fallback), see references/installation-guide.md.

Staying Updated

This skill ships with an UPDATES.md changelog and UPDATE-GUIDE.md for your AI agent.

After installing, tell your agent: "Check UPDATES.md in the trueskill-rank skill for any new features or changes."

When updating, tell your agent: "Read UPDATE-GUIDE.md and apply the latest changes from UPDATES.md."

Follow UPDATE-GUIDE.md so customized local files are diffed before any overwrite.

Quick Start

PYTHON="python3"
CLI="~/.claude/skills/trueskill-rank/scripts/trueskill-rank.py"
 
# Full pipeline: prepare + dispatch + aggregate
$PYTHON $CLI run \
  --input items.json \
  --overlap 3 \
  --rubric ~/.claude/skills/trueskill-rank/rubrics/practitioner-signal.md \
  --output results.json
 
# Or step by step:
$PYTHON $CLI prepare --input items.json --overlap 3 \
  --rubric ~/.claude/skills/trueskill-rank/rubrics/practitioner-signal.md \
  --output-dir /tmp/ts-run/
# Dispatch is handled internally by trueskill-rank.py (no separate script needed)
$PYTHON $CLI aggregate --run-dir /tmp/ts-run/ --output results.json

Cost: each subset of 10 items produces C(10,2)=45 implicit pairwise comparisons. 100 items at overlap 3 = 30 subsets = 30 API calls = 1,350 implicit comparisons.

Decision Tree

Mode Selection

Overlap Selection

Rubric Selection

Input Format

{"items": [{"id": "item_001", "text": "...", "metadata": {...}}, ...]}

Source doesn't matter -- TG messages, HN posts, articles, papers, tweets. The id field is required, text is the content to rank, metadata is optional and passed through to output.

CLI Reference

prepare

Generate subsets and prompt files from input items.

$PYTHON $CLI prepare \
  --input items.json \         # Required. JSON with {"items": [...]}
  --mode batch \               # batch (default) or pairwise
  --overlap 3 \                # 2-4, controls statistical robustness
  --subset-size 10 \           # Items per subset (batch) or matchups per batch (pairwise)
  --rubric rubric.md \         # Required. Scoring criteria file
  --output-dir /tmp/ts-run/ \  # Where to write subsets.json and prompts/
  --seed 42 \                  # Random seed for reproducibility
  --text-cap 1500              # Max chars per item in prompts

Output: subsets.json + prompts/subset-NN.txt (batch) or prompts/batch-NN.txt (pairwise)

aggregate

Parse dispatch results and run TrueSkill rating.

$PYTHON $CLI aggregate \
  --run-dir /tmp/ts-run/ \     # Directory with subsets.json and results/
  --output results.json        # Final rankings JSON

run

Full pipeline: prepare + dispatch + aggregate.

$PYTHON $CLI run \
  --input items.json \
  --overlap 3 \
  --rubric rubric.md \
  --output results.json

Dispatch runs automatically via the built-in dispatch_workers() function. No external scripts required.

Output Format

{
  "rankings": [
    {"id": "item_001", "mu": 35.2, "sigma": 2.1, "conservative": 28.9,
     "rank": 1, "appearances": 3, "wins": 8, "losses": 2}
  ],
  "stats": {
    "total_items": 100, "subsets": 30, "results_parsed": 30,
    "parse_errors": 0, "coverage_gaps": 0, "mode": "batch",
    "overlap": 3, "rubric": "practitioner-signal"
  }
}

conservative = mu - 3*sigma. This is the ranking key. Penalizes items with few appearances (high uncertainty).

Dispatch

Built into trueskill-rank.py via dispatch_workers(). No external scripts.

Primary: agent-mux --engine codex --model gpt-5.3-codex-spark --reasoning low --effort low (free via GPT subscription). Resolves agent-mux via AGENT_MUX_PATH env var, then which agent-mux, then relative to skill directory. Runs 6 workers in parallel via concurrent.futures.ThreadPoolExecutor.

Fallback: If agent-mux not found, falls back to direct OpenAI API via urllib.request (stdlib, zero deps). Requires OPENAI_API_KEY env var. Uses gpt-4o-mini. Results written as {"success": true, "response": "..."} JSON.

Creating Custom Rubrics

Copy rubrics/example-template.md and fill in:

# Rubric Name
 
## Criteria (ordered by importance)
1.**CRITERION** -- Description
2.**CRITERION** -- Description
 
## Tiebreaker
How to break ties.
 
## Context
What kind of content this is for.

3-6 criteria recommended. Order matters -- most important first.

Anti-Patterns

Error Handling

Bundled Resources Index