moonlight-lupin/agent-skills/model-compare

Model Compare — Blind Side-by-Side Multi-Model Testing

Send one prompt to multiple models simultaneously, present responses anonymously, let the user pick a winner, then reveal which model is which.

Inspired by the Compare feature in PewDiePie's Odysseus project, adapted for Hermes's multi-provider architecture (OpenRouter, NVIDIA, Ollama Cloud, any OpenAI-compatible endpoint).

When to use

• "Compare these models on..."
• "Which model is better for [task]?"
• "A/B test [model1] vs [model2]"
• "Blind comparison of..."
• "Test how different models handle this prompt"
• User wants to evaluate models before committing to one for a workflow
• Prompt engineering — seeing how different models interpret instructions

When NOT to use

• Benchmarking (MMLU, GSM8K, etc.) → use evaluating-llms-harness skill
• Cost analysis → just check provider pricing pages
• Single model test → just switch model and ask directly
• Multi-source research synthesis → use deep-research skill (iterative research loop, not model comparison)

Architecture

Four comparison modes, all driven by scripts/compare.py:

User prompt + model list
  → Step 1: Resolve models to provider endpoints (free providers first)
  → Step 2: Send prompt to all models in parallel
     ├─ simple/coding/review: one-shot chat completion
     └─ tools: multi-turn loop (Think→Search→Extract→Execute→Synthesize→Stop, max 10 turns by default, per-test configurable)
  → Step 3: Quality check responses (handle errors/empty)
  → Step 4: Present anonymously (shuffle + label A/B/C/D)
  → Step 5: Efficiency table (tokens in/out, turns, tool calls — auto for tools mode)
  → Step 6: User votes OR judge model evaluates
  → Step 7: Reveal identities + show mapping
  → Step 8 (optional): Save to JSON file

Script: scripts/compare.py

The primary interface — a standalone CLI tool (no pip dependencies, pure stdlib + urllib). See references/provider-tool-support.md for which models support tool calling. For empirical model behavior findings from 16 head-to-head tests (GLM 5.2, MiniMax M3, HY3, Laguna), see references/model-behavior.md. (Running the test suite under tests/ needs pytest — see requirements-dev.txt; the skill itself needs nothing installed.)

Tool-mode environment dependency: --mode tools runs real web_search. That needs either SEARXNG_URL set to a SearXNG instance or the `ddgs` CLI available on PATH (the fallback). Neither is a Python import dependency, but one of them must be present for live search; without both, web_search returns an error result. The other three modes (simple, coding, review) need neither.

# Simple 2-model blind comparison
python3 scripts/compare.py --prompt "Explain X" --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5"
 
# Reasoning-effort A/B: SAME model, two effort levels (native syntax)
# Registers a virtual provider per arm; judge can also take an effort suffix.
python3 scripts/compare.py --mode review --test H7 \
  --models "ollama-cloud:@medium:glm-5.3-flash" "ollama-cloud:@high:glm-5.3-flash" \
  --judge "ollama-cloud:@medium:glm-5.3" --reveal
 
# Tool calling with test bank prompt A + judge + reveal
python3 scripts/compare.py --mode tools --test A --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5" --judge "ollama-cloud:glm-5.2" --reveal
 
# Coding test J with efficiency table
python3 scripts/compare.py --mode coding --test J --models "ollama-cloud:glm-5.2" "ollama-cloud:qwen3-coder:480b" --efficiency
 
# Code review test O with judge
python3 scripts/compare.py --mode review --test O --models "ollama-cloud:glm-5.2" "ollama-cloud:kimi-k2.5" --judge "ollama-cloud:glm-5.2"
 
# List available tests
python3 scripts/compare.py --list-tests
 
# List providers
python3 scripts/compare.py --list-providers

Key flags: --prompt, --models, --mode, --test (test bank ID), --judge, --efficiency, --reveal, --output, --timeout, --list-providers, --list-models.

Reasoning-effort A/B (:@effort suffix, added 2026-08-29)

Model specs and --judge accept an optional effort marker provider:@effort:model_id, e.g. ollama-cloud:@medium:glm-5.3-flash. Each unique (provider, effort) pair registers a virtual provider (ollama-cloud@medium) that clones the base provider's config — same endpoint, same env key, inherited max_tokens — and injects reasoning_effort into the payload. This enables comparing the SAME model at two reasoning levels in one blind run; the reveal labels the arms provider@effort. Findings from the 2026-08-29 glm-5.3-flash A/B (Codex-judged, 8 tests): effort→thinking-token count is non-monotonic; high was ~3x faster median but non-converged on a 12-turn tool-chain test that medium completed; medium won on judged quality overall (8.6 vs 7.8 avg). See ~/.hermes/data/glm53flash_effort_ab/ab_results.jsonl for full traces.

`--max-turns` CLI override (added 2026-07-17): Pass --max-turns N to override the test bank's max_turns field at runtime without editing source. If not passed, the test bank default is used. This was added because killing a background run to patch TEST_BANK["A"]["max_turns"] in source code is fragile and disrupts parallel execution.

Available Providers

Three providers are wired in (the canonical config is PROVIDERS in scripts/compare.py):

Model inventories live elsewhere, on purpose. Exact model counts and IDs

drift constantly, so they are deliberately kept out of this doc. For the live

list run python3 scripts/compare.py --list-models <provider> (needs that

provider's key); references/providers.json holds a curated, count-free

snapshot with representative model IDs per provider.

Cost-aware provider routing (mandatory)

When the user does not specify a provider, route to free providers first:

1. Ollama Cloud — free (GLM, Qwen, Kimi, Gemini, Gemma)
2. NVIDIA — free (Yi, Llama, Nemotron)
3. OpenRouter — paid (per-token cost). Only use when:

• The user explicitly requests an OpenRouter-only model (e.g. Claude, GPT-4o)
• The user explicitly says to use OpenRouter
• The free providers don't have a suitable model for the task

Before any OpenRouter call, confirm with the user:

"This comparison will use OpenRouter which has per-token costs. Estimated cost: ~$0.01–0.05 per model per call (varies by model). Proceed?"

Only proceed after explicit confirmation. When in doubt, default to free providers.

To call a model, POST to the provider's /v1/chat/completions endpoint with:

{
  "model": "<model_id>",
  "messages": [{"role": "user", "content": "<prompt>"}],
  "max_tokens": 8192,
  "temperature": 0.7
}

Header: Authorization: Bearer <API_KEY>

max_tokens is set to 8192 in both call_model_simple and

call_model_with_tools. The previous default of 4096 caused truncated

code output on coding test J (LRU cache) — the model generated so much

code it hit the 4096 limit before finishing the decorator section.

If a model still truncates, increase the value in the payload dict.

Step 1 — Resolve Models

Determine which models to compare. The user may specify:

• Explicit model names: "compare claude-sonnet-4.6 vs gpt-4o vs gemini-2.5-flash"
• Provider + model: "compare OpenRouter claude-sonnet-4.6 vs Ollama glm-5.2"
• Task-based: "which model is best for coding?" → suggest 2-4 candidates
• All from a provider: "test 3 OpenRouter models" → pick diverse ones

Model resolution

Map the user's model names to a provider:model_id spec. Free providers (Ollama Cloud, NVIDIA) are preferred. OpenRouter is only used for models not on a free provider (e.g. Claude, GPT) or when the user explicitly asks for it.

The rules, not a hardcoded catalogue (which would rot):

• Free, leave the prefix off or name an Ollama/NVIDIA model → resolve on a

free provider, e.g. ollama-cloud:glm-5.2, ollama-cloud:qwen3-coder:480b, nvidia:meta/llama-3.3-70b-instruct.

• A proprietary model (Claude, GPT, Gemini-Pro, DeepSeek-R1, …) → only

OpenRouter carries it, e.g. openrouter:anthropic/claude-sonnet-4.6, openrouter:openai/gpt-4o — paid, confirm first (see routing rule above).

• Unsure of the exact ID? Run `python3 scripts/compare.py --list-models

<provider> for the live list, or see references/providers.json` for a curated set of representative IDs per provider. Don't hand-maintain a model table here.

When user doesn't specify models:

1. Check if the task type maps to free models (e.g. coding → qwen3-coder, glm-5.2; general → glm-5.2, kimi-k2.5)
2. Suggest 2-4 free models from Ollama Cloud and NVIDIA
3. Only suggest OpenRouter models if the user asks for premium models (Claude, GPT-4o) or the free providers lack suitable options
4. If suggesting any paid models, flag the cost before running

If unsure which provider has a model, check with:

curl -s https://openrouter.ai/api/v1/models -H "Authorization: Bearer $OPENROUTER_API_KEY" | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data'] if 'KEYWORD' in m['id'].lower()]"

Rules:

• 2-4 models per comparison (more = unwieldy in chat)
• If user doesn't specify, suggest a diverse set (different providers/sizes)
• Always confirm the model list with the user before running

Step 2 — Send Prompt to All Models

Send the prompt to all models in parallel. The script handles this automatically via concurrent.futures.ThreadPoolExecutor — no need for delegate_task or manual parallelism.

Simple / coding / review modes

One-shot chat completion per model. All calls fired concurrently by the script.

Tool calling mode

Multi-turn loop per model (also concurrent across models):

Turn 1: Send prompt + tool definitions → Model returns tool_call(s)
Turn 2: Execute real tool → inject result → Model returns tool_call(s) or answer
Turn 3: ... until final answer or max turns (10 by default, configurable per-test via `max_turns` field in TEST_BANK)

Real tool execution — the script executes web_search and web_extract for real:

• web_search → SearXNG (self-hosted, via SEARXNG_URL) with DDGS fallback
• web_extract → direct HTTP fetch with HTML-to-text conversion (3000 chars/page)

No mock tools, no fake results. Models must formulate good queries, pick the right URLs, and synthesize from real content. A model that generates a bad search query gets bad results and must recover.

Tool definitions passed via the OpenAI tools parameter:

• web_search(query, limit) — returns titles, URLs, snippets
• web_extract(urls) — returns page content as text