contextjet-ai/awesome-llm-observability/compare-llm-models

Compare LLM models for your task

The best model on a public leaderboard is often not the best model for your task at your cost and latency. Public benchmarks narrow the field; your own eval set makes the call.

Use benchmarks to shortlist, not to decide

• General leaderboards (Chatbot Arena / LMArena for human preference, HELM for multi-metric, MMLU/GPQA for reasoning) tell you the rough tier a model is in. Use them to pick 2-4 candidates, not to declare a winner for your app.
• Watch for contamination and overfitting to popular benchmarks. A high MMLU score does not mean the model is good at your specific extraction/RAG/agent task.
• Task-relevant benchmarks beat general ones: if you do code, look at code evals; if RAG, look at long-context/faithfulness; if tools, look at agent/tool-use benchmarks.

Then evaluate the shortlist on YOUR eval set

This is the part that actually decides it. Run each candidate model through your own eval suite (see build-eval-dataset and add-llm-evals) and compare on the axes that matter:

Run it as an apples-to-apples eval: same inputs, same rubric, same judge. Report a small table, not a vibe.

Decide

• Pick the cheapest/fastest model that clears your quality bar, not the highest absolute quality. Most tasks do not need the frontier model.
• Consider routing: cheap model for easy calls, frontier model for hard ones (a gateway makes this easy).
• Re-run this when a provider ships a new model, but gate switches behind the eval, model upgrades sometimes regress your task even when the leaderboard goes up.

Verify

• You have a table comparing candidates on quality + cost + latency for your task.
• The decision is defensible from that table, not from "it is the newest."
• Switching is behind an eval gate so a regression is caught before shipping.

Anti-patterns

• Choosing by leaderboard rank alone (leaderboards are not your task).
• Switching to the newest model without re-running evals (silent regressions).
• Comparing quality while ignoring the 5x cost/latency difference.
• One-off manual spot-check instead of a repeatable eval.

Grounding

Human-preference evaluation: Chatbot Arena, Zheng et al. 2023 (arXiv:2306.05685). Multi-metric holistic evaluation: HELM, Liang et al. (arXiv:2211.09110). Benchmark harness: EleutherAI lm-evaluation-harness.