thinkfleetai/memmesh/benchmark

benchmark

Put numbers on the comparison. MemMesh ships a real benchmark harness that runs the public LOCOMO dataset end-to-end against competing systems.

What it compares

Systems: thinkfleet (MemMesh) vs full_context vs naive_rag, and — with keys — Mem0 / Zep. Metrics: answer accuracy (rubric-scored), tokens consumed, latency, and cost per conversation.

Run it

The harness lives in the engine repo at crates/eval/competitive/:

cd crates/eval/competitive
python bench.py --systems thinkfleet,mem0,full_context --dataset locomo
# results land in results/

(Set the competitors' API keys via env for a head-to-head; without them you still get MemMesh vs full-context vs naive-RAG.)

Report honestly

MemMesh's positioning is calibration over raw accuracy — "80% means 80%" and honest abstention beat a slightly higher accuracy with overconfident wrong answers. So report the full picture:

• accuracy and calibration error,
• tokens / latency / cost (MemMesh's retrieval is far cheaper than full-context),
• where MemMesh abstained vs. where a competitor answered confidently and wrong.

Don't cherry-pick a single accuracy number. If a competitor wins on one axis, say so, and show where MemMesh's calibration/cost advantage pays off.

Cost gating

Prove the win on the cheap tiers (LOCOMO, BEAM-100K) before spending on BEAM-1M/10M — a single 10M-token conversation is expensive. Escalate tiers only once the cheaper tier shows a clear, defensible lead.