agentic-in/elephant-agent/evaluation

Evaluation

Use this built-in skill when the user wants to compare prompts, models, retrieval setups, or agent behaviors with something stronger than anecdotes.

Core rules

• Define the task, dataset, metric, and baseline before running comparisons.
• Keep eval inputs and scoring rules stable enough to reproduce.
• Separate offline benchmarking from product acceptance criteria.
• Report both quantitative outcomes and obvious failure modes.

Default workflow

1. Identify the decision the evaluation should support.
2. Build or select the smallest credible dataset and metric set.
3. Run the baseline and candidate systems under the same conditions.
4. Summarize the tradeoffs, regressions, and confidence level.

Guardrails

• Do not declare wins from cherry-picked examples.
• Do not mix incomparable prompts, models, or retrieval settings in one score line.
• Do not skip failure-case inspection when the average looks good.