yuyangsunshine/bioprobench/evaluate-protocol-outputs

Evaluate Protocol Outputs

Normalize prediction data, run deterministic metrics, and report scored and failed samples. Do not assume access to a particular repository.

Workflow

1. Select one task from the table below.

1. Read references/input-formats.md when the input does not already use prediction, reference, confidence, and items.
2. Run the evaluator from this skill directory:

python scripts/evaluate.py --task step-ordering --input /absolute/path/predictions.json

1. Map custom fields instead of rewriting the source data:

python scripts/evaluate.py \
  --task question-answering \
  --input /absolute/path/predictions.json \
  --prediction-field model_output \
  --reference-field gold_answer \
  --confidence-field probability

1. Add --output metrics.json, --errors errors.jsonl, or --strict when needed.
2. Report the task, profile, metrics, total/scored/failed counts, and parsing assumptions.

Prefer structured predictions. Use --parser answer-tags or --parser last-line only for raw model text. Use --task auto only when the task is structurally unambiguous.

Generation metrics

Use --metrics core by default for dependency-free exact match and token-overlap metrics. Use --metrics semantic or --metrics all for BLEU, METEOR, ROUGE, keyword, embedding, and step metrics after installing:

python -m pip install -r requirements-semantic.txt

Use --offline only when NLTK resources and sentence-transformer models are already cached or supplied as local model paths.

BioProBench compatibility

Use --profile bioprobench with descriptive task names, or pass PQA, ERR, ORD, GEN, or REA-ERR directly. The shorter names qa, error-detection, ordering, and generation remain CLI aliases only. Read references/bioprobench-profile.md for field and parsing mappings.

Read references/metric-definitions.md before interpreting positive classes, units, semantic settings, or failure handling.