qhjqhj00/research-skills-pool/aide-benchmark-eval

aide-benchmark-eval

AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuning — Jiayu Li et al. (arXiv:2412.06136, 2024)

What this evaluates

Evaluates the performance of LLMs fine-tuned on synthetic data generated by AIDE across a suite of standard knowledge and reasoning benchmarks. It probes zero-shot and few-shot generalization capabilities compared to models fine-tuned on human-curated gold data.

Datasets

• MMLU — total ?; splits: test (-1)
• FinBen — total ?; splits: test (-1)
• ARC-Challenge — total ?; splits: test (-1)
• GSM8K — total ?; splits: test (-1)
• TruthfulQA — total ?; splits: test (-1)
• MedQA — total ?; splits: test (-1)
• BIG-Bench — total ?; splits: test (-1)

Metrics

• zero-shot accuracy (primary) — range: percent
• Percentage of correctly predicted answers across all benchmark tasks. For GSM8K, the paper reports 8-shot maj@8 performance instead.

Input / output format

Input: Benchmark-specific prompts. Most tasks use zero-shot prompts. GSM8K uses 8-shot prompts with in-context examples.

Output: Model-generated answer strings or multiple-choice selections.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) * 100
def compute_maj_at_8(predictions_8shot):
    # predictions_8shot is a list of lists, each inner list has 8 generations
    maj_votes = [Counter(p).most_common(1)[0][0] for p in predictions_8shot]
    return maj_votes

Common pitfalls

• GSM8K uses a different evaluation protocol (8-shot maj@8) than the zero-shot accuracy used for other benchmarks.
• The paper reports average performance across benchmarks, which can mask task-specific variance.
• Self-BLEU is used for diversity analysis but is not the primary performance metric.

Evidence (verbatim from paper)

We evaluated all models using zero-shot accuracy as the primary metric on the benchmarks. For GSM8K, we report 8-shot maj@8 performance using prompts from Wang et al. (2023).

Citation

@misc{li2024aide,
  title={AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuning},
  author={Jiayu Li et al.},
  year={2024},
  note={arXiv:2412.06136}
}

• arXiv: 2412.06136