qhjqhj00/research-skills-pool/agentboard-eval

agentboard-eval

Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents — Bao et al. (2025) (arXiv:2508.03991, 2025)

What this evaluates

Evaluates an agent's ability to complete multi-round interactive tasks and achieve target goals across diverse task types. It simulates real-world environments where the model must navigate sequential decision-making to reach a defined endpoint.

Datasets

• AgentBoard — total ?; splits: test (-1)

Metrics

• target achievement rate (primary) — range: percent
• Percentage of tasks where the agent successfully completes the entire behavior chain to achieve the target goal.

Input / output format

Input: Multi-round interactive task prompts simulating real-world scenarios.

Output: Agent actions and decisions across multiple turns until task completion or termination.

Scoring recipe

achieved = 0
total = len(tasks)
for task in tasks:
  if task.target_achieved:
    achieved += 1
return (achieved / total) * 100

Common pitfalls

• Evaluating only single-turn success instead of the full behavior chain.
• Not accounting for the multi-round interactive nature of the tasks.

Evidence (verbatim from paper)

AgentBoard uses six types of tasks to simulate a multi-round interactive environment, and it uses the target achievement rate across the entire behavior chain as the model evaluation metric.

Citation

@misc{bao2025galaxy,
  title={Galaxy: A Cognition-Centered Framework for Proactive, Privacy-Preserving, and Self-Evolving LLM Agents},
  author={Bao et al. (2025)},
  year={2025},
  note={arXiv:2508.03991}
}

• arXiv: 2508.03991