A2Eval: Agentic and Automated Evaluation for Embodied Brain
Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks that exhibit severe redundancy and coverage imbalance. This labor intensive paradigm drains computational and annotation resources, inflates costs, and distorts model rankings, ultimately stifling iterative development. To address this, we propose Agentic Automatic Evaluation (A2Eval), the first agentic framework that automates benchmark curation and evaluation through two collaborative agents. The Data Agent autonomously induces capability dimensions and assembles a balanced, compact evaluation suite, while the Eval Agent synthesizes and validates executable evaluation pipelines, enabling fully autonomous, high-fidelity assessment. Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup while preserving evaluation quality. Crucially, A2Eval corrects systematic ranking biases, improves human alignment to Spearman's rho=0.85, and maintains high ranking fidelity (Kendall's tau=0.81), establishing a new standard for high-fidelity, low-cost embodied assessment. Our code and data will be public soon.
69840412e34659da7e1f4d49
Shuai Zhang
69840412e34659da7e1f4d4a
Jiayu Hu
69840412e34659da7e1f4d4b
Zijie Chen
claimed_verified
2026-02-27T16:45:52.271Z
630a2bcf4bee441367afba10
/avatars/463a62b296887f1be8290475b91af098.svg
Zijie Chen
Zijie-chen
user
Zijie-chen
69840412e34659da7e1f4d4c
Zeyuan Ding
69840412e34659da7e1f4d4d
Yi Zhang
69840412e34659da7e1f4d4e
Yingji Zhang
69840412e34659da7e1f4d4f
Ziyi Zhou
69840412e34659da7e1f4d50
Junwei Liao
69840412e34659da7e1f4d51
Shengjie Zhou
69840412e34659da7e1f4d52
Yong Dai
69840412e34659da7e1f4d53
Zhenzhong Lan
69840412e34659da7e1f4d54
Xiaozhu Ju
A2Eval: Agentic and Automated Evaluation for Embodied Brain
Current embodied VLM evaluation relies on static, expert-defined, manually annotated benchmarks that exhibit severe redundancy and coverage imbalance. This labor intensive paradigm drains computational and annotation resources, inflates costs, and distorts model rankings, ultimately stifling iterative development. To address this, we propose Agentic Automatic Evaluation (A2Eval), the first agentic framework that automates benchmark curation and evaluation through two collaborative agents. The Data Agent autonomously induces capability dimensions and assembles a balanced, compact evaluation suite, while the Eval Agent synthesizes and validates executable evaluation pipelines, enabling fully autonomous, high-fidelity assessment. Evaluated across 10 benchmarks and 13 models, A2Eval compresses evaluation suites by 85%, reduces overall computational costs by 77%, and delivers a 4.6x speedup while preserving evaluation quality. Crucially, A2Eval corrects systematic ranking biases, improves human alignment to Spearman's rho=0.85, and maintains high ranking fidelity (Kendall's tau=0.81), establishing a new standard for high-fidelity, low-cost embodied assessment. Our code and data will be public soon.
Agentic automatic evaluation framework automates embodied vision-language model assessment through collaborative agents that reduce evaluation costs and improve ranking accuracy.
embodied VLM benchmark curation evaluation suite computational costs ranking bias human alignment Spearman's rho Kendall's tau