One-Eval: An Agentic System for Automated and Traceable LLM Evaluation
Reliable evaluation is essential for developing and deploying large language models, yet in practice it often requires substantial manual effort: practitioners must identify appropriate benchmarks, reproduce heterogeneous evaluation codebases, configure dataset schema mappings, and interpret aggregated metrics. To address these challenges, we present One-Eval, an agentic evaluation system that converts natural-language evaluation requests into executable, traceable, and customizable evaluation workflows. One-Eval integrates (i) NL2Bench for intent structuring and personalized benchmark planning, (ii) BenchResolve for benchmark resolution, automatic dataset acquisition, and schema normalization to ensure executability, and (iii) Metrics \& Reporting for task-aware metric selection and decision-oriented reporting beyond scalar scores. The system further incorporates human-in-the-loop checkpoints for review, editing, and rollback, while preserving sample evidence trails for debugging and auditability. Experiments show that One-Eval can execute end-to-end evaluations from diverse natural-language requests with minimal user effort, supporting more efficient and reproducible evaluation in industrial settings. Our framework is publicly available at https://github.com/OpenDCAI/One-Eval.
69ba1a005d2b9c28ea09f158
Chengyu Shen
69ba1a005d2b9c28ea09f159
Yanheng Hou
69ba1a005d2b9c28ea09f15a
Minghui Pan
69ba1a005d2b9c28ea09f15b
Runming He
69ba1a005d2b9c28ea09f15c
Zhen Hao Wong
69ba1a005d2b9c28ea09f15d
Meiyi Qiang
69ba1a005d2b9c28ea09f15e
Zhou Liu
69ba1a005d2b9c28ea09f15f
Hao Liang
claimed_verified
2026-04-07T08:48:05.201Z
6751a4fedf636b0140a9b873
/avatars/d75f7f6cfbfb4d646e0e557d1cfacdce.svg
Hao Liang
lhpku20010120
user
lhpku20010120
69ba1a005d2b9c28ea09f160
Peichao Lai
69ba1a005d2b9c28ea09f161
Zeang Sheng
69ba1a005d2b9c28ea09f162
Wentao Zhang
One-Eval: An Agentic System for Automated and Traceable LLM Evaluation
Reliable evaluation is essential for developing and deploying large language models, yet in practice it often requires substantial manual effort: practitioners must identify appropriate benchmarks, reproduce heterogeneous evaluation codebases, configure dataset schema mappings, and interpret aggregated metrics. To address these challenges, we present One-Eval, an agentic evaluation system that converts natural-language evaluation requests into executable, traceable, and customizable evaluation workflows. One-Eval integrates (i) NL2Bench for intent structuring and personalized benchmark planning, (ii) BenchResolve for benchmark resolution, automatic dataset acquisition, and schema normalization to ensure executability, and (iii) Metrics \& Reporting for task-aware metric selection and decision-oriented reporting beyond scalar scores. The system further incorporates human-in-the-loop checkpoints for review, editing, and rollback, while preserving sample evidence trails for debugging and auditability. Experiments show that One-Eval can execute end-to-end evaluations from diverse natural-language requests with minimal user effort, supporting more efficient and reproducible evaluation in industrial settings. Our framework is publicly available at https://github.com/OpenDCAI/One-Eval.
One-Eval is an agentic evaluation system that automates large language model assessment by converting natural-language requests into executable workflows with integrated benchmark planning, dataset handling, and customizable reporting.
NL2Bench BenchResolve Metrics & Reporting natural-language evaluation requests executable workflows benchmark planning dataset acquisition schema normalization task-aware metric selection human-in-the-loop checkpoints sample evidence trails