Automating Benchmark Design
The rapid progress and widespread deployment of LLMs and LLM-powered agents
has outpaced our ability to evaluate them. Hand-crafted, static benchmarks are
the primary tool for assessing model capabilities, but these quickly become
saturated. In contrast, dynamic benchmarks evolve alongside the models they
evaluate, but are expensive to create and continuously update. To address these
challenges, we develop BeTaL (Benchmark Tuning with an LLM-in-the-loop), a
framework that leverages environment design principles to automate the process
of dynamic benchmark design. BeTaL works by parameterizing key design choices
in base benchmark templates and uses LLMs to reason through the resulting
parameter space to obtain target properties (such as difficulty and realism) in
a cost-efficient manner. We validate this approach on its ability to create
benchmarks with desired difficulty levels. Using BeTaL, we create two new
benchmarks and extend a popular agentic benchmark tau-bench. Extensive
evaluation on these three tasks and multiple target difficulty levels shows
that BeTaL produces benchmarks much closer to the desired difficulty, with
average deviations ranging from 5.3% to 13.2% -- a 2-4x improvement over the
baselines.
69039a7d2c556835fa67eed8
Amanda Dsouza
69039a7d2c556835fa67eed9
Harit Vishwakarma
69039a7d2c556835fa67eeda
Zhengyang Qi
69039a7d2c556835fa67eedb
Justin Bauer
69039a7d2c556835fa67eedc
Derek Pham
69039a7d2c556835fa67eedd
Thomas Walshe
69039a7d2c556835fa67eede
Armin Parchami
69039a7d2c556835fa67eedf
Frederic Sala
69039a7d2c556835fa67eee0
Paroma Varma
Automating Benchmark Design
The rapid progress and widespread deployment of LLMs and LLM-powered agents
has outpaced our ability to evaluate them. Hand-crafted, static benchmarks are
the primary tool for assessing model capabilities, but these quickly become
saturated. In contrast, dynamic benchmarks evolve alongside the models they
evaluate, but are expensive to create and continuously update. To address these
challenges, we develop BeTaL (Benchmark Tuning with an LLM-in-the-loop), a
framework that leverages environment design principles to automate the process
of dynamic benchmark design. BeTaL works by parameterizing key design choices
in base benchmark templates and uses LLMs to reason through the resulting
parameter space to obtain target properties (such as difficulty and realism) in
a cost-efficient manner. We validate this approach on its ability to create
benchmarks with desired difficulty levels. Using BeTaL, we create two new
benchmarks and extend a popular agentic benchmark tau-bench. Extensive
evaluation on these three tasks and multiple target difficulty levels shows
that BeTaL produces benchmarks much closer to the desired difficulty, with
average deviations ranging from 5.3% to 13.2% -- a 2-4x improvement over the
baselines.
BeTaL, a framework using LLMs to automate dynamic benchmark design, creates benchmarks with desired difficulty levels more accurately and efficiently than static methods.
LLMs LLM-powered agents dynamic benchmarks static benchmarks environment design principles parameter space difficulty realism $\tau$-bench