CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in
  Dynamic Environments for LLM Tool-Use Agents
Current evaluations of Large Language Model (LLM) agents primarily emphasize
task completion, often overlooking resource efficiency and adaptability. This
neglects a crucial capability: agents' ability to devise and adjust
cost-optimal plans in response to changing environments. To bridge this gap, we
introduce CostBench, a scalable, cost-centric benchmark designed to evaluate
agents' economic reasoning and replanning abilities. Situated in the
travel-planning domain, CostBench comprises tasks solvable via multiple
sequences of atomic and composite tools with diverse, customizable costs. It
also supports four types of dynamic blocking events, such as tool failures and
cost changes, to simulate real-world unpredictability and necessitate agents to
adapt in real time. Evaluating leading open-sourced and proprietary models on
CostBench reveals a substantial gap in cost-aware planning: agents frequently
fail to identify cost-optimal solutions in static settings, with even GPT-5
achieving less than 75% exact match rate on the hardest tasks, and performance
further dropping by around 40% under dynamic conditions. By diagnosing these
weaknesses, CostBench lays the groundwork for developing future agents that are
both economically rational and robust.
690ad523d70e173c84529016
Jiayu Liu
claimed_verified
2025-11-06T11:48:28.858Z
66783baec3f824dde8f783ac
/avatars/eb9a0441986be50274c5e661f7039e2c.svg
Jeff
JiayuJeff
user
JiayuJeff
690ad523d70e173c84529017
Cheng Qian
690ad523d70e173c84529018
Zhaochen Su
690ad523d70e173c84529019
Qing Zong
690ad523d70e173c8452901a
Shijue Huang
690ad523d70e173c8452901b
Bingxiang He
690ad523d70e173c8452901c
Yi R. Fung
CostBench: Evaluating Multi-Turn Cost-Optimal Planning and Adaptation in
  Dynamic Environments for LLM Tool-Use Agents
Current evaluations of Large Language Model (LLM) agents primarily emphasize
task completion, often overlooking resource efficiency and adaptability. This
neglects a crucial capability: agents' ability to devise and adjust
cost-optimal plans in response to changing environments. To bridge this gap, we
introduce CostBench, a scalable, cost-centric benchmark designed to evaluate
agents' economic reasoning and replanning abilities. Situated in the
travel-planning domain, CostBench comprises tasks solvable via multiple
sequences of atomic and composite tools with diverse, customizable costs. It
also supports four types of dynamic blocking events, such as tool failures and
cost changes, to simulate real-world unpredictability and necessitate agents to
adapt in real time. Evaluating leading open-sourced and proprietary models on
CostBench reveals a substantial gap in cost-aware planning: agents frequently
fail to identify cost-optimal solutions in static settings, with even GPT-5
achieving less than 75% exact match rate on the hardest tasks, and performance
further dropping by around 40% under dynamic conditions. By diagnosing these
weaknesses, CostBench lays the groundwork for developing future agents that are
both economically rational and robust.
CostBench evaluates Large Language Model agents' cost-aware planning and adaptability in response to dynamic changes, revealing significant gaps in current models' performance.
Large Language Model CostBench economic reasoning replanning abilities travel-planning domain atomic tools composite tools dynamic blocking events tool failures cost changes cost-optimal solutions exact match rate