CCTU: A Benchmark for Tool Use under Complex Constraints
Solving problems through tool use under explicit constraints constitutes a highly challenging yet unavoidable scenario for large language models (LLMs), requiring capabilities such as function calling, instruction following, and self-refinement. However, progress has been hindered by the absence of dedicated evaluations. To address this, we introduce CCTU, a benchmark for evaluating LLM tool use under complex constraints. CCTU is grounded in a taxonomy of 12 constraint categories spanning four dimensions (i.e., resource, behavior, toolset, and response). The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios, each involving an average of seven constraint types and an average prompt length exceeding 4,700 tokens. To enable reliable evaluation, we develop an executable constraint validation module that performs step-level validation and enforces compliance during multi-turn interactions between models and their environments. We evaluate nine state-of-the-art LLMs in both thinking and non-thinking modes. Results indicate that when strict adherence to all constraints is required, no model achieves a task completion rate above 20%. Further analysis reveals that models violate constraints in over 50% of cases, particularly in the resource and response dimensions. Moreover, LLMs demonstrate limited capacity for self-refinement even after receiving detailed feedback on constraint violations, highlighting a critical bottleneck in the development of robust tool-use agents. To facilitate future research, we release the data and code.
69b8d1aa6b3f36c249fbc458
Junjie Ye
claimed_verified
2026-03-17T08:19:32.244Z
66384be673c2c55f2ded89fa
/avatars/1d8721074f0f51fab405f81474f2035f.svg
Junjie Ye
Junjie-Ye
user
Junjie-Ye
69b8d1aa6b3f36c249fbc459
Guoqiang Zhang
69b8d1aa6b3f36c249fbc45a
Wenjie Fu
69b8d1aa6b3f36c249fbc45b
Tao Gui
69b8d1aa6b3f36c249fbc45c
Qi Zhang
69b8d1aa6b3f36c249fbc45d
Xuanjing Huang
admin_assigned
2026-03-18T10:18:53.795Z
67f9c4ee171948c38302ae0f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/Cqb3ijr_sZkpLhEEEEybK.png
Xuanjing Huang
xjhuang
user
xjhuang
CCTU: A Benchmark for Tool Use under Complex Constraints
Solving problems through tool use under explicit constraints constitutes a highly challenging yet unavoidable scenario for large language models (LLMs), requiring capabilities such as function calling, instruction following, and self-refinement. However, progress has been hindered by the absence of dedicated evaluations. To address this, we introduce CCTU, a benchmark for evaluating LLM tool use under complex constraints. CCTU is grounded in a taxonomy of 12 constraint categories spanning four dimensions (i.e., resource, behavior, toolset, and response). The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios, each involving an average of seven constraint types and an average prompt length exceeding 4,700 tokens. To enable reliable evaluation, we develop an executable constraint validation module that performs step-level validation and enforces compliance during multi-turn interactions between models and their environments. We evaluate nine state-of-the-art LLMs in both thinking and non-thinking modes. Results indicate that when strict adherence to all constraints is required, no model achieves a task completion rate above 20%. Further analysis reveals that models violate constraints in over 50% of cases, particularly in the resource and response dimensions. Moreover, LLMs demonstrate limited capacity for self-refinement even after receiving detailed feedback on constraint violations, highlighting a critical bottleneck in the development of robust tool-use agents. To facilitate future research, we release the data and code.