The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
Autonomous agents based on large language models (LLMs) are rapidly evolving to handle multi-turn tasks, but ensuring their trustworthiness remains a critical challenge. A fundamental pillar of this trustworthiness is calibration, which refers to an agent's ability to express confidence that reliably reflects its actual performance. While calibration is well-established for static models, its dynamics in tool-integrated agentic workflows remain underexplored. In this work, we systematically investigate verbalized calibration in tool-use agents, revealing a fundamental confidence dichotomy driven by tool type. Specifically, our pilot study identifies that evidence tools (e.g., web search) systematically induce severe overconfidence due to inherent noise in retrieved information, while verification tools (e.g., code interpreters) can ground reasoning through deterministic feedback and mitigate miscalibration. To robustly improve calibration across tool types, we propose a reinforcement learning (RL) fine-tuning framework that jointly optimizes task accuracy and calibration, supported by a holistic benchmark of reward designs. We demonstrate that our trained agents not only achieve superior calibration but also exhibit robust generalization from local training environments to noisy web settings and to distinct domains such as mathematical reasoning. Our results highlight the necessity of domain-specific calibration strategies for tool-use agents. More broadly, this work establishes a foundation for building self-aware agents that can reliably communicate uncertainty in high-stakes, real-world deployments.
69671055c5e371f6b235d154
Weihao Xuan
claimed_verified
2026-01-14T09:51:23.840Z
65b8909c89eb3dfbe8d26780
https://cdn-avatars.huggingface.co/v1/production/uploads/65b8909c89eb3dfbe8d26780/aJExXsCN9oQOdd9bsQOEO.jpeg
Weihao XUAN
weihao1115
user
weihao1115
69671055c5e371f6b235d155
Qingcheng Zeng
69671055c5e371f6b235d156
Heli Qi
69671055c5e371f6b235d157
Yunze Xiao
claimed_verified
2026-01-14T12:47:43.795Z
67c8af4885b7a86d7515b77c
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/4bRu0MKg6voGX5fyqninn.png
Lorenzo Xiao
lrzneedresearch
user
lrzneedresearch
69671055c5e371f6b235d158
Junjue Wang
admin_assigned
2026-01-14T12:54:17.992Z
65aaf7bb494b2faa870f04d2
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/5twYdL2f-2h3Dx0BhC7HW.png
JJ Wang
junjuewang
user
junjuewang
69671055c5e371f6b235d159
Naoto Yokoya
admin_assigned
2026-01-14T12:54:24.392Z
635c8bafe3737b9e4e29406a
/avatars/884e8e086fe374512ed6956ab59038af.svg
naoto yoko
Naotoyokoyama
user
Naotoyokoyama
The Confidence Dichotomy: Analyzing and Mitigating Miscalibration in Tool-Use Agents
Autonomous agents based on large language models (LLMs) are rapidly evolving to handle multi-turn tasks, but ensuring their trustworthiness remains a critical challenge. A fundamental pillar of this trustworthiness is calibration, which refers to an agent's ability to express confidence that reliably reflects its actual performance. While calibration is well-established for static models, its dynamics in tool-integrated agentic workflows remain underexplored. In this work, we systematically investigate verbalized calibration in tool-use agents, revealing a fundamental confidence dichotomy driven by tool type. Specifically, our pilot study identifies that evidence tools (e.g., web search) systematically induce severe overconfidence due to inherent noise in retrieved information, while verification tools (e.g., code interpreters) can ground reasoning through deterministic feedback and mitigate miscalibration. To robustly improve calibration across tool types, we propose a reinforcement learning (RL) fine-tuning framework that jointly optimizes task accuracy and calibration, supported by a holistic benchmark of reward designs. We demonstrate that our trained agents not only achieve superior calibration but also exhibit robust generalization from local training environments to noisy web settings and to distinct domains such as mathematical reasoning. Our results highlight the necessity of domain-specific calibration strategies for tool-use agents. More broadly, this work establishes a foundation for building self-aware agents that can reliably communicate uncertainty in high-stakes, real-world deployments.
Tool-integrated language model agents exhibit different calibration behaviors based on tool type, with a reinforcement learning framework improving both task accuracy and reliable uncertainty estimation across diverse domains.
large language models tool-integrated agents calibration reinforcement learning reward design verbalized calibration evidence tools verification tools