Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents
As Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering, they frequently encounter underspecified instructions that lack crucial context. While human developers naturally resolve underspecification by asking clarifying questions, current agents are largely optimized for autonomous execution. In this work, we systematically evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified. We propose an uncertainty-aware multi-agent scaffold that explicitly decouples underspecification detection from code execution. Our results demonstrate that this multi-agent system using OpenHands + Claude Sonnet 4.5 achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup (61.20%) and closing the performance gap with agents operating on fully specified instructions. Furthermore, we find that the multi-agent system exhibits well-calibrated uncertainty, conserving queries on simple tasks while proactively seeking information on more complex issues. These findings indicate that current models can be turned into proactive collaborators, where agents independently recognize when to ask questions to elicit missing information in real-world, underspecified tasks.
69ccd93e3f84a0971be66cc3
Nicholas Edwards
claimed_verified
2026-04-07T08:46:52.258Z
66227ed24fc5602afb2a79ca
/avatars/e1d2c49d96aae919e70f783b986bacb9.svg
Nicholas Edwards
nedwards99
user
nedwards99
69ccd93e3f84a0971be66cc4
Sebastian Schuster
claimed_verified
2026-04-02T15:32:10.887Z
613fa15bbe1df5145babcace
/avatars/bad9367d35df0252a193548926d19ac2.svg
Sebastian Schuster
sebschu
user
sebschu
Ask or Assume? Uncertainty-Aware Clarification-Seeking in Coding Agents
As Large Language Model (LLM) agents are increasingly deployed in open-ended domains like software engineering, they frequently encounter underspecified instructions that lack crucial context. While human developers naturally resolve underspecification by asking clarifying questions, current agents are largely optimized for autonomous execution. In this work, we systematically evaluate the clarification-seeking abilities of LLM agents on an underspecified variant of SWE-bench Verified. We propose an uncertainty-aware multi-agent scaffold that explicitly decouples underspecification detection from code execution. Our results demonstrate that this multi-agent system using OpenHands + Claude Sonnet 4.5 achieves a 69.40% task resolve rate, significantly outperforming a standard single-agent setup (61.20%) and closing the performance gap with agents operating on fully specified instructions. Furthermore, we find that the multi-agent system exhibits well-calibrated uncertainty, conserving queries on simple tasks while proactively seeking information on more complex issues. These findings indicate that current models can be turned into proactive collaborators, where agents independently recognize when to ask questions to elicit missing information in real-world, underspecified tasks.
A multi-agent system using uncertainty-aware design improves LLM agent performance on underspecified software development tasks by detecting ambiguity and proactively seeking clarification.
Large Language Model agents underspecification SWE-bench Verified multi-agent scaffold uncertainty-aware code execution OpenHands Claude Sonnet 4.5 task resolve rate well-calibrated uncertainty