Exploration and Exploitation Errors Are Measurable for Language Model Agents
Language Model (LM) agents are increasingly used in complex open-ended decision-making tasks, from AI coding to physical AI. A core requirement in these settings is the ability to both explore the problem space and exploit acquired knowledge effectively. However, systematically distinguishing and quantifying exploration and exploitation from observed actions without access to the agent's internal policy remains challenging. To address this, we design controllable environments inspired by practical embodied AI scenarios. Each environment consists of a partially observable 2D grid map and an unknown task Directed Acyclic Graph (DAG). The map generation can be programmatically adjusted to emphasize exploration or exploitation difficulty. To enable policy-agnostic evaluation, we design a metric to quantify exploration and exploitation errors from agent's actions. We evaluate a variety of frontier LM agents and find that even state-of-the-art models struggle on our task, with different models exhibiting distinct failure modes. We further observe that reasoning models solve the task more effectively and show both exploration and exploitation can be significantly improved through minimal harness engineering. We release our code https://github.com/jjj-madison/measurable-explore-exploit{here}.
69e04fae5b704445ba50b71e
Jaden Park
claimed_verified
2026-04-16T07:59:25.020Z
670f5267d1b58394145c1ca3
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/t-YchgvZCbDW-plR8DZbA.png
Jaden Park
jpark677
user
jpark677
69e04fae5b704445ba50b71f
Jungtaek Kim
claimed_verified
2026-04-16T20:21:28.341Z
651ce91c610889eda1800463
/avatars/255e51c71464f549fe50ddb83de9f41c.svg
Jungtaek Kim
jungtaekkim
user
jungtaekkim
69e04fae5b704445ba50b720
Jongwon Jeong
claimed_verified
2026-04-16T07:59:20.638Z
654224d4eccc4f48dcea725d
/avatars/a325691bac8e3ff53dddd96c22d7fa67.svg
Jongwon Jeong
jongwon-jeong
user
jongwon-jeong
69e04fae5b704445ba50b721
Robert D. Nowak
69e04fae5b704445ba50b722
Kangwook Lee
69e04fae5b704445ba50b723
Yong Jae Lee
Exploration and Exploitation Errors Are Measurable for Language Model Agents
Language Model (LM) agents are increasingly used in complex open-ended decision-making tasks, from AI coding to physical AI. A core requirement in these settings is the ability to both explore the problem space and exploit acquired knowledge effectively. However, systematically distinguishing and quantifying exploration and exploitation from observed actions without access to the agent's internal policy remains challenging. To address this, we design controllable environments inspired by practical embodied AI scenarios. Each environment consists of a partially observable 2D grid map and an unknown task Directed Acyclic Graph (DAG). The map generation can be programmatically adjusted to emphasize exploration or exploitation difficulty. To enable policy-agnostic evaluation, we design a metric to quantify exploration and exploitation errors from agent's actions. We evaluate a variety of frontier LM agents and find that even state-of-the-art models struggle on our task, with different models exhibiting distinct failure modes. We further observe that reasoning models solve the task more effectively and show both exploration and exploitation can be significantly improved through minimal harness engineering. We release our code https://github.com/jjj-madison/measurable-explore-exploit{here}.
Controllable environments with programmable exploration-exploitation balance are designed to evaluate language model agents' performance on embodied AI tasks, revealing distinct failure modes and demonstrating that reasoning models outperform other approaches.
language model agents embodied AI exploration-exploitation tradeoff partially observable 2D grid map Directed Acyclic Graph policy-agnostic evaluation reasoning models harness engineering