AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts
The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce AgentLongBench, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues.
697b161fdf3e800774f13d05
Shicheng Fang
697b161fdf3e800774f13d06
Yuxin Wang
697b161fdf3e800774f13d07
XiaoRan Liu
claimed_verified
2026-02-02T17:00:39.556Z
64f033ef82c6eea604c4da8b
/avatars/51b93fea7fd68b4274ee03701245dcca.svg
Xiaoran Liu (SII)
SII-xrliu
user
SII-xrliu
697b161fdf3e800774f13d08
Jiahao Lu
697b161fdf3e800774f13d09
Chuanyuan Tan
697b161fdf3e800774f13d0a
Xinchi Chen
697b161fdf3e800774f13d0b
Yining Zheng. Xuanjing Huang
697b161fdf3e800774f13d0c
Xipeng Qiu
AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts
The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce AgentLongBench, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues.
AgentLongBench evaluates large language models as autonomous agents through dynamic environment interactions, revealing challenges in handling high-information-density tool responses compared to memory fragmentation in long conversations.
Large Language Models autonomous agents dynamic contexts AgentLongBench Lateral Thinking Puzzles environment rollouts knowledge-intensive scenarios knowledge-free scenarios information density tool responses memory fragmentation long-turn dialogues