ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking
Reinforcement learning has substantially improved the performance of LLM agents on tasks with verifiable outcomes, but it still struggles on open-ended agent tasks with vast solution spaces (e.g., complex travel planning). Due to the absence of objective ground-truth for these tasks, current RL algorithms largely rely on reward models that assign scalar scores to individual responses. We contend that such pointwise scoring suffers from an inherent discrimination collapse: the reward model struggles to distinguish subtle advantages among different trajectories, resulting in scores within a group being compressed into a narrow range. Consequently, the effective reward signal becomes dominated by noise from the reward model, leading to optimization stagnation. To address this, we propose ArenaRL, a reinforcement learning paradigm that shifts from pointwise scalar scoring to intra-group relative ranking. ArenaRL introduces a process-aware pairwise evaluation mechanism, employing multi-level rubrics to assign fine-grained relative scores to trajectories. Additionally, we construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals. Empirical results confirm that the built seeded single-elimination scheme achieves nearly equivalent advantage estimation accuracy to full pairwise comparisons with O(N^2) complexity, while operating with only O(N) complexity, striking an optimal balance between efficiency and precision. Furthermore, to address the lack of full-cycle benchmarks for open-ended agents, we build Open-Travel and Open-DeepResearch, two high-quality benchmarks featuring a comprehensive pipeline covering SFT, RL training, and multi-dimensional evaluation. Extensive experiments show that ArenaRL substantially outperforms standard RL baselines, enabling LLM agents to generate more robust solutions for complex real-world tasks.
69660763fc8c4ecc02c7fab6
Qiang Zhang
admin_assigned
2026-01-14T12:53:23.892Z
66259e3f9277b825c8e11168
/avatars/078ab932f6837f502851b75bc719ab2c.svg
Zhang
QiangZhang
user
QiangZhang
69660763fc8c4ecc02c7fab7
Boli Chen
claimed_verified
2026-01-13T15:45:22.457Z
61454d930aac1efe3e8d842e
https://cdn-avatars.huggingface.co/v1/production/uploads/61454d930aac1efe3e8d842e/sQELHgJP6c_547OrT8JZJ.jpeg
boli
bcol
user
bcol
69660763fc8c4ecc02c7fab8
Fanrui Zhang
claimed_verified
2026-01-15T15:06:09.432Z
665305eff0c8c891cae7fe01
/avatars/1f372e3bc6a4eb19ef702ec96a391c96.svg
SII-Fanrui Zhang
fanrui00
user
fanrui00
69660763fc8c4ecc02c7fab9
Ruixue Ding
69660763fc8c4ecc02c7faba
Shihang Wang
69660763fc8c4ecc02c7fabb
Qiuchen Wang
claimed_verified
2026-01-14T09:52:59.514Z
657429d833e5a4bf5b278615
/avatars/ed7e28c1b9a7bed1cad864c992cdcc69.svg
QiuchenWang
Qiuchen-Wang
user
Qiuchen-Wang
69660763fc8c4ecc02c7fabc
Yinfeng Huang
69660763fc8c4ecc02c7fabd
Haonan Zhang
admin_assigned
2026-01-14T12:53:47.932Z
623437bd4f78e3acb8bd14bd
https://cdn-avatars.huggingface.co/v1/production/uploads/623437bd4f78e3acb8bd14bd/6ZIRpE9We4TiMtzDBtPFS.jpeg
Haonan Zhang
haonanzhang
user
haonanzhang
69660763fc8c4ecc02c7fabe
Rongxiang Zhu
69660763fc8c4ecc02c7fabf
Pengyong Wang
69660763fc8c4ecc02c7fac0
Ailin Ren
69660763fc8c4ecc02c7fac1
Xin Li
69660763fc8c4ecc02c7fac2
Pengjun Xie
69660763fc8c4ecc02c7fac3
Jiawei Liu
69660763fc8c4ecc02c7fac4
Ning Guo
69660763fc8c4ecc02c7fac5
Jingren Zhou
69660763fc8c4ecc02c7fac6
Zheng-Jun Zha
ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking
Reinforcement learning has substantially improved the performance of LLM agents on tasks with verifiable outcomes, but it still struggles on open-ended agent tasks with vast solution spaces (e.g., complex travel planning). Due to the absence of objective ground-truth for these tasks, current RL algorithms largely rely on reward models that assign scalar scores to individual responses. We contend that such pointwise scoring suffers from an inherent discrimination collapse: the reward model struggles to distinguish subtle advantages among different trajectories, resulting in scores within a group being compressed into a narrow range. Consequently, the effective reward signal becomes dominated by noise from the reward model, leading to optimization stagnation. To address this, we propose ArenaRL, a reinforcement learning paradigm that shifts from pointwise scalar scoring to intra-group relative ranking. ArenaRL introduces a process-aware pairwise evaluation mechanism, employing multi-level rubrics to assign fine-grained relative scores to trajectories. Additionally, we construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals. Empirical results confirm that the built seeded single-elimination scheme achieves nearly equivalent advantage estimation accuracy to full pairwise comparisons with O(N^2) complexity, while operating with only O(N) complexity, striking an optimal balance between efficiency and precision. Furthermore, to address the lack of full-cycle benchmarks for open-ended agents, we build Open-Travel and Open-DeepResearch, two high-quality benchmarks featuring a comprehensive pipeline covering SFT, RL training, and multi-dimensional evaluation. Extensive experiments show that ArenaRL substantially outperforms standard RL baselines, enabling LLM agents to generate more robust solutions for complex real-world tasks.
Reinforcement learning for large language model agents suffers from discrimination collapse in open-ended tasks due to pointwise scalar scoring, which ArenaRL addresses through relative ranking and pairwise evaluation mechanisms.
reinforcement learning large language model agents reward models scalar scores discrimination collapse intra-group relative ranking pairwise evaluation multi-level rubrics adversarial arena tournament-based ranking advantage estimation efficiency precision benchmarks SFT RL training multi-dimensional evaluation