Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leaving the rich information in NL feedback underutilized and leading to inefficient exploration. In this work, we propose GOLF, an RL framework that explicitly exploits group-level language feedback to guide targeted exploration through actionable refinements. GOLF aggregates two complementary feedback sources: (i) external critiques that pinpoint errors or propose targeted fixes, and (ii) intra-group attempts that supply alternative partial ideas and diverse failure patterns. These group-level feedbacks are aggregated to produce high-quality refinements, which are adaptively injected into training as off-policy scaffolds to provide targeted guidance in sparse-reward regions. Meanwhile, GOLF jointly optimizes generation and refinement within a unified RL loop, creating a virtuous cycle that continuously improves both capabilities. Experiments on both verifiable and non-verifiable benchmarks show that GOLF achieves superior performance and exploration efficiency, achieving 2.2times improvements in sample efficiency compared to RL methods trained solely on scalar rewards. Code is available at https://github.com/LuckyyySTA/GOLF.
69b222ca206bad330cd7ff54
Lei Huang
admin_assigned
2026-03-12T10:41:48.423Z
61f4c2e981c4d30f58140279
/avatars/c4a69f6563c952354e33682e86045b14.svg
HuangMeow
Luckyyy
user
Luckyyy
69b222ca206bad330cd7ff55
Xiang Cheng
69b222ca206bad330cd7ff56
Chenxiao Zhao
admin_assigned
2026-03-12T10:41:59.630Z
63fc5b724c57549ad5e54558
/avatars/1374c1e8969533dd7543959666f16d1a.svg
Chenxiao Zhao
ChenShawn
user
ChenShawn
69b222ca206bad330cd7ff57
Guobin Shen
admin_assigned
2026-03-12T10:42:05.510Z
67737f334944e2c332a201f6
/avatars/17483d416f1dd19bb9c55e1c9eedb697.svg
guobin.shen
ussop1
user
ussop1
69b222ca206bad330cd7ff58
Junjie Yang
69b222ca206bad330cd7ff59
Xiaocheng Feng
69b222ca206bad330cd7ff5a
Yuxuan Gu
69b222ca206bad330cd7ff5b
Xing Yu
69b222ca206bad330cd7ff5c
Bing Qin
Bootstrapping Exploration with Group-Level Natural Language Feedback in Reinforcement Learning
Large language models (LLMs) typically receive diverse natural language (NL) feedback through interaction with the environment. However, current reinforcement learning (RL) algorithms rely solely on scalar rewards, leaving the rich information in NL feedback underutilized and leading to inefficient exploration. In this work, we propose GOLF, an RL framework that explicitly exploits group-level language feedback to guide targeted exploration through actionable refinements. GOLF aggregates two complementary feedback sources: (i) external critiques that pinpoint errors or propose targeted fixes, and (ii) intra-group attempts that supply alternative partial ideas and diverse failure patterns. These group-level feedbacks are aggregated to produce high-quality refinements, which are adaptively injected into training as off-policy scaffolds to provide targeted guidance in sparse-reward regions. Meanwhile, GOLF jointly optimizes generation and refinement within a unified RL loop, creating a virtuous cycle that continuously improves both capabilities. Experiments on both verifiable and non-verifiable benchmarks show that GOLF achieves superior performance and exploration efficiency, achieving 2.2times improvements in sample efficiency compared to RL methods trained solely on scalar rewards. Code is available at https://github.com/LuckyyySTA/GOLF.
Language feedback is leveraged in reinforcement learning to improve exploration efficiency and sample utilization through grouped critique aggregation and joint generation-refinement optimization.
reinforcement learning natural language feedback scalar rewards targeted exploration group-level feedback off-policy scaffolding joint optimization sample efficiency