Efficient Exploration at Scale
We develop an online learning algorithm that dramatically improves the data efficiency of reinforcement learning from human feedback (RLHF). Our algorithm incrementally updates reward and language models as choice data is received. The reward model is fit to the choice data, while the language model is updated by a variation of reinforce, with reinforcement signals provided by the reward model. Several features enable the efficiency gains: a small affirmative nudge added to each reinforcement signal, an epistemic neural network that models reward uncertainty, and information-directed exploration. With Gemma large language models (LLMs), our algorithm matches the performance of offline RLHF trained on 200K labels using fewer than 20K labels, representing more than a 10x gain in data efficiency. Extrapolating from our results, we expect our algorithm trained on 1M labels to match offline RLHF trained on 1B labels. This represents a 1,000x gain. To our knowledge, these are the first results to demonstrate that such large improvements are possible.
69bb74d0ba2b62a5fb0b0cdc
Seyed Mohammad Asghari
69bb74d0ba2b62a5fb0b0cdd
Chris Chute
69bb74d0ba2b62a5fb0b0cde
Vikranth Dwaracherla
69bb74d0ba2b62a5fb0b0cdf
Xiuyuan Lu
69bb74d0ba2b62a5fb0b0ce0
Mehdi Jafarnia
69bb74d0ba2b62a5fb0b0ce1
Victor Minden
69bb74d0ba2b62a5fb0b0ce2
Zheng Wen
69bb74d0ba2b62a5fb0b0ce3
Benjamin Van Roy
Efficient Exploration at Scale
We develop an online learning algorithm that dramatically improves the data efficiency of reinforcement learning from human feedback (RLHF). Our algorithm incrementally updates reward and language models as choice data is received. The reward model is fit to the choice data, while the language model is updated by a variation of reinforce, with reinforcement signals provided by the reward model. Several features enable the efficiency gains: a small affirmative nudge added to each reinforcement signal, an epistemic neural network that models reward uncertainty, and information-directed exploration. With Gemma large language models (LLMs), our algorithm matches the performance of offline RLHF trained on 200K labels using fewer than 20K labels, representing more than a 10x gain in data efficiency. Extrapolating from our results, we expect our algorithm trained on 1M labels to match offline RLHF trained on 1B labels. This represents a 1,000x gain. To our knowledge, these are the first results to demonstrate that such large improvements are possible.
An online learning algorithm for reinforcement learning from human feedback that achieves significant data efficiency improvements through incremental model updates, reward uncertainty modeling, and information-directed exploration.
reinforcement learning from human feedback reward model language model reinforce epistemic neural network information-directed exploration data efficiency