Rank-GRPO: Training LLM-based Conversational Recommender Systems with
  Reinforcement Learning
Large language models (LLMs) are reshaping the recommender system paradigm by
enabling users to express preferences and receive recommendations through
conversations. Yet, aligning LLMs to the recommendation task remains
challenging: pretrained LLMs often generate out-of-catalog items, violate
required output formats, and their ranking quality degrades sharply toward the
end of the generated list. To this end, we propose ConvRec-R1, a two-stage
framework for end-to-end training of LLM-based conversational recommender
systems. In Stage 1, we construct a behavioral-cloning dataset with a
Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded
demonstrations from powerful blackbox LLMs to warm-start the RL training. In
Stage 2, we propose Rank-GRPO, a principled extension of group relative policy
optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats
each rank in the recommendation list as the unit instead of token (too
fine-grained) or sequence (too coarse), redefining rewards to remove non-causal
credit assignment and introducing a rank-level importance ratio based on the
geometric mean of rank-wise token probabilities to stabilize policy updates.
Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges
faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and
datasets are released at https://github.com/yaochenzhu/Rank-GRPO.
68fad999f158a71c5a2f57fc
Yaochen Zhu
claimed_verified
2025-11-03T20:57:59.417Z
647a248bed75e95d3e98e3d6
/avatars/e10e2f8516d1451fd85e17b5a0ba978d.svg
Yaochen Zhu
yaochenzhu
user
yaochenzhu
68fad999f158a71c5a2f57fd
Harald Steck
68fad999f158a71c5a2f57fe
Dawen Liang
68fad999f158a71c5a2f57ff
Yinhan He
claimed_verified
2026-02-27T16:47:31.012Z
654f058c35d81e5153c4dce0
/avatars/87480482d8191cd63cac39e7a6a00285.svg
Jonathan Yinhan He
jonathanhe123
user
jonathanhe123
68fad999f158a71c5a2f5800
Jundong Li
68fad999f158a71c5a2f5801
Nathan Kallus
Rank-GRPO: Training LLM-based Conversational Recommender Systems with
  Reinforcement Learning
Large language models (LLMs) are reshaping the recommender system paradigm by
enabling users to express preferences and receive recommendations through
conversations. Yet, aligning LLMs to the recommendation task remains
challenging: pretrained LLMs often generate out-of-catalog items, violate
required output formats, and their ranking quality degrades sharply toward the
end of the generated list. To this end, we propose ConvRec-R1, a two-stage
framework for end-to-end training of LLM-based conversational recommender
systems. In Stage 1, we construct a behavioral-cloning dataset with a
Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded
demonstrations from powerful blackbox LLMs to warm-start the RL training. In
Stage 2, we propose Rank-GRPO, a principled extension of group relative policy
optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats
each rank in the recommendation list as the unit instead of token (too
fine-grained) or sequence (too coarse), redefining rewards to remove non-causal
credit assignment and introducing a rank-level importance ratio based on the
geometric mean of rank-wise token probabilities to stabilize policy updates.
Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges
faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and
datasets are released at https://github.com/yaochenzhu/Rank-GRPO.
ConvRec-R1, a two-stage framework, enhances LLM-based conversational recommender systems by using behavioral cloning and Rank-GRPO to improve recommendation quality and convergence.
LLMs recommender system behavioral-cloning Remap-Reflect-Adjust Rank-GRPO group relative policy optimization GRPO rank-level importance ratio Reddit-v2 dataset Recall NDCG