Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs
Exploration capacity shapes both inference-time performance and reinforcement learning (RL) training for large (vision-) language models, as stochastic sampling often yields redundant reasoning paths with little high-level diversity. This paper proposes Reasoning Palette, a novel latent-modulation framework that endows the model with a stochastic latent variable for strategic contextualization, guiding its internal planning prior to token generation. This latent context is inferred from the mean-pooled embedding of a question-answer pair via a variational autoencoder (VAE), where each sampled latent potentially encodes a distinct reasoning context. During inference, a sampled latent is decoded into learnable token prefixes and prepended to the input prompt, modulating the model's internal reasoning trajectory. In this way, the model performs internal sampling over reasoning strategies prior to output generation, which shapes the style and structure of the entire response sequence. A brief supervised fine-tuning (SFT) warm-up phase allows the model to adapt to this latent conditioning. Within RL optimization, Reasoning Palette facilitates structured exploration by enabling on-demand injection for diverse reasoning modes, significantly enhancing exploration efficiency and sustained learning capability. Experiments across multiple reasoning benchmarks demonstrate that our method enables interpretable and controllable control over the (vision-) language model's strategic behavior, thereby achieving consistent performance gains over standard RL methods.
694a20cc335742716e932304
Rujiao Long
694a20cc335742716e932305
Yang Li
claimed_verified
2026-01-04T20:32:06.364Z
6570587b44d0620d221c722b
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/T-B6q6sqhWAyXM7Hk_qWp.jpeg
Yang Li (SJTU & SII)
yangcole
user
yangcole
694a20cc335742716e932306
Xingyao Zhang
694a20cc335742716e932307
Weixun Wang
694a20cc335742716e932308
Tianqianjin Lin
694a20cc335742716e932309
Xi Zhao
694a20cc335742716e93230a
Yuchi Xu
694a20cc335742716e93230b
Wenbo Su
694a20cc335742716e93230c
Junchi Yan
694a20cc335742716e93230d
Bo Zheng
Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs
Exploration capacity shapes both inference-time performance and reinforcement learning (RL) training for large (vision-) language models, as stochastic sampling often yields redundant reasoning paths with little high-level diversity. This paper proposes Reasoning Palette, a novel latent-modulation framework that endows the model with a stochastic latent variable for strategic contextualization, guiding its internal planning prior to token generation. This latent context is inferred from the mean-pooled embedding of a question-answer pair via a variational autoencoder (VAE), where each sampled latent potentially encodes a distinct reasoning context. During inference, a sampled latent is decoded into learnable token prefixes and prepended to the input prompt, modulating the model's internal reasoning trajectory. In this way, the model performs internal sampling over reasoning strategies prior to output generation, which shapes the style and structure of the entire response sequence. A brief supervised fine-tuning (SFT) warm-up phase allows the model to adapt to this latent conditioning. Within RL optimization, Reasoning Palette facilitates structured exploration by enabling on-demand injection for diverse reasoning modes, significantly enhancing exploration efficiency and sustained learning capability. Experiments across multiple reasoning benchmarks demonstrate that our method enables interpretable and controllable control over the (vision-) language model's strategic behavior, thereby achieving consistent performance gains over standard RL methods.
Reasoning Palette enhances large language models by using a latent-modulation framework to guide internal planning and improve both inference and reinforcement learning performance.
latent-modulation framework stochastic latent variable strategic contextualization variational autoencoder (VAE) latent context mean-pooled embedding token prefixes internal reasoning trajectory supervised fine-tuning (SFT) structured exploration diverse reasoning modes reinforcement learning (RL) strategic behavior