Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18-20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering-artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models. Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts.
696df8e73f1837bfb8970afa
Lecheng Yan
696df8e73f1837bfb8970afb
Ruizhe Li
admin_assigned
2026-01-20T09:47:51.813Z
633f536250d83f5065d28f6d
https://cdn-avatars.huggingface.co/v1/production/uploads/633f536250d83f5065d28f6d/eZCdJJpU6OJFYUWYoUAVn.jpeg
Ruizhe Li
rzdiversity
user
rzdiversity
696df8e73f1837bfb8970afc
Guanhua Chen
696df8e73f1837bfb8970afd
Qing Li
696df8e73f1837bfb8970afe
Jiahui Geng
admin_assigned
2026-01-20T09:48:21.466Z
6826332f82eb9b05042e453b
/avatars/e0d8cc3f43364c7a927b3b9ae4ac74cd.svg
Jiahui Geng
JiahuiGengNLP
user
JiahuiGengNLP
696df8e73f1837bfb8970aff
Wenxi Li
admin_assigned
2026-01-20T09:48:33.704Z
67daa551e0835e51e8e79283
/avatars/1ae54760a05b495ae7d60236c236cc07.svg
wenxi li
kksinn
user
kksinn
696df8e73f1837bfb8970b00
Vincent Wang
696df8e73f1837bfb8970b01
Chris Lee
admin_assigned
2026-01-20T09:48:40.220Z
625b7ed33f54d7f4e5dbcf8b
/avatars/8b301202fec42d5d4298211f7d7f3d63.svg
Lee
ChrisLee
user
ChrisLee
Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs
Reinforcement Learning with Verifiable Rewards (RLVR) is highly effective for enhancing LLM reasoning, yet recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards. We investigate this phenomenon and identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades, suggesting the model is bypassing reasoning in favor of memorization. Using Path Patching, Logit Lens, JSD analysis, and Neural Differential Equations, we uncover a hidden Anchor-Adapter circuit that facilitates this shortcut. We localize a Functional Anchor in the middle layers (L18-20) that triggers the retrieval of memorized solutions, followed by Structural Adapters in later layers (L21+) that transform representations to accommodate the shortcut signal. Finally, we demonstrate that scaling specific MLP keys within this circuit allows for bidirectional causal steering-artificially amplifying or suppressing contamination-driven performance. Our results provide a mechanistic roadmap for identifying and mitigating data contamination in RLVR-tuned models. Code is available at https://github.com/idwts/How-RLVR-Activates-Memorization-Shortcuts.
Spurious rewards in reinforcement learning with verifiable rewards trigger a memorization shortcut in LLMs, identified through neural circuit analysis and causal steering techniques.
Reinforcement Learning with Verifiable Rewards perplexity answer-token perplexity prompt-side coherence Path Patching Logit Lens JSD analysis Neural Differential Equations Anchor-Adapter circuit Functional Anchor Structural Adapters MLP keys causal steering