KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance
RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose KnowRL (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.
69df07e2d3c84fcbf2921876
Linhao Yu
claimed_verified
2026-04-15T10:17:01.030Z
641e69355c348064a8251471
/avatars/acad3877df27ff44ea3921bb43e34d53.svg
Linhao Yu
HasuerYu
user
HasuerYu
69df07e2d3c84fcbf2921877
Tianmeng Yang
claimed_verified
2026-04-16T08:02:59.423Z
67bbe929593452cc18877606
/avatars/f50fd1cb35d628c26cf21ad0c95c55b1.svg
tmyangcs
youngtimmy
user
youngtimmy
69df07e2d3c84fcbf2921878
Siyu Ding
69df07e2d3c84fcbf2921879
Renren Jin
69df07e2d3c84fcbf292187a
Naibin Gu
69df07e2d3c84fcbf292187b
Xiangzhao Hao
claimed_verified
2026-04-16T08:03:01.556Z
64ec6025b96ff0e175728ac0
/avatars/a1acbe22b5e5105703e7912da2cfced2.svg
hxz
CUDAOUTOFMEMORY
user
CUDAOUTOFMEMORY
69df07e2d3c84fcbf292187c
Shuaiyi Nie
69df07e2d3c84fcbf292187d
Deyi Xiong
69df07e2d3c84fcbf292187e
Weichong Yin
69df07e2d3c84fcbf292187f
Yu Sun
69df07e2d3c84fcbf2921880
Hua Wu
KnowRL: Boosting LLM Reasoning via Reinforcement Learning with Minimal-Sufficient Knowledge Guidance
RLVR improves reasoning in large language models, but its effectiveness is often limited by severe reward sparsity on hard problems. Recent hint-based RL methods mitigate sparsity by injecting partial solutions or abstract templates, yet they typically scale guidance by adding more tokens, which introduce redundancy, inconsistency, and extra training overhead. We propose KnowRL (Knowledge-Guided Reinforcement Learning), an RL training framework that treats hint design as a minimal-sufficient guidance problem. During RL training, KnowRL decomposes guidance into atomic knowledge points (KPs) and uses Constrained Subset Search (CSS) to construct compact, interaction-aware subsets for training. We further identify a pruning interaction paradox -- removing one KP may help while removing multiple such KPs can hurt -- and explicitly optimize for robust subset curation under this dependency structure. We train KnowRL-Nemotron-1.5B from OpenMath-Nemotron-1.5B. Across eight reasoning benchmarks at the 1.5B scale, KnowRL-Nemotron-1.5B consistently outperforms strong RL and hinting baselines. Without KP hints at inference, KnowRL-Nemotron-1.5B reaches 70.08 average accuracy, already surpassing Nemotron-1.5B by +9.63 points; with selected KPs, performance improves to 74.16, establishing a new state of the art at this scale. The model, curated training data, and code are publicly available at https://github.com/Hasuer/KnowRL.
KnowRL is a knowledge-guided reinforcement learning framework that improves reasoning in language models by optimizing compact, interaction-aware guidance subsets through constrained subset search and addressing pruning interaction paradoxes.
reinforcement learning reasoning reward sparsity hint-based RL minimal-sufficient guidance atomic knowledge points constrained subset search pruning interaction paradox subset curation language models