Improving Data and Reward Design for Scientific Reasoning in Large Language Models
Solving open-ended science questions remains challenging for large language models, particularly due to inherently unreliable supervision and evaluation. The bottleneck lies in the data construction and reward design for scientific post-training. We develop a large-scale, systematic data processing pipeline that transforms heterogeneous open-source science data into Dr. SCI dataset, which comprises of 1M questions across eight STEM subjects, with explicit verifiable/open-ended splits, scalable difficulty annotation, and fine-grained rubrics that operationalize evaluation for open-ended answers. Building on this dataset, we propose the Dr. SCI post-training pipeline, which redesigns the standard SFT -> RL workflow through three components: (i) Exploration-Expanding SFT, which broadens the model's reasoning pattern coverage prior to RL; (ii) Dynamic Difficulty Curriculum, which adapts training data to the model's evolving scientific capability; and (iii) SciRubric-Guided RL, which enables stable reinforcement learning on open-ended scientific questions via rubric-based evaluation with explicit answer correctness. Qwen3-4B-Base trained using Dr. SCI pipeline achieves 63.2 on GPQA-diamond and 32.4 on GPQA-general, consistently improves over strong post-trained baselines such as o1-mini and GPT-4o, demonstrating substantial gains in scientific reasoning, especially in open-ended settings.
698be5636052d3bed963099b
Zijie Chen
claimed_verified
2026-02-27T16:45:31.064Z
630a2bcf4bee441367afba10
/avatars/463a62b296887f1be8290475b91af098.svg
Zijie Chen
Zijie-chen
user
Zijie-chen
698be5636052d3bed963099c
Zhenghao Lin
698be5636052d3bed963099d
Xiao Liu
claimed_verified
2026-02-11T11:15:18.917Z
63fb6e281b4b1bd4e7ffc5be
https://cdn-avatars.huggingface.co/v1/production/uploads/63fb6e281b4b1bd4e7ffc5be/aiRu_bulgnxvEMrjipGoQ.jpeg
Xiao Liu
lx865712528
user
lx865712528
698be5636052d3bed963099e
Zhenzhong Lan
698be5636052d3bed963099f
Yeyun Gong
698be5636052d3bed96309a0
Peng Cheng
Improving Data and Reward Design for Scientific Reasoning in Large Language Models
Solving open-ended science questions remains challenging for large language models, particularly due to inherently unreliable supervision and evaluation. The bottleneck lies in the data construction and reward design for scientific post-training. We develop a large-scale, systematic data processing pipeline that transforms heterogeneous open-source science data into Dr. SCI dataset, which comprises of 1M questions across eight STEM subjects, with explicit verifiable/open-ended splits, scalable difficulty annotation, and fine-grained rubrics that operationalize evaluation for open-ended answers. Building on this dataset, we propose the Dr. SCI post-training pipeline, which redesigns the standard SFT -> RL workflow through three components: (i) Exploration-Expanding SFT, which broadens the model's reasoning pattern coverage prior to RL; (ii) Dynamic Difficulty Curriculum, which adapts training data to the model's evolving scientific capability; and (iii) SciRubric-Guided RL, which enables stable reinforcement learning on open-ended scientific questions via rubric-based evaluation with explicit answer correctness. Qwen3-4B-Base trained using Dr. SCI pipeline achieves 63.2 on GPQA-diamond and 32.4 on GPQA-general, consistently improves over strong post-trained baselines such as o1-mini and GPT-4o, demonstrating substantial gains in scientific reasoning, especially in open-ended settings.
A large-scale scientific question dataset and post-training pipeline are developed to improve open-ended science question answering through enhanced data processing and reinforcement learning with rubric-guided evaluation.
large language models open-ended science questions data construction reward design Dr. SCI dataset SFT RL exploration-expanding SFT dynamic difficulty curriculum SciRubric-Guided RL scientific reasoning GPQA-diamond GPQA-general