Light-R1:从零开始及超越的长链推理COT课程训练、DPO与强化学习方法

Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond

摘要 Abstract

本文介绍了Light-R1,这是一个开源套件,用于通过可重复且成本效益高的方法训练长推理模型。鉴于DeepSeek-R1系列所使用的数据具有专有性质,我们开发了一种仅依赖公共数据和模型的替代方法。我们的课程训练逐步增加数据难度,并结合多阶段后训练。我们的Light-R1-32B模型基于Qwen2.5-32B-Instruct训练,其在数学推理方面优于DeepSeek-R1-Distill-Qwen-32B。实验结果表明,当不同训练阶段有独特且多样化的数据集可用时,这种课程方法变得更加有效:使用来自我们课程数据集的3,000个挑战性示例对DeepSeek-R1-Distilled模型(由DeepSeek团队在其专有数据上预调优)进行微调,产生了最先进的7B和14B模型,而32B模型Light-R1-32B-DS的表现与QwQ-32B和DeepSeek-R1相当。此外,我们将工作扩展到对长推理模型应用GRPO。我们的最终Light-R1-14B-DS在数学领域取得了14B模型中的最佳性能,AIME24和AIME25得分分别为74.0和60.2,超过了多个32B模型和DeepSeek-R1-Distill-Llama-70B。尽管主要专注于数学训练,Light-R1-14B-DS在跨领域泛化方面也表现出色。Light-R1代表了在使复杂推理模型更易于实际应用方面的重要进展。我们的模型、训练数据和代码已发布在https://github.com/Qihoo360/Light-R1。

This paper introduces Light-R1, an open-source suite for training long reasoning models using reproducible and cost-effective methodology. Given the proprietary nature of data used in the DeepSeek-R1 series, we develop an alternative approach leveraging exclusively public data and models. Our curriculum training progressively increases data difficulty, combined with multi-staged post-training. Our Light-R1-32B model, trained from Qwen2.5-32B-Instruct, outperforms DeepSeek-R1-Distill-Qwen-32B in math reasoning. Experimental results show that this curriculum approach becomes more effective when distinct, diverse datasets are available for different training stages: fine-tuning DeepSeek-R1-Distilled models (pre-tuned by DeepSeek team on proprietary data) with 3,000 challenging examples from our curriculum dataset yielded state-of-the-art 7B and 14B models, while the 32B model, Light-R1-32B-DS performed comparably to QwQ-32B and DeepSeek-R1. Furthermore, we extend our work by applying GRPO on long reasoning models. Our final Light-R1-14B-DS achieves SOTA performance among 14B models in math, with AIME24 \& 25 scores of 74.0 and 60.2 respectively, surpassing many 32B models and DeepSeek-R1-Distill-Llama-70B. Despite math-focused training, Light-R1-14B-DS demonstrates strong cross-domain generalization. Light-R1 represents a significant advancement in making sophisticated reasoning models more accessible and implementable in real-world applications. Our models, training data and code have been made available at https://github.com/Qihoo360/Light-R1.

Light-R1:从零开始及超越的长链推理COT课程训练、DPO与强化学习方法 - arXiv