DeepSeek R1 Distill Qwen 1.5B represents a significant achievement in model compression and efficient AI design, offering powerful reasoning capabilities in a compact form factor. This distilled model is part of the broader DeepSeek R1 family, which includes six distilled variants ranging from 1.5B to 70B parameters based on the Qwen and Llama architectures.
Architecture and Development
The model is built upon the Qwen2.5-Math-1.5B architecture and is derived from the much larger DeepSeek R1 model, which features a Mixture-of-Experts (MoE) architecture with 671 billion total parameters (37 billion activated). The development process involved a sophisticated training pipeline that combined reinforcement learning (RL) with supervised fine-tuning (SFT), as detailed in the DeepSeek R1 paper.
The training methodology evolved from the earlier DeepSeek-R1-Zero model, which relied solely on RL. The improved approach incorporated cold-start data before the RL stages, which helped mitigate issues like repetition and poor readability while maintaining strong reasoning capabilities. This refined process has enabled the distilled models, including the 1.5B variant, to achieve state-of-the-art results for dense models of their size.
Performance and Capabilities
DeepSeek R1 Distill Qwen 1.5B demonstrates impressive performance across various benchmarks, particularly in mathematics, coding, and reasoning tasks. While specific benchmark numbers for the 1.5B model aren't provided, the DeepSeek R1 family's performance is notable, with the larger DeepSeek R1 achieving results comparable to OpenAI's o1 model.
For context, the full-size DeepSeek R1 model achieves:
- 79.8% Pass@1 on AIME 2024
- 97.3% on MATH-500
- Performance exceeding 96.3% of human participants on Codeforces
- 90.8% on MMLU
- 71.5% on GPQA Diamond
The distilled models, including the 1.5B variant, retain significant portions of these capabilities while requiring far fewer computational resources. For instance, the 7B distilled version achieves 55.5% on AIME 2024, surpassing the larger QwQ-32B-Preview model's 50% score, demonstrating the effectiveness of the distillation process.
Implementation and Usage
The model can be run locally using vLLM with the following recommended command:
vllm serve deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B --tensor-parallel-size 2 --max-model-len 32768 --enforce-eager
For optimal results, users should:
- Set the temperature between 0.5 and 0.7 to avoid repetitive or incoherent outputs
- Include specific directives in prompts for math problems
- Average multiple test results during performance evaluation
The model uses safetensors with BF16 parameters, totaling 1,777,088,000 parameters.
Licensing and Legal Considerations
The code for DeepSeek R1 Distill Qwen 1.5B is licensed under the MIT License, allowing for commercial use and derivative works. However, it's important to note that the base models used for distillation are subject to their respective licenses:
- Qwen models: Apache 2.0 License
- Llama models: Llama 3.1/3.3 licenses
References
