DeepSeek-R1-Distill-Qwen-32B is a distilled language model that represents a significant advancement in efficient AI model design. Built upon the Qwen2.5-32B base model, it inherits its core architecture while incorporating distilled reasoning capabilities from the larger DeepSeek-R1 model. With 32.8 billion parameters in BF16 format and a context length of 32,768 tokens, it achieves state-of-the-art results for dense models.
The model is part of the broader DeepSeek-R1 family, which employs a novel multi-stage training pipeline. As detailed in the DeepSeek R1 Paper, the original DeepSeek-R1 uses a Mixture-of-Experts (MoE) architecture with 37B activated parameters and 671B total parameters. The distillation process successfully transfers the reasoning capabilities from this larger model to the more efficient 32B parameter version.
Training and Capabilities
The training process for DeepSeek-R1-Distill-Qwen-32B represents a significant improvement over its predecessor, DeepSeek-R1-Zero. While DeepSeek-R1-Zero relied solely on reinforcement learning (RL), the newer model incorporates a more comprehensive pipeline that includes cold-start data before RL. This approach helps improve reasoning performance while avoiding issues like repetition and poor readability that were present in the earlier model.
The model demonstrates exceptional performance across various benchmarks, surpassing OpenAI's o1-mini on multiple metrics. Notable achievements include strong performance on:
- MMLU-Redux (EM)
- MMLU-Pro (EM)
- DROP (3-shot F1)
- AIME 2024
- MATH-500
- LiveCodeBench
Implementation and Usage
For optimal performance, DeepSeek-R1-Distill-Qwen-32B should be run with specific parameter settings:
- Temperature: 0.5 to 0.7 (0.6 recommended)
- Top-p value: 0.95 for sampling tasks
- Maximum generation length: 32,768 tokens
The model can be deployed locally using frameworks like vLLM or SGLang. When working with mathematical problems, users should include specific directives in prompts, such as "put your final answer within \boxed". For reliable evaluation, it's recommended to average results across multiple runs.
Licensing and Availability
The code and model weights are released under the MIT License, allowing for commercial use and derivative works. However, users must comply with the underlying base model licenses:
- Apache 2.0 license for Qwen components
- Llama license for Llama-based components
Model Family and Variants
DeepSeek AI has released several distilled models in the R1 family, ranging from 1.5B to 70B parameters. These include:
- DeepSeek-R1-Distill-Qwen-1.5B
- DeepSeek-R1-Distill-Qwen-7B
- DeepSeek-R1-Distill-Llama-8B
- DeepSeek-R1-Distill-Qwen-14B
- DeepSeek-R1-Distill-Qwen-32B (this model)
- DeepSeek-R1-Distill-Llama-70B
Each variant offers different trade-offs between performance and resource requirements, with the 32B model representing a sweet spot for many applications.
References
