DeepSeek-R1-Distill-Llama-8B represents a significant advancement in efficient language model design, offering strong reasoning capabilities in a relatively compact 8B parameter architecture. This model is part of the broader DeepSeek-R1 family, which includes various distilled versions ranging from 1.5B to 70B parameters based on both Llama and Qwen architectures.
Architecture and Development
The model is a distilled version derived from the much larger DeepSeek-R1, which features a Mixture-of-Experts (MoE) architecture with 671B total parameters (37B activated). The development process involved multiple stages, beginning with the creation of DeepSeek-R1-Zero, which was trained purely through reinforcement learning. While DeepSeek-R1-Zero showed promising reasoning capabilities, it suffered from issues like repetition and poor readability.
To address these limitations, researchers developed DeepSeek-R1 using a novel approach that incorporated a cold-start data phase before reinforcement learning. This improved model was then distilled into smaller, more efficient versions, including the 8B parameter variant built on the Llama-3.1-8B base model.
Performance and Capabilities
DeepSeek-R1-Distill-Llama-8B demonstrates impressive performance across various benchmarks, particularly in reasoning, mathematics, and code-related tasks. The model shows comparable results to OpenAI's o1 model in several areas, though specific performance metrics vary by task.
The model performs optimally with specific parameter settings:
- Recommended temperature range: 0.5 to 0.7
- Suggested top-p value: 0.95
These settings help mitigate potential issues such as infinite loops or incoherent outputs while maintaining high-quality reasoning capabilities.
Model Family and Variants
The DeepSeek-R1 distilled family includes several variants:
- DeepSeek-R1-Distill-Qwen-1.5B
- DeepSeek-R1-Distill-Qwen-7B
- DeepSeek-R1-Distill-Llama-8B
- DeepSeek-R1-Distill-Qwen-14B
- DeepSeek-R1-Distill-Qwen-32B
- DeepSeek-R1-Distill-Llama-70B
Each variant leverages either the Qwen or Llama architecture as its base model, with the 8B Llama variant offering a balanced compromise between performance and computational efficiency. The entire family demonstrates that smaller models can achieve high reasoning capabilities when trained with knowledge distilled from more powerful predecessors.
Licensing and Availability
The model is open-sourced under multiple licenses:
- Code: MIT License
- Model weights: Custom DeepSeek license
- Base models: Respective Llama and Qwen licenses
The model can be accessed through Hugging Face and run locally using tools like vLLM.
References
