DeepSeek Coder V2 Lite represents a significant advancement in open-source code language models, combining efficient architecture with powerful capabilities. As part of the DeepSeek-Coder-V2 family, it implements a Mixture-of-Experts (MoE) architecture that achieves an impressive balance between performance and computational efficiency.
Architecture and Training
The model features 16B total parameters but maintains only 2.4B active parameters through its MoE architecture, making it significantly more resource-efficient than traditional models. It was developed through further pre-training from an intermediate checkpoint of DeepSeek-V2, incorporating an additional 6 trillion tokens to the existing 4.2 trillion token foundation.
The training data composition was carefully curated:
- 60% source code (1,170B tokens from GitHub and CommonCrawl)
- 10% math corpus (221B tokens from CommonCrawl)
- 30% natural language corpus (sampled from DeepSeek-V2's training data)
The model employs several advanced training techniques, including:
- Fill-In-Middle (FIM) training at a 0.5 rate for enhanced code completion
- Cosine learning rate scheduler
- AdamW optimizer
- Context length extension to 128K tokens using Yarn
- Supervised fine-tuning and reinforcement learning with Group Relative Policy Optimization (GRPO)
Capabilities and Performance
DeepSeek Coder V2 Lite demonstrates impressive capabilities across various benchmarks, particularly in code-related tasks. The model supports an extensive range of 338 programming languages, a significant expansion from its predecessor's 86 languages. Its 128K token context length enables handling of larger code bases and more complex problems.
In benchmark evaluations:
- Code Generation:
- HumanEval: 81.1% in Python (compared to GPT-4-Turbo's 88.2%)
- MBPP+: 65.6% average accuracy (compared to GPT-4-Turbo's 72.2%)
- Code Completion: Strong performance in RepoBench and Fill-in-the-Middle benchmarks, often matching larger models
- Code Fixing: Competitive results on Defects4J, SWE-Bench, and Aider benchmarks
- Mathematical Reasoning: Solid performance on GSM8K, MATH, AIME 2024, and Math Odyssey
- General Language Tasks: Comparable to DeepSeek-V2, with slight improvements in reasoning
Model Family Comparison and Licensing
DeepSeek Coder V2 Lite is the efficient variant of the larger DeepSeek-Coder-V2 model family. While the full version boasts 236B total parameters with 21B active parameters, the Lite variant maintains strong performance with significantly reduced resource requirements. This makes it particularly suitable for environments where computational resources are constrained.
The model is released under multiple licenses:
References
