DeepSeek Coder V2 represents a significant advancement in open-source code language models, built upon the innovative DeepSeekMoE framework. This Mixture-of-Experts (MoE) architecture model has been further pre-trained from an intermediate DeepSeek-V2 checkpoint using an additional 6 trillion tokens, marking a substantial improvement over its predecessor.
Architecture and Training
The model employs a sophisticated MoE architecture available in two variants:
- DeepSeek-Coder-V2-Lite: 16B parameters (2.4B active)
- DeepSeek-Coder-V2: 236B parameters (21B active)
The training data composition includes:
- 60% source code (1.17 trillion tokens from GitHub and Common Crawl)
- 10% math corpus (221 billion tokens from Common Crawl)
- 30% natural language corpus (sampled from DeepSeek-V2's training data)
The training process utilized Next-Token-Prediction and Fill-In-Middle (FIM) objectives, with FIM enabled specifically for the 16B variant. The model underwent additional fine-tuning using an instruction training dataset and was aligned using Group Relative Policy Optimization (GRPO), incorporating compiler feedback and test cases for code-related tasks.
Features and Capabilities
DeepSeek Coder V2 introduces several notable improvements over its predecessor, DeepSeek-Coder-33B:
-
Extended Language Support: Coverage expanded from 86 to 338 programming languages, as detailed in the supported languages list.
-
Increased Context Length: Context window expanded from 16K to 128K tokens, enabling handling of more complex coding tasks.
-
Enhanced Performance: The model demonstrates superior capabilities in:
- Code generation (HumanEval and MBPP+ benchmarks)
- Mathematical reasoning (GSM8K, MATH, AIME 2024, Math Odyssey)
- Code completion (RepoBench, Single-Line Infilling)
- Code fixing (Defects4J, SWE-Bench, Aider)
- Code understanding/reasoning (CRUXEval)
Performance and Benchmarks
According to the research paper, DeepSeek Coder V2 achieves remarkable performance across various benchmarks, often surpassing both open-source and closed-source competitors. The model demonstrates:
- State-of-the-art results among open-source models in code generation
- Comparable performance to top closed-source models in mathematical reasoning
- Strong capabilities in code completion and fixing tasks
- Maintained general language performance comparable to DeepSeek-V2
While the model shows particular strength in reasoning tasks, it exhibits a relative weakness in knowledge-intensive tasks compared to DeepSeek-V2. There remains a gap in instruction-following capabilities compared to state-of-the-art closed-source models, particularly in complex scenarios.
Implementation and Licensing
The model is released under permissive licensing terms, with the code available under the MIT License and the model under a separate Model License. Both research and commercial use are permitted.
Key References:
