DeepSeek VL2 represents a significant advancement in vision-language modeling, introducing a series of large Mixture-of-Experts (MoE) Vision-Language Models that build upon its predecessor, DeepSeek-VL. The model family consists of three variants with varying capabilities and parameter counts, designed to accommodate different computational requirements while maintaining high performance standards.
Architecture and Technical Design
The architecture follows a LLaVA-style design, incorporating three main components: a SigLIP-SO400M-384 vision encoder, a vision-language adaptor, and a DeepSeekMoE Large Language Model. A key innovation is the implementation of a dynamic tiling vision encoding strategy, which efficiently handles high-resolution images with varying aspect ratios. This approach divides high-resolution images into tiles for processing by the vision encoder, significantly improving performance on tasks requiring ultra-high resolution processing, such as visual grounding and document analysis.
The model utilizes the Multi-head Latent Attention (MLA) mechanism, which compresses the Key-Value cache for more efficient inference and higher throughput. This technical advancement, combined with the MoE architecture, allows DeepSeek VL2 to achieve competitive or state-of-the-art performance while using similar or fewer activated parameters compared to other open-source models, as detailed in the research paper.
Model Variants and Capabilities
The DeepSeek VL2 family includes three distinct variants:
- DeepSeek-VL2-Tiny: Features 1.0B activated parameters
- DeepSeek-VL2-Small: Incorporates 2.8B activated parameters
- DeepSeek-VL2: The largest variant with 4.5B activated parameters
All variants maintain a 4096 sequence length and excel in various tasks, including:
- Visual question answering
- Optical character recognition (OCR)
- Document/table/chart understanding
- Visual grounding
- Multi-image processing (up to two images with tiling, three or more with direct padding to 384x384)
Training and Performance
The training process involved three comprehensive stages:
-
Vision-language alignment using ShareGPT4V
-
Vision-language pretraining on a diverse dataset combining:
- Interleaved image-text data (WIT, WikiHow, OBELICS, Wanjuan)
- Image captioning data
- OCR data (LaTeX OCR, RenderedText)
- Visual question answering data
- Visual grounding data
- Grounded conversation data
-
Supervised fine-tuning using various datasets covering tasks from general VQA to visual grounding
The model demonstrates exceptional performance across multiple benchmarks, including DocVQA, ChartQA, InfoVQA, TextVQA, OCRBench, AI2D, MMMU, MMStar, MathVista, MME, MMBench, MMBench-V1.1, MMT-Bench, and RealWorldQA. For optimal generation quality, a temperature setting of T ≤ 0.7 is recommended.
Licensing and Availability
The DeepSeek VL2 ecosystem is distributed under a dual licensing structure:
Commercial use is supported under these licensing terms. The models were initially released on December 13th, 2024, with additional features including a Gradio demo, incremental prefilling, and VLMEvalKit support added on December 25th, 2024.
Reference Links
