Qwen 2.5 14B is a decoder-only language model with 14.7 billion parameters (13.1B non-embedding parameters) spread across 48 layers. The model employs a transformer architecture incorporating several key technologies: RoPE (Rotary Position Embedding), SwiGLU (Gated Linear Unit), RMSNorm (Root Mean Square Layer Normalization), and Attention QKV bias. As detailed in the model specifications, it supports an impressive context length of 131,072 tokens.
The model is part of the broader Qwen 2.5 family, which includes variants ranging from 0.5B to 72B parameters. All models in the family were pretrained on a massive dataset of up to 18 trillion tokens, as described in the official blog post. This extensive training has contributed to significant improvements over the previous Qwen-2 generation, particularly in areas like knowledge retention, coding capabilities, and mathematical reasoning.
Capabilities and Performance
Qwen 2.5 14B demonstrates impressive capabilities across multiple domains. The model excels in instruction following, can generate text spanning over 8,000 tokens, and shows enhanced understanding and generation of structured data, particularly JSON formats. It supports over 29 languages, making it a versatile multilingual model.
Performance benchmarks show strong results, with the model outperforming comparable or larger models like Phi-3.5-MoE-Instruct and Gemma2-27B-IT. Within the Qwen 2.5 family, while the 72B variant represents the peak of performance, the 14B model maintains an excellent balance of capabilities and resource requirements. The model's MMLU score exceeds 85, its HumanEval score surpasses 85, and its MATH score is above 80, demonstrating strong performance across knowledge, coding, and mathematical reasoning tasks.
Implementation and Usage
The model requires the Hugging Face transformers library version 4.37.0 or higher for proper functionality. It's important to note that the base model is not recommended for direct conversational use - additional training through methods like Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), or continued pretraining is suggested for conversational applications.
The model is available under the Apache 2.0 license, allowing for broad use and modification. It can be deployed through various frameworks, including Hugging Face Transformers, vLLM, and Ollama, with support for tool calling functionality across these platforms.
References
