Qwen 2.5 Math 72B represents a significant advancement in mathematical language models, building upon the previous Qwen2-Math series. Released in September 2024, this specialized large language model (LLM) is designed to solve complex mathematical problems in both English and Chinese languages. The model utilizes the Qwen2ForCausalLM architecture and is stored using the Safetensors format.
The model family includes several variants:
- Base models: Qwen2.5-Math-1.5B/7B/72B
- Instruction-tuned models: Qwen2.5-Math-1.5B/7B/72B-Instruct
- Mathematical reward model: Qwen2.5-Math-RM-72B
A key architectural innovation is the integration of both Chain-of-Thought (CoT) and Tool-Integrated Reasoning (TIR) methods, expanding upon its predecessor which only supported CoT for English mathematics problems. This dual-reasoning approach enables more sophisticated problem-solving capabilities across both languages.
Training and Capabilities
The model was trained on the expanded Qwen Math Corpus v2, which contains over 1 trillion tokens - a significant increase from the previous version's 700 billion tokens. The dataset incorporates:
- Synthesized data from Qwen2-Math-72B-Instruct
- High-quality mathematical content from various sources
- Initialization from the Qwen2.5 base model
Qwen 2.5 Math 72B demonstrates exceptional capabilities in:
- Complex mathematical problem-solving using both CoT and TIR approaches
- Bilingual mathematical reasoning (English and Chinese)
- Precise computation and symbolic manipulation
- Algorithmic reasoning tasks
Performance and Benchmarks
The model family, particularly the 72B instruction-tuned variant, has achieved state-of-the-art results across multiple benchmarks. Notable achievements include:
- MATH benchmark performance:
- Qwen2.5-Math-1.5B-Instruct: 79.7% accuracy with TIR
- Qwen2.5-Math-7B-Instruct: 85.3% accuracy with TIR
- Qwen2.5-Math-72B-Instruct: 87.8% accuracy with TIR
- With RM@8: 92.9% accuracy, surpassing some closed-source models
The model excels in various other benchmarks:
- GSM8K
- MMLU-STEM (English)
- CMATH
- GaoKao Math Cloze
- GaoKao Math QA (Chinese)
- OlympiadBench
- CollegeMath
- AIME 2024
- AMC 2023
Usage and Implementation
The model requires the Transformers library (version 4.37.0 or later) for implementation. The base model (Qwen2.5-Math-72B) is optimized for completion and few-shot learning tasks, while the instruction-tuned variant (Qwen2.5-Math-72B-Instruct) is designed for conversational interactions.
For optimal performance, the following parameter values are recommended when using different sampling methods:
- Maj@8 sampling
- RM@8 sampling
The model is specifically designed for mathematical problem-solving and is not recommended for general-purpose tasks.
References
- Qwen2-Math Blog Post - Detailed information about the model family
- Qwen2.5-Math GitHub Repository - Official code repository
- Qwen2.5-Math Blog Post - Technical details and performance metrics
- Qwen Documentation - Comprehensive documentation
- Math Evaluation Harness - Evaluation framework used for benchmarking
- Qwen-Agent Repository - Demo supporting TIR mode
