The Phi-3.5 Mini Instruct model represents a significant advancement in small language models (SLMs), offering powerful capabilities in a compact 3.8B parameter package. Released in August 2024 as an update to the June 2024 version, this model combines efficiency with state-of-the-art performance across multiple languages and tasks.
Architecture and Training
The model builds upon the foundation of the larger Phi-3 family, utilizing a combination of synthetic data and carefully filtered publicly available websites that emphasize high-quality, reasoning-dense information. The training process involved approximately 3.4 trillion tokens from various sources, followed by multiple optimization stages including supervised fine-tuning (SFT), proximal policy optimization (PPO), and direct preference optimization (DPO).
The model's implementation leverages PyTorch, the Hugging Face Transformers library, and Flash Attention (requiring compatible NVIDIA GPUs, with fallback options available). Development followed Microsoft's Responsible AI Standard, incorporating robust safety post-training strategies using both open-source and proprietary datasets.
Capabilities and Performance
A standout feature of Phi-3.5 Mini Instruct is its impressive 128K token context length, significantly surpassing the 8K limit of comparable models like the Gemma-2 family. This extended context enables superior performance on long-document tasks such as summarization and question answering.
The model demonstrates strong multilingual capabilities, supporting 22 languages including Arabic, Chinese, Czech, Danish, Dutch, English, Finnish, French, German, Hebrew, Hungarian, Italian, Japanese, Korean, Norwegian, Polish, Portuguese, Russian, Spanish, Swedish, Thai, Turkish, and Ukrainian. Compared to its predecessor, Phi-3 Mini, it shows a 25-50% performance improvement in languages like Arabic, Dutch, Finnish, Polish, Thai, and Ukrainian.
In benchmark testing, Phi-3.5 Mini Instruct competes effectively with larger models across various tasks. It shows particularly strong performance on the RULER and RepoQA benchmarks, demonstrating its capabilities in long-context understanding and code comprehension. While its smaller size does limit its factual knowledge compared to larger models, its performance-to-size ratio makes it an efficient choice for many applications.
Model Family and Variants
The Phi-3.5 family includes several specialized variants alongside the Mini Instruct model:
- Phi-3.5-MoE-instruct: A Mixture-of-Experts model offering enhanced capabilities
- Phi-3.5-vision-instruct: Specialized for visual tasks and multi-modal applications
- Phi-3.5-mini-instruct-onnx: An ONNX-optimized version for improved inference performance
The model is released under the MIT license, making it accessible for both research and commercial applications. For optimal performance in multi-lingual knowledge-intensive tasks, a Retrieval-Augmented Generation (RAG) setup is recommended.
References
