The Phi-3.5-vision-instruct model represents a significant advancement in lightweight multimodal AI, combining powerful visual understanding capabilities with efficient architecture. As part of Microsoft's Phi-3.5 model family released in August 2024, this 4.2B parameter model demonstrates competitive performance against larger models while maintaining a smaller footprint.
Architecture and Technical Specifications
The model's architecture integrates several key components:
- Image encoder
- Connector
- Projector
- Phi-3 Mini language model base
With a 128K context length (in tokens), the model leverages flash attention by default, requiring specific GPU hardware like NVIDIA A100, A6000, or H100. The training process involved 500B tokens of data and took 6 days using 256 A100-80G GPUs.
Training Data and Capabilities
The model's training incorporated diverse data sources:
- High-quality filtered public web data
- Synthetic data for various skills (math, coding, reasoning)
- High-quality image-text pairs
- Supervised chat data focused on helpfulness and safety
Privacy considerations were paramount during training, with personal data being removed or scrubbed. The model excels at several key tasks:
- General image understanding
- OCR capabilities
- Chart/table comprehension
- Multi-image comparison
- Video summarization
Performance and Benchmarks
When compared to other models like LlaVA-Interleave-Qwen-7B, InternVL-2-4B, and larger models, Phi-3.5-vision-instruct shows impressive performance across various benchmarks including MMMU, MMBench, TextVQA, BLINK, and Video-MME. While GPT-4 variants sometimes outperform it, the model demonstrates particular strength in multi-frame capabilities and video summarization tasks.
For optimal performance, users should configure:
num_crops=4 for multi-frame inputsnum_crops=16 for single-frame inputs
Model Family Context
The Phi-3.5 family includes several variants:
- Phi-3.5-mini (3.8B parameters)
- Phi-3.5-vision (focused on visual tasks)
- Phi-3.5-MoE (Mixture-of-Experts model)
While Phi-3.5-mini excels at multilingual support with capabilities in over 20 languages, Phi-3.5-vision focuses specifically on visual understanding tasks. The MoE variant utilizes 16 experts with 3.8B parameters each, activating 6.6B parameters during inference for enhanced performance.
References and Additional Resources
- Phi-3 Portal - Official product portal
- Technical Report - Detailed technical documentation
- Phi-3 Cookbook - Implementation guides and examples
- Vision Finetuning Recipe - Specific guidance for vision model training
- MIT License - Model license details
- Flash-Attention Implementation - Attention mechanism details
