Dolphin 2.6 Mistral is an open-source language model built on the Mistral-7b architecture, featuring a 16k context window. The model is part of the broader Dolphin project, which creates instruct-tuned language models inspired by Microsoft's Orca paper methodology. This particular iteration represents a significant advancement in the project's evolution, combining the capabilities of the Mistral architecture with specialized training approaches.
The model's development, sponsored by Convai, involved an intensive training process spanning three epochs over two days, utilizing four A100 GPUs. The training implemented full-weight finetuning through the Axolotl framework. A notable technical aspect is its ChatML prompt format implementation, where <|im_end|> maps to token ID 2 (identical to </s>), ensuring broad compatibility with various applications including koboldAI.
Training and Capabilities
The Dolphin project's training methodology builds upon a substantial dataset containing approximately 1 million FLANv2 examples enhanced with GPT-4 completions and 3.5 million FLANv2 examples augmented with GPT-3.5 completions. For Dolphin 2.6 Mistral specifically, the model underwent DPO tuning using the argilla/ultrafeedback-binarized-preferences-cleaned dataset.
A distinguishing characteristic of Dolphin 2.6 Mistral is its "uncensored and unbiased" nature, achieved through careful filtering of the training dataset to remove alignment and bias. However, the creators recommend implementing a custom alignment layer before deployment in production services, acknowledging the model's potential to generate responses to unethical prompts.
Performance and Benchmarks
The model's capabilities have been extensively evaluated on the Open LLM Leaderboard across multiple datasets, demonstrating strong performance across various tasks:
- AI2 Reasoning Challenge (25-shot): 65.61% normalized accuracy
- HellaSwag (10-shot): 85.48% accuracy
- MMLU (5-shot): 63.24% accuracy
- TruthfulQA (0-shot): 61.47% mc2
- Winogrande (5-shot): 78.61% accuracy
- GSM8k (5-shot): 48.75% accuracy
Future Development
The Dolphin project continues to evolve, with plans for Dolphin 3.0 focusing on expanding capabilities in several key areas:
- General chat improvements
- Structured output enhancement
- Advanced agent functionalities (including Autogen, Memgpt, and Functions)
- Enhanced role-playing capabilities
The broader Dolphin project roadmap includes releases on various base models, including Xgen 7b 8k, LLaMA 13b, MPT 30b 8k, LLaMA 33b, Falcon 40b, and LLaMA 65b, with licensing dependent on the underlying base models.
References
