Dolphin 2.7 Mixtral 8X7B represents a significant advancement in the Dolphin model family, built upon the Mixtral-8x7b base model. This version incorporates several key improvements over its predecessors (versions 2.5 and 2.6), including Mixtral-specific performance optimizations within the transformers library and unfreezing of the gate layer. The model features a 16,000 token context window, which was fine-tuned from the base model's original 32,000 token capacity.
The model follows the Orca methodology for instruction tuning, which emphasizes learning from complex explanation traces and step-by-step reasoning processes. This approach has proven effective in improving model capabilities across various tasks, particularly in coding and reasoning scenarios.
Training and Dataset Composition
The training process for Dolphin 2.7 Mixtral 8X7B was comprehensive, spanning 3 days on 4 A100 GPUs using qLoRA and the Axolotl framework. The model completed 1.5 epochs of training on a diverse dataset compilation including:
- cognitivecomputations/dolphin
- jondurbin/airoboros-2.2.1
- cognitivecomputations/dolphin-coder
- teknium/openhermes
- ise-uiuc/Magicoder-OSS-Instruct-75K
- ise-uiuc/Magicoder-Evol-Instruct-110K
- LDJnr/Capybara
The underlying Dolphin dataset comprises approximately 1 million FLANv2 instances augmented with GPT-4 completions and 3.5 million FLANv2 instances with GPT-3.5 completions. Version 2.7 specifically reintroduced Samantha-based empathy data in response to user feedback, while replacing previous Synthia and Pure-Dove datasets with Capybara.
Features and Capabilities
The model excels particularly in coding tasks, attributed to its extensive training on coding-specific datasets. It utilizes the ChatML prompt format and is described as highly obedient, though it may require appropriate system prompt guidance due to the absence of DPO tuning.
A notable characteristic of Dolphin 2.7 is its uncensored nature, though the training data underwent filtering to reduce bias and improve compliance. Users should note that the model will respond to any request, regardless of ethical or legal implications, making it crucial to implement appropriate alignment layers before deployment in any service capacity.
Future Development and Licensing
The Dolphin project operates under the Apache 2.0 license, allowing for both commercial and non-commercial use. Future developments, particularly Dolphin 3.0, are planned to focus on enhancing:
- General chat capabilities
- Structured output functionality
- Agent capabilities (Autogen, Memgpt, Functions)
- Role-playing features
Reference Links
