OpenHermes-2.5-Mistral-7B represents a significant advancement in open-source language models, building upon the successful OpenHermes series. This fine-tuned version of the Mistral architecture demonstrates substantial improvements across multiple benchmarks while maintaining efficient resource usage.
Architecture and Training
The model is built on the Mistral 7B architecture and was trained on approximately 1,000,000 entries. The training data primarily consists of GPT-4 generated content, supplemented with high-quality data from open datasets. A notable improvement in this version is the inclusion of additional code datasets, which unexpectedly enhanced performance on non-code benchmarks as well.
The training process involved extensive filtering and format conversion to ShareGPT and then ChatML formats. The model utilizes ChatML as its prompt format, enabling structured multi-turn conversations and maintaining compatibility with OpenAI endpoints. A key feature is the support for system prompts, allowing for more nuanced instructions across multiple conversation turns.
Performance and Benchmarks
OpenHermes-2.5-Mistral-7B shows significant performance improvements compared to its predecessors and other Mistral fine-tunes. The model achieved notable gains across several key benchmarks:
- HumanEval score (evaluated by Glaive) improved from 43% to 50.7% at Pass 1
- Substantial improvements in TruthfulQA, AGIEval, and the GPT4All suite
- Overall increase in average total score across benchmarks
The model demonstrates particular strength in coding tasks, as shown by its improved HumanEval scores. This improvement came somewhat surprisingly through the addition of code datasets, which enhanced performance across both code and non-code tasks.
Capabilities and Applications
OpenHermes-2.5-Mistral-7B exhibits versatile capabilities across different types of tasks. Here are some key application areas:
The model excels at:
- Complex philosophical discussions and self-reflection
- Detailed technical explanations and code generation
- Role-playing and creative writing
- Recipe generation and practical instructions
The model is available in various quantized formats (GGUF, GPTQ, AWQ, EXL2) to accommodate different deployment scenarios and hardware configurations.
References and Resources
- Model Repository - Main model page and documentation
- GGUF Quantized Model - Optimized version for efficient deployment
- Transformers Chat Templating - Documentation for chat format implementation
- Axolotl Github - Training framework documentation
- LMSys Chat Website - Interactive demo platform
Image Captions:
- HumanEval Scores - Comparative performance metrics for code generation capabilities
- Benchmark Comparisons - Model performance across multiple evaluation metrics
- AI Conversation Example - Demonstration of philosophical discussion capabilities
- Recipe Generation Example - Showcase of practical instruction generation
