Nous Hermes 13b represents a significant advancement in open-source language models, built upon the foundation of Llama 13b through extensive fine-tuning. The model emerged from a collaborative effort between Nous Research, Teknium, Karan4D, Redmond AI, and other contributors, showcasing the power of community-driven AI development.
The model's architecture leverages the base Llama 13b structure, enhanced through fine-tuning on over 300,000 instructions. Training was conducted on high-performance hardware, specifically an 8x A100 80GB DGX machine, with a sequence length of 2000 over a period exceeding 50 hours.
Training Data and Methodology
The training dataset for Nous Hermes 13b was carefully curated from multiple high-quality sources. The primary component consisted of synthetic GPT-4 outputs, including:
- GPTeacher datasets
- General and roleplay datasets (versions 1 & 2)
- Code instruction datasets
- Nous Instruct & PDACTL (unpublished)
- CodeAlpaca
- Evol_Instruct Uncensored
- GPT4-LLM
- Unnatural Instructions
Additional specialized training data incorporated Camel-AI's Biology, Physics, Chemistry, and Math Datasets, along with Airoboros' GPT-4 Dataset. This diverse training corpus contributes to the model's broad capabilities across different domains.
Performance and Capabilities
Nous Hermes 13b demonstrates exceptional performance across various benchmarks, achieving results comparable to GPT-3.5-turbo. The model particularly excels in several key areas, ranking first in multiple benchmark categories:
- ARC-c
- ARC-e
- Hellaswag
- OpenBookQA
It also achieved a notable second-place ranking in Winogrande, according to GPT4all's benchmark list.
Key features of the model include:
- Long-form response generation
- Low hallucination rate
- Unrestricted output compared to OpenAI's models
- Versatile application potential
Implementation and Usage
The model implements the Alpaca prompt format, supporting two primary structures:
### Instruction:
### Response:
or
### Instruction:
### Input:
### Response:
Practical applications of Nous Hermes 13b have been demonstrated through various implementations, including chatbots and roleplaying systems. The model's capabilities can be explored through the Alpaca Discord Bot and Alpaca Roleplay Discord Bot, which showcase its versatility in real-world applications.
Licensing and Availability
Nous Hermes 13b is released under the GPL license, making it accessible for research and development purposes. The model is available for download through Hugging Face, with ongoing development plans including GGML and GPTQ 4bit quantizations, as well as comprehensive benchmark testing.
References
- Alpaca Discord Bot - Example implementation of a chatbot using Hugging Face Transformers and Discord
- Alpaca Roleplay Discord Bot - Implementation example for roleplaying applications
- Base Model Documentation - Technical documentation for base model specifications
