WizardLM-70B-V1.0 is a large language model based on the Llama architecture, specifically using the LlamaForCausalLM implementation in PyTorch. Released on August 9, 2023, it represents a significant advancement in open-source language models, particularly in handling complex instructions. The model was developed using a novel instruction data generation technique called Evol-Instruct, which is detailed in the original WizardLM research paper.
The model's development focused on overcoming traditional limitations in instruction-following capabilities. Unlike previous approaches that relied heavily on human-generated training data, WizardLM utilizes an AI-powered evolutionary algorithm to generate its instruction dataset. This innovative approach addresses common constraints in human annotation, including cost, time, and the tendency toward simpler instructions.
Training and Methodology
The training process for WizardLM-70B involved a sophisticated approach to data generation and model fine-tuning. The Evol-Instruct methodology starts with a seed set of 52,000 instructions from the Alpaca dataset and evolves them through both in-depth and in-breadth processes. This evolution resulted in a comprehensive dataset of 250,000 instructions, with a 70,000-instruction subset used for specific comparative training.
The model uses a Vicuna-style prompt format and supports multi-turn conversations. While the exact training parameters aren't fully detailed in the sources, the research indicates significant computational resources were involved in the training process, with earlier versions requiring 70 hours across 8 V100 GPUs using the Adam optimizer and Deepspeed Zero-3.
Performance and Benchmarks
WizardLM-70B demonstrates impressive performance across various benchmarks:
- 77.6% score on GSM8K
- 92.91% score on AlpacaEval
- 50.6 pass@1 on HumanEval
Human evaluations show that WizardLM performs exceptionally well on complex instructions, even surpassing ChatGPT on high-complexity instructions (difficulty level ≥ 8). In GPT-4 automatic evaluation, the model achieved over 90% of ChatGPT's capacity on 17 out of 29 skills, though it shows some limitations in areas like code generation, math, and reasoning.
Model Family and Variants
The WizardLM family includes several variants, with WizardLM-70B-V1.0 representing the largest and most capable version. Other models in the family include:
- WizardLM-7B-V1.0
- WizardLM-13B-V1.0, V1.1, and V1.2
- WizardLM-30B-V1.0
The family also includes specialized variants:
- WizardCoder: Focused on code generation
- WizardMath: Specialized in mathematical reasoning
WizardLM-70B consistently demonstrates superior performance compared to its smaller siblings across various benchmarks, likely due to its increased model size and potentially different training techniques.
Licensing and Availability
WizardLM-70B-V1.0 is released under the Llama 2 license. The model's code and data are publicly available through the official GitHub repository.
References and Resources
