Llama 2 7B is a large language model (LLM) developed by Meta AI, released on July 18, 2023. It's part of the Llama 2 family of models, which includes variants ranging from 7B to 70B parameters. The model utilizes an optimized transformer architecture and is designed for auto-regressive text generation. Unlike its larger 70B parameter sibling, the 7B model does not employ Grouped-Query Attention (GQA).
The model was trained between January and July 2023 on approximately 2 trillion tokens of publicly available online data, explicitly excluding Meta user data. Training utilized a learning rate of 3.0 x 10^-4 and a global batch size of 4M tokens. The model supports a context length of 4k tokens, making it suitable for processing longer text sequences.
Capabilities and Performance
Llama 2 7B demonstrates significant improvements over its predecessor, particularly in instruction-following capabilities and reasoning abilities. The model has been benchmarked across various tasks, showing strong performance in:
- Code generation (HumanEval and MBPP)
- Commonsense reasoning
- World knowledge
- Reading comprehension
- Mathematical reasoning (GSM8K and MATH)
Performance evaluations primarily focus on English language tasks, and the model may not generalize well to other languages. Safety assessments using TruthfulQA and ToxiGen benchmarks indicate improvements in toxicity reduction compared to Llama 1.
Variants and Applications
The model is available in two main variants:
- A pretrained foundation model adaptable to various natural language generation tasks
- A fine-tuned chat model optimized for assistant-like applications
The fine-tuned versions utilize supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to enhance helpfulness and safety. For optimal chat performance, specific formatting is recommended, including the use of INST and <<SYS>> tags, along with appropriate BOS and EOS tokens.
Compared to its larger siblings (13B and 70B parameters), the 7B model offers a balance between performance and resource efficiency, making it more suitable for deployment on less powerful hardware while maintaining reasonable capabilities across various tasks.
Licensing and Usage Guidelines
Llama 2 7B is available under a custom commercial license from Meta, permitting both research and commercial use. Users must adhere to the Acceptable Use Policy, which prohibits generating illegal content or violating user rights. The model is static, trained on an offline dataset, with Meta planning future safety improvements based on community feedback.
References
