MusicGen is a single-stage autoregressive transformer model developed by Meta AI's FAIR team, designed for high-quality music generation. The model can generate music based on text descriptions or audio prompts, offering a simpler approach compared to previous cascading or hierarchical models. Published in the paper "Simple and Controllable Music Generation", MusicGen represents a significant advancement in AI-powered music creation.
Architecture and Technical Details
The model operates using a 32kHz EnCodec tokenizer with four codebooks sampled at 50 Hz, generating all codebooks in a single pass. Unlike models like MusicLM, MusicGen bypasses the need for a self-supervised semantic representation, achieving 50 autoregressive steps per second of audio through parallel prediction with a small delay between codebooks. The architecture includes codebook projection, positional embedding, and a causal self-attention block with cross-attention for conditioning.
Text conditioning is implemented using a T5 text encoder, which can be optionally combined with unsupervised melody conditioning through chromagram analysis. The model employs classifier-free guidance during sampling and uses Flash Attention for improved efficiency. For stereo audio generation, the model applies codebook interleaving patterns to left and right channels independently, achieving high-quality stereo output with minimal computational overhead.
Model Variants and Training
Several MusicGen variants exist in the model family:
- Small (300M parameters)
- Medium (1.5B parameters)
- Large (3.3B parameters)
- Corresponding melody-guided versions for each size
- Stereo versions fine-tuned with 200k updates
The model was trained on an extensive dataset of 20,000 hours of licensed music from Meta's Sound Collection, Shutterstock, and Pond5. The training data was preprocessed to remove vocals using source separation via HT-Demucs. All audio was sampled at 32 kHz and included metadata such as genre, BPM, and tags.
Performance and Evaluation
MusicGen's performance has been extensively evaluated using both automatic metrics and human assessments. The 3.3B parameter variant achieved a subjective quality rating of 84.8/100 on the MusicCaps benchmark, surpassing previous models like MusicLM (80.5/100) and Mousai (76.11/100).
Evaluation metrics included:
- Frechet Audio Distance
- Kullback-Leibler Divergence
- Text consistency
- Chroma cosine similarity
- Human assessments of quality, text relevance, and melody adherence
While the model shows promising results, it does have some limitations, particularly in:
- Realistic vocal generation
- Cross-lingual performance
- Consistent generation across styles and cultures
- Occasional silence at song endings
Licensing and Availability
The code for MusicGen is released under the MIT license, while the model weights are licensed under CC-BY-NC 4.0. The model and its variants are available through the Audiocraft GitHub repository, and the project was officially released in October 2023.
Reference Links
