CogVideoX-2B is a large-scale text-to-video generation model released on August 6, 2024, designed as an entry-level option in the CogVideoX family. Based on a diffusion transformer architecture, it represents a significant advancement in text-to-video generation technology while maintaining accessibility through lower hardware requirements compared to its larger siblings.
Architecture and Technical Specifications
The model employs a 3D Causal VAE for video reconstruction and compression, combined with an expert transformer featuring expert adaptive LayerNorm for enhanced text-video alignment. As detailed in the research paper, the architecture incorporates 3D full attention to comprehensively model both temporal and spatial dimensions.
CogVideoX-2B generates videos at 720x480 resolution with 8 frames per second and supports videos up to 6 seconds in length. The model utilizes 3D sincos positional encoding and currently only supports English prompts with a maximum length of 226 tokens.
Training and Performance
The training dataset comprised approximately 35 million single-shot video clips (averaging 6 seconds each) paired with text descriptions. The data preparation involved a sophisticated video captioning pipeline leveraging pre-trained models and GPT-4 for creating detailed descriptions. Additionally, 2 billion images from LAION-5B and COYO-700M datasets were incorporated into the training process.
The training methodology employed several innovative techniques:
- Progressive training with multi-resolution frame packing
- Explicit Uniform Sampling for loss curve stabilization
- Rigorous video filtering based on quality criteria
- Resolution progressive training for improved efficiency
Deployment and Optimization
CogVideoX-2B is notably flexible in terms of deployment options and hardware requirements. The model supports multiple precision formats:
- FP16 (recommended)
- BF16
- FP32
- FP8 (requires NVIDIA H100 or higher with CUDA 12.4)
- INT8
- INT4
Using the diffusers library with optimizations, the minimum GPU memory requirement is 4GB with FP16 and can be reduced to 3.6GB with INT8 quantization. Inference speeds are approximately 90 seconds on a single A100 and 45 seconds on an H100 (with 50 steps, using FP/BF16).
Model Family Comparison
Within the CogVideoX family, several variants offer different capabilities:
- CogVideoX-5B (August 27, 2024): Offers higher resolution (1360x768) and longer video generation (10 seconds), using 3D rope positional encoding
- CogVideoX-5B-I2V (September 19, 2024): Adds image-to-video generation capabilities
- CogVideoX1.5-5B and CogVideoX1.5-5B-I2V (November 8, 2024): Further improvements in resolution and video length
CogVideoX-2B distinguishes itself as the most accessible option, capable of running on older GPUs like the GTX 1080TI, unlike its larger siblings.
Licensing and Availability
CogVideoX-2B is released under the Apache 2.0 License, making it freely available for both research and commercial use. The model weights, including those for the 3D Causal VAE and video caption model, are publicly available through the official repository.
References
- CogVideoX GitHub Repository - Official source code and model weights
- CogVideoX Research Paper - Technical details and methodology
- HuggingFace Model Page - Model downloads and documentation
- SwissArmyTransformer - Inference and fine-tuning framework
- PytorchAO - Quantization library for optimization
- Optimum-quanto - Additional quantization tools
- ComfyUI-CogVideoXWrapper - ComfyUI integration
- CogVideoX-Controlnet - ControlNet implementation
- cogvideox-factory - Fine-tuning framework
