CogVideoX-5B is a large-scale text-to-video generation model based on a diffusion transformer architecture, released on August 27, 2024. The model represents a significant advancement in video generation capabilities, building upon the earlier CogVideo model released in May 2022.
The architecture incorporates several key components that enable high-quality video generation:
- A 3D Variational Autoencoder (VAE) for improved video compression and fidelity
- An expert transformer with expert adaptive LayerNorm for enhanced text-video alignment
- A 3D full attention mechanism for comprehensive video data modeling
- 3D positional encoding (specifically 3d_rope_pos_embed)
Training and Capabilities
The model was trained on approximately 35 million single-shot video clips averaging 6 seconds each, combined with 2 billion filtered images. A sophisticated video captioning pipeline was developed using a pre-trained video captioning model, CogVLM for image recaptioning, and GPT-4 for summarization, with a fine-tuned Llama 2 model for acceleration. The training process employed BF16 precision and utilized a progressive training pipeline with multi-resolution frame packing.
CogVideoX-5B can generate:
- 10-second continuous videos at 16 frames per second
- Resolution of 768x1360 pixels
- Videos from both text prompts and images
- English-language prompts up to 224 tokens
Performance and Technical Requirements
The model demonstrates state-of-the-art performance based on both automated metrics and human evaluations, particularly in areas such as Human Action, Scene, Dynamic Degree, Multiple Objects, Appearance Style, and Dynamic Quality. As detailed in the technical paper, inference times are approximately:
- 180 seconds on a single A100 GPU
- 90 seconds on a single H100 GPU
For optimal performance, BF16 precision is recommended, though FP16 and FP32 are also supported. The model can run on consumer-grade hardware through various optimization techniques, including quantization options (BF16, FP16, FP32, FP8, and INT8).
Model Family and Variants
The CogVideoX family includes several variants:
CogVideoX-2B (Released August 6, 2024):
- Smaller model size suitable for older GPUs like GTX 1080TI
- Generates 6-second videos at 8 frames per second
- Uses 3d_sincos_pos_embed positional encoding
- Lower resource requirements but maintains competitive performance
CogVideoX-5B-I2V (Released September 19, 2024):
- Specialized variant for image-to-video generation
- Supports video generation at any resolution
- Built on the same core architecture as CogVideoX-5B
CogVideoX 1.5 (Released November 8, 2024):
- Improved version of CogVideoX-5B
- Supports higher resolution 10-second videos
- Includes the CogVideoX 1.5-5B-I2V variant
References and Resources
