CogVideoX 1.5 5B I2V is a state-of-the-art image-to-video generation model released in November 2024 as part of the CogVideoX model family. Built on a diffusion transformer architecture, it represents a significant advancement in video generation capabilities, as detailed in the technical paper.
The model utilizes a sophisticated 3D Variational Autoencoder (VAE) for improved video compression and fidelity, combined with an expert transformer featuring expert adaptive LayerNorm for enhanced text-video alignment. This architecture enables deep fusion between text and video modalities while preventing common issues like flickering in generated videos.
A key innovation in the model's design is its progressive training approach and multi-resolution frame pack technique, which enables the generation of coherent, long-duration videos with significant motion. The model incorporates Explicit Uniform Sampling to stabilize training loss curves and accelerate convergence.
Training and Capabilities
The model was trained on an extensive dataset comprising approximately 35 million single-shot video clips (averaging 6 seconds each) with corresponding text descriptions, supplemented by 2 billion high-quality images filtered from LAION-5B and COYO-700M datasets. The training process involved a sophisticated data processing pipeline that included:
- Video filtering to remove low-quality, edited, or text-heavy content
- A comprehensive video captioning method utilizing pre-trained models and GPT-4
- A two-stage training approach, progressively increasing resolution
CogVideoX 1.5 5B I2V can generate videos with flexible resolutions, where the minimum dimension must be 768 pixels and the maximum dimension between 768 and 1360 pixels (always divisible by 16). It supports video generation lengths of 5 or 10 seconds at 16 frames per second, with English prompts up to 224 tokens.
Technical Specifications and Performance
The model offers multiple inference precision options:
- BF16 (recommended)
- FP16
- FP32
- FP8 (requires NVIDIA H100 or newer)
- INT8
- INT4 is not supported
Memory requirements and performance vary significantly based on configuration:
- Single GPU inference with BF16 requires minimum 9GB VRAM
- Multi-GPU inference with diffusers requires at least 24GB VRAM
- Inference times on high-end hardware:
- Single A100 GPU: ~1000 seconds for 5-second video
- Single H100 GPU: ~550 seconds for 5-second video
Memory optimization tools like PytorchAO and Optimum-quanto can be used for quantization to reduce memory footprint, enabling inference on lower-VRAM GPUs.
Model Family Comparison
The CogVideoX family includes several variants with different capabilities:
- CogVideoX 1.5 5B: The base text-to-video model
- CogVideoX 1.5 5B I2V: Specialized for image-to-video generation
- Previous versions (2B and 5B): Generated 6-second videos at 720x480 resolution
The 1.5 series represents a significant improvement over earlier versions, offering longer duration (10 seconds vs 6 seconds) and higher resolution capabilities (up to 1360x768 vs 720x480).
References
