Stable Video Diffusion XT represents a significant advancement in AI-powered video generation technology, building upon Stability AI's successful Stable Diffusion image model foundation. Released in November 2023 as a research preview, this latent video diffusion model specializes in converting single images into short video clips of up to 4 seconds in length.
Architecture and Training
The model's architecture extends the capabilities of Stable Diffusion 2.1 by incorporating temporal convolution and attention layers. The training process followed three critical stages:
- Text-to-image pretraining using Stable Diffusion 2.1
- Video pretraining on a massive dataset of approximately 152 million clips (LVD-F)
- High-quality video finetuning using roughly 250,000 carefully curated clips
The training process was computationally intensive, utilizing approximately 200,000 A100 80GB hours and resulting in an estimated carbon footprint of ~19,000kg CO2 eq. and energy consumption of ~64,000 kWh.
Data curation played a crucial role in the model's development. The team implemented a systematic process that included:
- Cut detection using PySceneDetect
- Motion annotation through optical flow analysis
- Caption generation utilizing CoCa, V-BLIP, and LLM technologies
- Aesthetics scoring based on CLIP embeddings
- Text detection for filtering undesirable content
Features and Capabilities
SVD-XT generates 25 frames at 576x1024 resolution, an improvement over the original SVD model's 14 frames. The model incorporates:
- A fine-tuned f8-decoder for improved temporal consistency
- A standard frame-wise decoder for convenience
- Customizable frame rates between 3-30 fps
- Default image-level watermarking for output detection
Human preference studies have demonstrated that SVD-Image-to-Video XT outperforms competing models like RunwayML GEN-2 and PikaLabs in terms of video quality. The model exhibits strong multi-view 3D-prior capabilities, making it suitable for generating multiple views of objects in a feedforward manner.
Applications and Limitations
The primary applications for SVD-XT include:
- Research on generative models
- Safe deployment studies of potentially harmful models
- Understanding model limitations and biases
- Artistic creation
- Educational tool development
However, the model does have several notable limitations:
- Short video length constraints
- Imperfect photorealism
- Inability to be controlled by text
- Potential issues with rendering text and faces
Licensing and Availability
The model is available under the "stable-video-diffusion-community" license, with commercial use requiring adherence to Stability AI's commercial license. The complete codebase is accessible through the official GitHub repository.
References
- Stability AI Commercial License - Official commercial licensing terms
- GitHub Repository - Source code and model implementation
- Research Paper - Technical details and methodology
- Model License - Community license terms
- RunwayML GEN-2 - Competing model reference
- PikaLabs - Competing model reference
- Hugging Face Model Page - Model weights and documentation
