Stable Video Diffusion (SVD) represents a significant advancement in AI-generated video technology, developed by Stability AI as their first foundation model for generative video. Released in late 2023, it builds upon their successful Stable Diffusion image model technology to create high-quality video content from both images and text prompts.
Architecture and Training
The model employs a latent diffusion architecture, modified from Stable Diffusion 2.1's UNet design with additional temporal convolutional and attention layers. The training process involves three crucial stages:
- Text-to-image pretraining using Stable Diffusion 2.1
- Video pretraining on a massive curated dataset (LVD-F) containing approximately 152 million clips
- High-quality video fine-tuning using roughly 250,000 carefully selected clips
Data curation played a vital role in the model's development, involving sophisticated processes such as cut detection, motion annotation through optical flow analysis, caption generation (using CoCa, V-BLIP, and LLM), and aesthetics scoring via CLIP embeddings. This meticulous approach to data curation has proven superior to using uncurated datasets, as detailed in the research paper.
Capabilities and Features
SVD comes in two main variants:
- Standard SVD: Generates 14 frames at 576x1024 resolution
- SVD-XT: Creates 25 frames at the same resolution
Both models support customizable frame rates between 3-30 fps and utilize specialized decoders for ensuring temporal consistency. The f8-decoder specifically focuses on maintaining visual coherence across frames.
Key capabilities include:
- Image-to-video generation
- Customizable camera motion through LoRA modules (horizontal, zooming, static)
- Frame interpolation for improved frame rates
- Strong multi-view 3D synthesis capabilities
- Default image-level watermarking for output detection
The model demonstrates impressive performance in human evaluation studies, showing higher preference rates compared to competing solutions like RunwayML GEN-2 and PikaLabs.
Technical Specifications and Limitations
The model's development required substantial computational resources, with training consuming approximately 200,000 A100 80GB hours and generating an estimated 19,000 kg CO2 eq. emission. Inference times on an A100 80GB card are:
- SVD: ~100 seconds
- SVD-XT: ~180 seconds
Notable limitations include:
- Short video length (≤4 seconds)
- Potential for limited motion or slow panning
- No direct text control capability
- Difficulties with legible text rendering
- Challenges with realistic face generation
Usage and Licensing
The model is primarily intended for research purposes, including:
- Generative model research
- Safe deployment investigations
- Bias detection studies
- Artistic applications
- Educational tools
Commercial use requires adherence to Stability AI's license. The model explicitly prohibits generating factually accurate representations of people or events and any activities violating their Acceptable Use Policy.
References
- Stability AI License - Commercial licensing information
- Research Paper - Technical details and evaluation results
- GitHub Repository - Code and model implementation
- Hugging Face Model - Model weights and documentation
- RunwayML GEN-2 - Competing model comparison
- PikaLabs - Competing model comparison
- Acceptable Use Policy - Usage guidelines and restrictions
- StableVideo FAQ - Detailed usage information
