Stable Video Diffusion XT 1.1 represents a significant advancement in AI-powered video generation, building upon the architecture of Stable Diffusion 2.1. The model utilizes latent diffusion technology, incorporating temporal convolution and attention layers specifically designed for video processing. This architecture enables the model to generate short video clips of up to 4 seconds from a single input image, producing output at 1024x576 resolution with 25 frames.
Training and Development
The development of Stable Video Diffusion models follows a sophisticated three-stage training process, as detailed in the research paper. The process begins with text-to-image pretraining using Stable Diffusion 2.1, followed by video pretraining on a massive dataset of approximately 152 million clips (LVD-F), and concludes with high-resolution video finetuning on a carefully curated dataset of about 1 million samples.
A distinguishing feature of the training methodology is its systematic data curation process. This includes:
- Cut detection using PySceneDetect
- Motion annotation through optical flow analysis
- Caption generation utilizing CoCa, V-BLIP, and LLM technologies
- Aesthetics scoring
- Text detection for filtering undesirable content
The XT 1.1 variant specifically represents a fine-tuned version of the SVD Image-to-Video model, with optimizations focused on output consistency at 6FPS and Motion Bucket Id 127. While these parameters were fixed during fine-tuning, they remain adjustable in use, though performance may vary compared to SVD 1.0 when using different settings.
Capabilities and Limitations
The model demonstrates several key capabilities:
- High-resolution video generation from single images
- Strong multi-view 3D prior, enabling effective multi-view synthesis
- Support for camera motion control through LoRA modules
- Competitive performance with closed-source alternatives
However, users should be aware of several limitations:
- Maximum video length of 4 seconds
- Imperfect photorealism
- Potential for videos with limited motion or slow panning
- No text prompt control
- Difficulties with legible text generation
- Challenges in accurately rendering faces and people
- Lossy autoencoding process
Licensing and Usage
The model is released under the "stable-video-diffusion-1-1-community" license, available for both research and commercial applications. Commercial use requires registration with Stability AI if annual revenue exceeds $1,000,000. Research applications can include:
- Studying generative models
- Investigating safe deployment methods
- Analyzing model limitations and biases
- Artistic applications
Users must adhere to Stability AI's Acceptable Use Policy, which prohibits generating factually accurate representations of people or events without proper authorization.
References
