Stable Diffusion 1.5, released in October 2022 by RunwayML, represents a significant advancement in text-to-image generation technology. Building upon the foundational research in Latent Diffusion Models, it introduces improvements in image quality and generation capabilities while maintaining accessibility through open-source availability.
Architecture and Technical Details
At its core, Stable Diffusion 1.5 is a latent diffusion model with 983 million parameters. The architecture consists of three main components:
- A variational autoencoder (VAE) for compressing and decompressing images between pixel space and latent space
- A U-Net backbone for denoising the latent representation
- A CLIP ViT-L/14 text encoder for processing text prompts
The model operates in a compressed latent space (reducing 512x512x3 images to 64x64x4 latent representations), significantly reducing computational requirements compared to pixel-space diffusion models. This efficiency allows the model to run on consumer hardware with as little as 4GB VRAM.
Training and Performance
The model underwent extensive training on the LAION-aesthetics v2 5+ dataset, with 595,000 fine-tuning steps at 512x512 resolution. The training process involved:
- 150,000 GPU hours on 256 Nvidia A100 GPUs
- Approximately 11,250 kg CO2 eq. emissions
- A 10% reduction in text conditioning to enhance classifier-free guidance sampling
Performance evaluations show improvements over previous versions (v1.1-v1.4) in terms of image quality and prompt adherence. The model performs particularly well with classifier-free guidance scales between 7.0 and 8.5, typically using 50 inference steps for optimal quality.
Features and Capabilities
The model supports multiple generation modes:
- Text-to-image generation (txt2img)
- Image modification (img2img)
- Inpainting and outpainting
Key parameters affecting generation include:
- Sampling type (PNDM default, with DDIM and K-LMS alternatives)
- Image dimensions (multiples of 8, optimal at 512x512)
- Seed value for reproducibility
- Classifier-free guidance scale
- Number of inference steps (typically 50, adjustable for speed/quality trade-off)
Limitations and Considerations
While powerful, Stable Diffusion 1.5 has several notable limitations:
- Base resolution restricted to 512x512
- Inconsistent rendering of human limbs and complex compositions
- Limited effectiveness with non-English prompts
- Potential biases from training data
- Imperfect photorealism
The model operates under the CreativeML OpenRAIL-M license, which permits commercial use of generated images while prohibiting harmful applications. Users must comply with content restrictions and include appropriate license documentation when redistributing the model.
References
- Stable Diffusion v1.5 Model Card - Official model documentation and implementation details
- Latent Diffusion Models Paper - Foundational research behind the architecture
- Classifier-Free Diffusion Guidance - Technical details on the guidance mechanism
- LAION-5B Dataset - Information about the training dataset
- Stable Diffusion License - Official licensing documentation
- Diffusers Library Documentation - Implementation and usage guidelines
- Interactive Diffusion Explainer - Visual explanation of the underlying technology
- U-Net Architecture Details - Technical documentation of the U-Net component
