ControlNet SDXL Canny is an edge detection model designed specifically for Stable Diffusion XL (SDXL). The model functions by creating simplified, sharp lines around areas of high contrast in input images, producing what's known as a "detectmap" that guides the SDXL image generation process. This approach allows users to control the composition and structure of generated images while maintaining flexibility in style and theme.
The model's architecture is based on ControlNet 1.1, which brought significant improvements over its predecessor. The training process involved a substantial investment of computational resources - 72 hours on 8 Nvidia A100 80G GPUs, costing approximately 2160 USD, as detailed in the ControlNet-v1-1-nightly repository.
A notable advancement in the technology came with the introduction of Control-LoRA variants, which offer significantly reduced model sizes. While the original ControlNet models were 4.7GB, the LoRA versions are available in Rank 256 (~738MB) and Rank 128 (~377MB) variants, making them more accessible and efficient to use.
Training and Capabilities
The training data for ControlNet SDXL Canny 1.1 addressed several issues present in the original version. The model was trained using Canny edge maps with randomly generated thresholds, eliminating problems with duplicated grayscale images and low-quality data that affected the earlier version. This resulted in improved robustness and visual quality compared to Canny 1.0.
One of the model's key strengths is its ability to handle relatively thick scribbles - up to 24-pixel width in a 512 canvas. This makes it particularly versatile for various input types and user skill levels. The model demonstrates robust performance across different use cases, achieving similar performance levels to the depth model in the ControlNet family, as noted in the Civitai guide.
Implementation and Variants
Several variants of the model exist across different Stable Diffusion versions (SD 1.5, SD 2.x, and SDXL) and developers. The sd-webui-controlnet version 1.1.400 introduced multiple Canny edge detection models for SDXL:
diffusers_xl_canny_full.safetensorsdiffusers_xl_canny_mid.safetensorsdiffusers_xl_canny_small.safetensors
For optimal performance, users should consider their hardware capabilities. The implementation recommends using --medvram-sdxl for systems with 8GB-16GB VRAM and --lowvram for systems with less than 8GB VRAM, as detailed in the sd-webui-controlnet discussions.
The model can be used in various interfaces, including Automatic1111's web UI, ComfyUI, and StableSwarmUI. When using the Automatic1111 interface, specific settings are crucial for achieving results identical to Stability AI's official ComfyUI workflows, including CPU seed settings, SGM noise multiplier, and deterministic sampling.
References
- ControlNet 1.1 Models (Civitai) - Comprehensive listing of ControlNet models
- ControlNet-v1-1-nightly Repository - Technical details and training information
- sd-webui-controlnet Discussion - Implementation details and optimization guidelines
- Stability AI Control-LoRA - Information about LoRA variants and model specifications
- Automatic1111 ControlNet Plugin - Usage instructions and implementation details
- ControlNet Main Repository - Core project documentation and resources
