ControlNet SDXL Depth

ControlNet SDXL Depth represents a significant advancement in controlled image generation, building upon the foundation established by the original ControlNet architecture. This model specializes in incorporating depth information into the image generation process, allowing for precise control over the three-dimensional aspects of generated images.

Architecture and Training

The model maintains the core ControlNet architecture while introducing specific optimizations for depth-based control. It was trained on a comprehensive dataset combining multiple depth map sources, including Midas, Leres, and Zoe depth maps, at various resolutions (256, 384, and 512 pixels). This diverse training approach, incorporating multiple depth map generators and resolutions, contributes to the model's robust performance across different input types.

Two main variants exist: the full ControlNet implementation and the Control-LoRA version. The Control-LoRA variant, developed by Stability AI, offers a more efficient alternative at approximately 738MB (Rank 256) or 377MB (Rank 128), compared to the original 4.7GB model size. This version was specifically trained on depth results from the MiDaS dpt_beit_large_512 model and further refined using Stability AI's Portrait Depth Estimation technology, as detailed in the Control-LoRA documentation.

Features and Capabilities

ControlNet SDXL Depth excels in manipulating image generation based on grayscale depth maps, which provide crucial information about object distances from the camera viewpoint. The model supports multiple depth map preprocessors:

• depth_midas
• depth_zoe
• depth_leres++
• depth_leres

Each preprocessor offers slightly different results, allowing users to choose the most appropriate option for their specific use case. The model demonstrates particular robustness when working with real depth maps from rendering engines, as noted in the ControlNet v1.1 documentation.

Implementation and Usage

The model can be integrated into various frameworks, with particular support for Automatic1111's Stable Diffusion WebUI through the sd-webui-controlnet extension. For optimal performance, specific VRAM configurations are recommended:

• --medvram-sdxl for systems with 8GB-16GB VRAM
• --lowvram for systems with less than 8GB VRAM

Multiple model variants are available for different use cases:

• diffusers_xl_depth_full.safetensors
• diffusers_xl_depth_mid.safetensors
• diffusers_xl_depth_small.safetensors

These models can be placed in either stable-diffusion-webui\extensions\sd-webui-controlnet\models or stable-diffusion-webui\models\ControlNet directories for implementation.

References

1. ControlNet Research Paper - Original research underlying ControlNet architecture
2. ControlNet GitHub Repository - Main development repository
3. Automatic1111 ControlNet Extension - Popular implementation for Stable Diffusion WebUI
4. ControlNet v1.1 Documentation - Technical details and implementation guidelines
5. Control-LoRA Documentation - Information about the efficient LoRA implementation
6. SDXL ControlNet Models - Collection of compatible SDXL ControlNet models