Bokeh Diffusion: Defocus Blur Control in Text-to-Image Diffusion Models

TL;DR

Bokeh Diffusion introduces a scene-consistent defocus control framework for text-to-image diffusion by conditioning on a physical defocus parameter. It employs a hybrid data pipeline combining in-the-wild images with synthetic blur, along with a grounded self-attention mechanism and a defocus-conditioned cross-attention to preserve scene content while tuning blur. The method demonstrates superior accuracy, consistency, and perceptual quality compared to pretrained and finetuned baselines, and adapts to both UNet-based Stable Diffusion and MMDiT-based FLUX architectures. It also enables practical editing workflows, including inversion-based real-image editing, highlighting its potential for controllable, lens-like photography effects in diffusion-based generation and editing systems.

Abstract

Recent advances in large-scale text-to-image models have revolutionized creative fields by generating visually captivating outputs from textual prompts; however, while traditional photography offers precise control over camera settings to shape visual aesthetics - such as depth-of-field via aperture - current diffusion models typically rely on prompt engineering to mimic such effects. This approach often results in crude approximations and inadvertently alters the scene content. In this work, we propose Bokeh Diffusion, a scene-consistent bokeh control framework that explicitly conditions a diffusion model on a physical defocus blur parameter. To overcome the scarcity of paired real-world images captured under different camera settings, we introduce a hybrid training pipeline that aligns in-the-wild images with synthetic blur augmentations, providing diverse scenes and subjects as well as supervision to learn the separation of image content from lens blur. Central to our framework is our grounded self-attention mechanism, trained on image pairs with different bokeh levels of the same scene, which enables blur strength to be adjusted in both directions while preserving the underlying scene. Extensive experiments demonstrate that our approach enables flexible, lens-like blur control, supports downstream applications such as real image editing via inversion, and generalizes effectively across both Stable Diffusion and FLUX architectures.

Figures (20)

• Figure 1: State-of-the-art text-to-image models labsFLUX2024 do not understand textual photographic cues, and unintentionally alter image composition.
• Figure 2: Comparison between our generative method and BokehMe pengBokehMeWhenNeural2022a, which depends on external depth estimation.
• Figure 3: Data collection and curation pipeline.
• Figure 4: In-the-wild image augmentation.
• Figure 5: Overview of Bokeh Diffusion's conditioning and scene-consistency mechanisms. Bokeh Cross-Attention (Sec. \ref{['sec:bokeh']}), applied at deeper layers of the text-to-image diffusion model, injects a defocus conditioning signal, decoupled from semantic guidance. Grounded Self-Attention (Sec. \ref{['sec:scene']}) ensures fine-grained scene consistency across different defocus levels, where the pivot image's queries anchor the scene structure, while shared keys allow the model to disentangle defocus blur from scene composition.