Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GBSD: Generative Bokeh with Stage Diffusion

Jieren Deng, Xin Zhou, Hao Tian, Zhihong Pan, Derek Aguiar

TL;DR

GBSD introduces semantic bokeh through a two-stage latent diffusion framework that applies bokeh selectively to semantically defined objects. It decomposes diffusion into a Global Layout Stage and a Focus Stage, interpolating prompts to sharpen a target object while blurring others without requiring high-dimensional masks or retraining. Quantitative and qualitative evaluations on text-to-image and image-to-image tasks demonstrate improved focus control (lower Variance of Laplacian and Brenner scores for non-targets) and more realistic bokeh effects, suggesting practical utility for AI-assisted image synthesis and editing. This approach broadens controllable artistic styling in diffusion models and enables robust, object-centric bokeh in real-world applications.

Abstract

The bokeh effect is an artistic technique that blurs out-of-focus areas in a photograph and has gained interest due to recent developments in text-to-image synthesis and the ubiquity of smart-phone cameras and photo-sharing apps. Prior work on rendering bokeh effects have focused on post hoc image manipulation to produce similar blurring effects in existing photographs using classical computer graphics or neural rendering techniques, but have either depth discontinuity artifacts or are restricted to reproducing bokeh effects that are present in the training data. More recent diffusion based models can synthesize images with an artistic style, but either require the generation of high-dimensional masks, expensive fine-tuning, or affect global image characteristics. In this paper, we present GBSD, the first generative text-to-image model that synthesizes photorealistic images with a bokeh style. Motivated by how image synthesis occurs progressively in diffusion models, our approach combines latent diffusion models with a 2-stage conditioning algorithm to render bokeh effects on semantically defined objects. Since we can focus the effect on objects, this semantic bokeh effect is more versatile than classical rendering techniques. We evaluate GBSD both quantitatively and qualitatively and demonstrate its ability to be applied in both text-to-image and image-to-image settings.

GBSD: Generative Bokeh with Stage Diffusion

TL;DR

GBSD introduces semantic bokeh through a two-stage latent diffusion framework that applies bokeh selectively to semantically defined objects. It decomposes diffusion into a Global Layout Stage and a Focus Stage, interpolating prompts to sharpen a target object while blurring others without requiring high-dimensional masks or retraining. Quantitative and qualitative evaluations on text-to-image and image-to-image tasks demonstrate improved focus control (lower Variance of Laplacian and Brenner scores for non-targets) and more realistic bokeh effects, suggesting practical utility for AI-assisted image synthesis and editing. This approach broadens controllable artistic styling in diffusion models and enables robust, object-centric bokeh in real-world applications.

Abstract

The bokeh effect is an artistic technique that blurs out-of-focus areas in a photograph and has gained interest due to recent developments in text-to-image synthesis and the ubiquity of smart-phone cameras and photo-sharing apps. Prior work on rendering bokeh effects have focused on post hoc image manipulation to produce similar blurring effects in existing photographs using classical computer graphics or neural rendering techniques, but have either depth discontinuity artifacts or are restricted to reproducing bokeh effects that are present in the training data. More recent diffusion based models can synthesize images with an artistic style, but either require the generation of high-dimensional masks, expensive fine-tuning, or affect global image characteristics. In this paper, we present GBSD, the first generative text-to-image model that synthesizes photorealistic images with a bokeh style. Motivated by how image synthesis occurs progressively in diffusion models, our approach combines latent diffusion models with a 2-stage conditioning algorithm to render bokeh effects on semantically defined objects. Since we can focus the effect on objects, this semantic bokeh effect is more versatile than classical rendering techniques. We evaluate GBSD both quantitatively and qualitatively and demonstrate its ability to be applied in both text-to-image and image-to-image settings.
Paper Structure (22 sections, 6 equations, 8 figures)

This paper contains 22 sections, 6 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Bokeh Synthesis.
  4. Deep Generative Models.
  5. Text Driven Image Generation and Editing Using DM.
  6. Variable Text Prompt Conditioning.
  7. Methods
  8. Diffusion Model Preliminaries
  9. Stage Diffusion
  10. Global Layout Stage
  11. Focus Stage
  12. Image Generation
  13. Results
  14. Evaluation Measures
  15. Variance of Laplacian
  16. ...and 7 more sections

Figures (8)

  • Figure 1: An illustration of stage diffusion for text-to-image and image-to-image generation with a bokeh style. A bokeh style image is generated by a two-stage semantic conditioning algorithm. The first stage (from $z_{T}$ to $z_{t}$) generates the global layout of the image (e.g., shape and color) while stage two (from $z_{t}$ to $x$) focuses detail and bokeh effects through semantic conditioning. In the text-to-image example (left), the stage 1 prompt was "A cute baby bunny standing on top of a pile of baby carrots under a spot light" and different prompts in stage 2 that focus either the carrots (bottom) or the bunny (top). In the image-to-image example (right), we use the prompt "A cute rabbit stands with carrots with green leaf" in stage 1 and "carrots with green leaf" in stage 2. The generated image demonstrates the previously blurry carrot coming into focus, revealing more clear and distinct textures, while creating a bokeh effect for the rabbit.
  • Figure 2: The generative bokeh with stage diffusion (GBSD) architecture. For brevity, we represent the diffusion process for each stage by a single representative denoising autoencoder.
  • Figure 3: Comparing image generation by adjusting $\sigma$. The proportion of denoising steps allocated to the global layout stage was set to {0.9,0.8,0.7,0.6,0.5} and the overall number of denoising steps and conditioning per stage was fixed. Both experiments (top and bottom rows) shared the same global prompt "a cute baby bunny standing on top of a pile of baby carrots under a spot light". The top row was given a local prompt of "a pile of carrots" and the bottom row was given "a cute rabbit". The proportion of denoising steps spent in the global layout (stage 1) and focus (stage 2) stages is given in the green and blue boxes, respectively. The rabbit features in the top experiment degenerated as stage 1 was shortened; similarly, the carrots features in the bottom experiment exhibited a similar trend.
  • Figure 4: Undesirable semantic mixing with a short global layout. We used a global prompt that incorporated both the terms "carrots" and "rabbit". In the subsequent focus stage, a local prompt that solely included the keyword "carrots" was added. When the global layout stage is too short (here, 20%), the "carrots" features merged with a "rabbit"-like object, producing an object that combined the features of both for two distinct random seeds (a) and (b).
  • Figure 5: Comparison between the baseline LDM and GBSD. We used the text conditioning "a cute baby bunny standing on top of a pile of baby carrots under a spot light" both as the input prompt for LDM (a) and the global prompt for GBSD (b). In (b), a local prompt "a pile of baby carrots under a spot light" was used in stage 2 with $80\%$ of denoising steps dedicated to stage 1 and $20\%$ to stage 2. The highlighted image segments demonstrate that GBSD produces (1) stem details on the carrots, (2) a sharper image due to the focus on the carrots in stage 2, and (3) a bokeh effect on the bunny.
  • ...and 3 more figures