EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Models

TL;DR

This paper proposes a novel approach termed Exemplar-guided Image Translation with Brownian-Bridge Diffusion Models (EBDM), which formulates the task as a stochastic Brownian bridge process, a diffusion process with a fixed initial point that translates into the corresponding photo-realistic image while being conditioned solely on the given exemplar image.

Abstract

Exemplar-guided image translation, synthesizing photo-realistic images that conform to both structural control and style exemplars, is attracting attention due to its ability to enhance user control over style manipulation. Previous methodologies have predominantly depended on establishing dense correspondences across cross-domain inputs. Despite these efforts, they incur quadratic memory and computational costs for establishing dense correspondence, resulting in limited versatility and performance degradation. In this paper, we propose a novel approach termed Exemplar-guided Image Translation with Brownian-Bridge Diffusion Models (EBDM). Our method formulates the task as a stochastic Brownian bridge process, a diffusion process with a fixed initial point as structure control and translates into the corresponding photo-realistic image while being conditioned solely on the given exemplar image. To efficiently guide the diffusion process toward the style of exemplar, we delineate three pivotal components: the Global Encoder, the Exemplar Network, and the Exemplar Attention Module to incorporate global and detailed texture information from exemplar images. Leveraging Bridge diffusion, the network can translate images from structure control while exclusively conditioned on the exemplar style, leading to more robust training and inference processes. We illustrate the superiority of our method over competing approaches through comprehensive benchmark evaluations and visual results.

Figures (11)

• Figure 1: Motivation.(a) Existing methods with matching-than-generation framework, (b) Widely used framework based on conditional noise-to-image diffusion model, and (c) our framework based on Brownian bridge diffusion models.
• Figure 2: Framework overview. The proposed EBDM framework is a based on (a) Brownian Bridge Diffusion Model and composed of (b) Exemplar Network and a (c) Global Encoder. Global Encoder encodes global style information and Exemplar Network extracts texture information from exemplar image. Extracted texture and global information is then used to guide the diffusion process via Exemplar Attention Module and cross-attention, respectively.
• Figure 3: Qualitative Results. Visual comparisons of the proposed EBDM and state-of-the-art methods over three types of exemplar-guided image translation tasks.
• Figure 4: Visual Comparison based on the choice of Exemplar Encoder.
• Figure 5: Qualitative Comparison to SOTA Diffusion-based Model.