FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation

TL;DR

This work tackles the limited controllability of large-scale text-to-image diffusion models for image-to-image translation. It introduces FBSDiff, a plug-and-play approach that steers T2I generation using a reference image by dynamically transplanting DCT frequency bands between diffusion features along the reverse trajectory, without any training or online optimization. By decomposing guiding cues into low-, mid-, and high-frequency components, FBSDiff enables flexible control over appearance, layout, and contours through the frequency-band substitution layer, and it proves effective across diverse prompts and references. The results show superior image quality, controllability, and diversity compared to prior methods, with practical implications for media editing and content creation where flexible reference-guided translation is valuable.

Abstract

Large-scale text-to-image diffusion models have been a revolutionary milestone in the evolution of generative AI and multimodal technology, allowing wonderful image generation with natural-language text prompt. However, the issue of lacking controllability of such models restricts their practical applicability for real-life content creation. Thus, attention has been focused on leveraging a reference image to control text-to-image synthesis, which is also regarded as manipulating (or editing) a reference image as per a text prompt, namely, text-driven image-to-image translation. This paper contributes a novel, concise, and efficient approach that adapts pre-trained large-scale text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a plug-and-play manner, realizing high-quality and versatile text-driven I2I translation without any model training, model fine-tuning, or online optimization process. To guide T2I generation with a reference image, we propose to decompose diverse guiding factors with different frequency bands of diffusion features in the DCT spectral space, and accordingly devise a novel frequency band substitution layer which realizes dynamic control of the reference image to the T2I generation result in a plug-and-play manner. We demonstrate that our method allows flexible control over both guiding factor and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band, respectively. Extensive qualitative and quantitative experiments verify superiority of our approach over related methods in I2I translation visual quality, versatility, and controllability. The code is publicly available at: https://github.com/XiangGao1102/FBSDiff.

Figures (39)

• Figure 1: Based on the pre-trained text-to-image diffusion model, FBSDiff enables efficient text-driven image-to-image translation by proposing a plug-and-play reference image guidance mechanism. It allows flexible control over different guiding factors (e.g., image appearance, image layout, image contours) of the reference image to the T2I generated image, simply by dynamically substituting different types of DCT frequency bands during the reverse sampling process of the diffusion model. Better viewed with zoom-in.
• Figure 2: Overview of FBSDiff. Based on the pre-trained latent diffusion model (LDM), FBSDiff starts with an inversion trajectory that inverts reference image into the LDM Gaussian noise space, then a reconstruction trajectory is applied to reconstruct the reference image from the inverted Gaussian noise, providing intermediate denoising results as pivotal guidance features. The guidance features are leveraged to guide the text-driven sampling trajectory of the LDM to exert reference image control, which is realized by dynamically transplanting certain DCT frequency bands from diffusion features along the reconstruction trajectory into the corresponding features along the sampling trajectory. The dynamic DCT frequency band transplantation is implemented in a plug-and-play manner with our proposed frequency band substitution layer (FBS layer).
• Figure 3: Illustration of the proposed frequency band substitution (FBS) layer. The FBS layer takes in two diffusion features and substitutes a certain frequency band of one feature with the corresponding frequency band of the other feature. This is realized by converting the two diffusion features into the frequency domain via 2D DCT, extracting and transplanting a certain DCT frequency band, and converting the fused DCT features back to spatial domain via 2D IDCT. The frequency band extraction and transplantation are implemented with binary masking.
• Figure 4: Qualitative results of our method with different types of frequency band substitution. For low-frequency band substitution (low-FBS), the generated image is controlled by the reference image in terms of image appearance and layout; for high-frequency band substitution (high-FBS), the reference image controls image contours of the generated image; as for mid-frequency band substitution (mid-FBS), only image layout of the generated image is controlled by the reference image. Better viewed with zoom-in.
• Figure 5: Comparison among different reference image control effects achieved by low-FBS, mid-FBS, and high-FBS. Low-FBS controls image appearance and layout, mid-FBS controls only image layout, and high-FBS controls image contours.