RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidance

TL;DR

This work tackles the problem of controllable consistency in image and video generation using diffusion-based approaches. It introduces RefDrop, a training-free, plug-and-play mechanism that replaces self-attention with Reference Feature Guidance (RFG), enabling precise control over the influence of reference features through a scalar coefficient $c$ and supporting multi-reference blending. The authors show that concatenated attention can be viewed as a linear interpolation between self-attention and cross-attention, motivating a simpler, more flexible RFG formulation and a multi-reference extension; RefDrop can also improve temporal consistency in video generation. Through extensive experiments, RefDrop demonstrates strong performance in consistent and diverse image generation, seamless feature blending from multiple references, and enhanced temporal consistency in personalized videos, without requiring encoder training. The method is lightweight, competitive with encoder-based baselines like IP-Adapter, and broadly applicable across diffusion architectures, with limitations including occasional object replication failures and sensitivity to tuning.

Abstract

There is a rapidly growing interest in controlling consistency across multiple generated images using diffusion models. Among various methods, recent works have found that simply manipulating attention modules by concatenating features from multiple reference images provides an efficient approach to enhancing consistency without fine-tuning. Despite its popularity and success, few studies have elucidated the underlying mechanisms that contribute to its effectiveness. In this work, we reveal that the popular approach is a linear interpolation of image self-attention and cross-attention between synthesized content and reference features, with a constant rank-1 coefficient. Motivated by this observation, we find that a rank-1 coefficient is not necessary and simplifies the controllable generation mechanism. The resulting algorithm, which we coin as RefDrop, allows users to control the influence of reference context in a direct and precise manner. Besides further enhancing consistency in single-subject image generation, our method also enables more interesting applications, such as the consistent generation of multiple subjects, suppressing specific features to encourage more diverse content, and high-quality personalized video generation by boosting temporal consistency. Even compared with state-of-the-art image-prompt-based generators, such as IP-Adapter, RefDrop is competitive in terms of controllability and quality while avoiding the need to train a separate image encoder for feature injection from reference images, making it a versatile plug-and-play solution for any image or video diffusion model.

Figures (22)

• Figure 1: RefDrop achieves controllable consistency in visual content synthesis for free. RefDrop exihibits great flexibility in (Upper) multi-subject consistency generation given one reference image, (Middle) blending different characters from multiple images seamlessly, (Buttom) enhancing temporal consistency for personalized video generation. RefDrop is short for "reference drop". We named our method RefDrop to metaphorically represent the process by which a drop of colored water influences a larger body of clear water.
• Figure 2: During each diffusion denoising step, we facilitate the injection of features from a generated reference image $I_1$ into the generation process of other images through RFG. The RFG layer produces a linear combination of the attention outputs from both the standard and referenced routes. A negative coefficient $c$ encourages divergence of $I_i$ from $I_1$, while a positive coefficient fosters consistency.
• Figure 3: We allow flexible control over the reference effect through a reference strength coefficient.
• Figure 4: The reference image for all methods is framed in red. Our method tends to produce more consistent outfits, hairstyles, and facial features compared to IP-Adapter and BLIPD. The visual quality of BLIPD is not comparable, as it utilizes SD1.5 rombach2022highresolution as its base model.
• Figure 5: Multiple Reference Images: The reference images are highlighted with a red frame, and the third image in each set is the resultant blended image. RefDrop effectively assimilates features from the distinct reference images into a single and cohesive entity, demonstrating robust feature integration capability.