BLENDER: Blended Text Embeddings and Diffusion Residuals for Intra-Class Image Synthesis in Deep Metric Learning

TL;DR

BLenDeR, a diffusion sampling method designed to increase intra-class diversity for DML in a controllable way by leveraging set-theory inspired union and intersection operations on denoising residuals, is introduced.

Abstract

The rise of Deep Generative Models (DGM) has enabled the generation of high-quality synthetic data. When used to augment authentic data in Deep Metric Learning (DML), these synthetic samples enhance intra-class diversity and improve the performance of downstream DML tasks. We introduce BLenDeR, a diffusion sampling method designed to increase intra-class diversity for DML in a controllable way by leveraging set-theory inspired union and intersection operations on denoising residuals. The union operation encourages any attribute present across multiple prompts, while the intersection extracts the common direction through a principal component surrogate. These operations enable controlled synthesis of diverse attribute combinations within each class, addressing key limitations of existing generative approaches. Experiments on standard DML benchmarks demonstrate that BLenDeR consistently outperforms state-of-the-art baselines across multiple datasets and backbones. Specifically, BLenDeR achieves 3.7% increase in Recall@1 on CUB-200 and a 1.8% increase on Cars-196, compared to state-of-the-art baselines under standard experimental settings.

Figures (11)

• Figure 1: Motivated by the observation that Stable Diffusion personalized with LoRA and Textual Inversion struggles to generate learned concepts with novel attributes, e.g., $[\texttt{Hooded Oriole}]$ with the attribute flying, we propose BLenDeR, a novel diffusion sampling method that steers the personalized model to generate target concepts with novel and challenging target attributes.
• Figure 2: Overview of the proposed BLenDeR approach. BLenDeR uses multiple text prompts: a target anchor prompt, an attribute donor prompt, and multiple context prior prompts. The attribute donor prompt is used with Text Embedding Interpolation to pre-align latents with the target attribute. BLenDeR utilizes the noise predictions from each text embedding in the proposed residual set operations, which use context priors to steer the denoising process toward the target attribute and concept specified in the target anchor prompt.
• Figure 3: Visual demonstration when using different approaches for generating novel background and poses. Prompts used in generation are consisting of two parts, the invoking part that contains the target concept $[V_i]$, "A photo of a $[V_i]$ bird.", followed by the target attribute description $a$. For generation, either only the Target Anchor prompt $c_1$ is used as Baseline (TA, Sec. \ref{['subsec:prompts']}), or $c_1$ with Text Embedding Interpolation with attribute donor prompt $c_2$ (TEI, Eq. \ref{['eq:mixembed']}), or $c_1$ with BLenDeR Residual Space Operation Union ($\cup$) or Intersection ($\cap$) (RSO, Eq. \ref{['eq:union']} and Eq. \ref{['eq:inter']}), or full BLenDeR operation which combines TEI and RSO. As can be seen, when the base model is not able to generate the concept with the targeted attribute, BLenDeR is able to synthesize the concept in conjunction with the targeted attribute.
• Figure 4: Example images of birds with attribute descriptions.
• Figure 5: Comparison of images generated by Stable Diffusion models trained on unaligned versus aligned datasets.