CLUE: Controllable Latent space of Unprompted Embeddings for Diversity Management in Text-to-Image Synthesis

TL;DR

CLUE addresses the diversity-stability trade-off in medical text-to-image synthesis by introducing a Style Encoder that learns unprompted visual attributes and injects them via a second cross-attention path in Stable Diffusion 2.1, with KL regularization enforcing a Gaussian latent. The approach enables controllable, diverse generation from fixed prompts and no extra data, demonstrated on otitis media endoscopic images and generalizable to external data via synthetic augmentation. Across metrics including $FID$, $k$-NN recall, and $F_1$, CLUE yields substantial improvements over vanilla fine-tuning and demonstrates strong cross-institution generalization, particularly when synthetic data is combined with real data. This work highlights the practical potential of disentangled style representations for data augmentation and domain-specific image synthesis in settings with scarce labeled data.

Abstract

Text-to-image synthesis models require the ability to generate diverse images while maintaining stability. To overcome this challenge, a number of methods have been proposed, including the collection of prompt-image datasets and the integration of additional data modalities during training. Although these methods have shown promising results in general domains, they face limitations when applied to specialized fields such as medicine, where only limited types and insufficient amounts of data are available. We present CLUE (Controllable Latent space of Unprompted Embeddings), a generative model framework that achieves diverse generation while maintaining stability through fixed-format prompts without requiring any additional data. Based on the Stable Diffusion architecture, CLUE employs a Style Encoder that processes images and prompts to generate style embeddings, which are subsequently fed into a new second attention layer of the U-Net architecture. Through Kullback-Leibler divergence, the latent space achieves continuous representation of image features within Gaussian regions, independent of prompts. Performance was assessed on otitis media dataset. CLUE reduced FID to 9.30 (vs. 46.81) and improved recall to 70.29% (vs. 49.60%). A classifier trained on synthetic-only data at 1000% scale achieved an F1 score of 83.21% (vs. 73.83%). Combining synthetic data with equal amounts of real data achieved an F1 score of 94.76%, higher than when using only real data. On an external dataset, synthetic-only training achieved an F1 score of 76.77% (vs. 60.61%) at 1000% scale. The combined approach achieved an F1 score of 85.78%, higher than when using only the internal dataset. These results demonstrate that CLUE enables diverse yet stable image generation from limited datasets and serves as an effective data augmentation method for domain-specific applications.

Figures (14)

• Figure 1: Left: Key characteristics of medical datasets. (1) There are notable visual differences between medical and general domain images. (2) Even for the same diagnosis, clinical reports A, B, and C may use different phrasing or expression styles. (3) A model trained on general domain images is not directly applicable to medical image generation. Right: Uncertainty in generative models. When fine-tuned on report A, the model generates accurate images with A-style descriptions, but may produce inappropriate outputs when encountering an unfamiliar phrasing style such as report B.
• Figure 2: Structure of CLUE. The noised image passed through the U-Net with the prompt to predict a velocity. The clean image and prompt are sent into the style encoder to parameterize a Gaussian posterior from which a style latent $s \in \mathbb{R}^{1024}$ is sampled.
• Figure 3: Overview of the four stage preprocessing applied to each endoscopic frame
• Figure 4: (Right) Examples of generated images by CLUE across different style embeddings. Each row represents a different diagnostic category (Normal, OME, COM), while columns Style A, B, and C demonstrate the sampled style vectors while maintaining the same textual prompt and gaussian noise. (Left) Examples of generated COM images by CLUE. Changing two components of style vector from -3 to 3 and fixed the gaussian noise. Images are continuously changed with style.
• Figure 5: PCA visualization of feature space distributions