CUPID: Contextual Understanding of Prompt-conditioned Image Distributions

TL;DR

CUPID addresses the challenge of understanding prompt-conditioned image distributions from text-to-image diffusion models by focusing on object-level densities. It introduces density-based embeddings (dSNE) that map high-dimensional object features to a low-dimensional space while preserving density estimates, enabling analysis of typical vs. rare object styles and object dependencies via joint densities $P_d(z_{i,s}, z_{j,t})$ and marginals $P_d(z_{i,s})$. The method relies on RAM-based object localization and Swin features to extract object representations, and KDE-based density estimation to quantify distributions, with conditional densities and PMI to reveal dependencies. An interactive visualization combines 1D violin density plots, 2D marginalized density scatterplots, linked brushing, and projection steering to verify prompt faithfulness, discover unspecified objects, and analyze scene-assembly biases.results on SDXL-generated scenes reveal language-understanding gaps and biases in object co-occurrence and perspective, illustrating CUPID’s potential as a diagnostic and discovery tool for prompt-conditioned image synthesis.

Abstract

We present CUPID: a visualization method for the contextual understanding of prompt-conditioned image distributions. CUPID targets the visual analysis of distributions produced by modern text-to-image generative models, wherein a user can specify a scene via natural language, and the model generates a set of images, each intended to satisfy the user's description. CUPID is designed to help understand the resulting distribution, using contextual cues to facilitate analysis: objects mentioned in the prompt, novel, synthesized objects not explicitly mentioned, and their potential relationships. Central to CUPID is a novel method for visualizing high-dimensional distributions, wherein contextualized embeddings of objects, those found within images, are mapped to a low-dimensional space via density-based embeddings. We show how such embeddings allows one to discover salient styles of objects within a distribution, as well as identify anomalous, or rare, object styles. Moreover, we introduce conditional density embeddings, whereby conditioning on a given object allows one to compare object dependencies within the distribution. We employ CUPID for analyzing image distributions produced by large-scale diffusion models, where our experimental results offer insights on language misunderstanding from such models and biases in object composition, while also providing an interface for discovery of typical, or rare, synthesized scenes.

Figures (9)

• Figure 1: CUPID aims to support users in understanding image distributions produced by generative models. For the given prompt describing the scene (top), one may brush 1D density embeddings of objects unspecified in the prompt, e.g. carpet (A), resulting in linked updates to the remaining views, e.g. 1D densities for specified objects (B). By selecting an image corresponding to the brushed object (C), we can show more detailed information on object relationships (D), here highlighting dependencies that exist between the selected carpet object, and all couches in the distribution.
• Figure 2: Our analysis of prompt-conditioned image distributions is organized around (1) objects that appear in the distribution, whether specified in the prompt or not, and (2) the properties of objects, specified or unspecified. Ideally, generated images are consistent with the prompt (A), while still exhibiting diversity characteristic of the scene (D). Issues that arise can be due to language misunderstanding, whether omitting a specified object's property (C), or biasing the properties of an unspecified object (B).
• Figure 3: Density estimation enables us to quantify what is typical, and what is rare, within a given image distribution. For instance, ceiling fan objects whose lights are turned on can have similar feature representations, and thus report high density. In contrast, ceiling fans and lamps whose lights are both turned on will report a low joint density, thus reflecting a rare object relationship.
• Figure 4: We compare our density-based embedding approach (dSNE) to that of tSNE, where the provided density is both size and color-encoded in the plots. Across varying bandwidths in the KDE ($h = 40$ top row, $h= 80$ bottom row), our method obtains superior results in density preservation.
• Figure 5: An overview of the design for CUPID. (a) Density-based object embeddings are shown as violin plots for both prompt objects and discovered objects, with object-marginalized density embeddings shown as 2D scatterplots. (b) Our interface supports linked brushing to relate density representations, in this instance highlighting a rare Shih Tzu that has an angular limb deformity. (c) Further, the interface allows for projection steering via conditioning on a selected image of a particular object, here for the paw feature of the chosen dog.