Diffusion Models as Data Mining Tools

TL;DR

The paper tackles scalable visual data mining by leveraging diffusion models trained for image synthesis. It finetunes conditional latent diffusion models on target datasets and defines a pixel-level typicality score to identify the most representative visual elements, then mines patches and clusters them with DIFT embeddings to summarize data. The authors demonstrate the approach on four diverse datasets (Cars, Faces, Geo, Places) and show its ability to translate visual elements across locations and localize pathologies in medical images without localization supervision. Finetuning is essential to mitigate base model biases and improve cross-label translation, yielding semantically meaningful clusters and scalable summaries. Overall, the work presents a general, scalable framework for extracting informative visual patterns from large, heterogeneous image collections using diffusion-model-based data mining.

Abstract

This paper demonstrates how to use generative models trained for image synthesis as tools for visual data mining. Our insight is that since contemporary generative models learn an accurate representation of their training data, we can use them to summarize the data by mining for visual patterns. Concretely, we show that after finetuning conditional diffusion models to synthesize images from a specific dataset, we can use these models to define a typicality measure on that dataset. This measure assesses how typical visual elements are for different data labels, such as geographic location, time stamps, semantic labels, or even the presence of a disease. This analysis-by-synthesis approach to data mining has two key advantages. First, it scales much better than traditional correspondence-based approaches since it does not require explicitly comparing all pairs of visual elements. Second, while most previous works on visual data mining focus on a single dataset, our approach works on diverse datasets in terms of content and scale, including a historical car dataset, a historical face dataset, a large worldwide street-view dataset, and an even larger scene dataset. Furthermore, our approach allows for translating visual elements across class labels and analyzing consistent changes.

Figures (11)

• Figure 1: Mining typical visual elements with diffusion models. We demonstrate how to use diffusion models to mine visual data through a simple pixel-based score and a standard clustering approach. We present high-quality mining results for a diverse range of datasets (from left to right: 10,130 photographs of cars tagged with a creation year between 1920-1999 cardb, 24,874 portraits from the 19th to the 21st century ftt, 344,224 Street View images tagged with country names g3, and 1,803,460 images of scenes images associated with descriptive names places365). Our results highlight both expected elements and more unforeseen ones.
• Figure 2: Typical elements are informative of the conditioning label. We visualize the top-6 patches ranked according to typicality (${\mathbf{T}}$) with respect to the conditioning class label, negative typicality ($-{\mathbf{T}}$), and randomly (Rand.). The two rows correspond to different classes from each of the four datasets.
• Figure 3: Effect of finetuning. (a) For the same USA image (top), finetuning changes the spatial allocation of typicality before (middle) and after (bottom) finetuning. (b) This results in different typical clusters (USA), which, after finetuning (bottom), select for more typical elements like mailboxes. (c) Translation (Sec. \ref{['sub:trends']}) of a picture of a road from France (top) to Thailand without finetuning (middle) suffers from data biases in the base model turning the road into a river and erasing utility poles. After finetuning on the G^3 dataset (bottom), the translated image is more consistent with the original.
• Figure 4: Clusters of CarDB cardb visual elements. Our visual summaries of typical car elements show elements unique to a period and elements that evolve with time. Evolving elements include the shapes of the car's body or headlights, which are parts of the 6 most typical clusters for most periods. More specific elements include running boards in the 1920s ((a), 6th row) or large engine side grills in the 1930s ((b), 3rd, 4th and 6th row). In the 1980s (c), we observe two typical yet very discrete clusters of car design styles, of the curvy French 2CV (1-4 row) juxtaposed to the square American chevy-style cars (5-6 rows).
• Figure 5: Clusters of FTT ftt visual elements. Our cluster analysis of faces revealed that eyeglasses of varying designs are indicative of a portrait's decade throughout the history captured by FTT. Observing the 6 most typical clusters for the 1920s (a), the 1940s (b), and the 1950s (c), we see how the shape of glasses is highly informative of each period. We also located fashion items that uniquely trended only in a particular period, such as aviator goggles in the 1920s (2nd row), military caps in the 1940s (1st and 2nd row), and baseball caps in the 1950s (1st row). Consistent with prior analysis ginosar2017yearbooks, we also found clusters corresponding to smiles and makeup.