ImageGem: In-the-wild Generative Image Interaction Dataset for Generative Model Personalization

TL;DR

ImageGem introduces the first large-scale, in-the-wild dataset of fine-grained user preferences for generative models, collected from 57K users across 242K LoRAs and 5M images with 3M prompts, complemented by safety checks. It demonstrates three core applications: aggregated preference alignment and DiffusionDPO-style training, retrieval and generative recommendations via hybrid CF/VLM pipelines, and a novel latent LoRA editing framework (Weights2Weights) to personalize diffusion models in a latent weight space. Key contributions include enabling both aggregate and individual-level preference studies, improving alignment and retrieval metrics, and showing a practical path to personalized generative models through W2W space editing. The dataset and methods open new directions for fine-grained personalization in diffusion-based generation and related AI systems, with implications for scalable, explainable, and user-tailored image synthesis.

Abstract

We introduce ImageGem, a dataset for studying generative models that understand fine-grained individual preferences. We posit that a key challenge hindering the development of such a generative model is the lack of in-the-wild and fine-grained user preference annotations. Our dataset features real-world interaction data from 57K users, who collectively have built 242K customized LoRAs, written 3M text prompts, and created 5M generated images. With user preference annotations from our dataset, we were able to train better preference alignment models. In addition, leveraging individual user preference, we investigated the performance of retrieval models and a vision-language model on personalized image retrieval and generative model recommendation. Finally, we propose an end-to-end framework for editing customized diffusion models in a latent weight space to align with individual user preferences. Our results demonstrate that the ImageGem dataset enables, for the first time, a new paradigm for generative model personalization.

Figures (14)

• Figure 1: Our proposed ImageGem dataset and its applications. The left side illustrates image and generative model retrieval. On the right, we demonstrate a novel task of generative model personalization through LoRA weights-to-weights (W2W) space construction.
• Figure 2: WizMap wangWizMapScalableInteractive2023-based visualization for our ImageGem dataset, divided into two parts. The left panel shows a UMAP embedding of 1M images sampled from the dataset, while the right panel illustrates a contour plot of LoRA model checkpoints. Both visualizations use grid tiles to display key words extracted from image prompts or model tags.
• Figure 3: Qualitative DiffusionDPO results comparison of images generated with OOD prompts in three topics sampled from DiffusionDB. For each prompt, random seed and all other hyperparameters are kept the same. Zoom in for the best view.
• Figure 4: Perform tuning along the direction real-ani (top) and direction ani-real (bottom). The first column displays images generated by the unedited SDXL base model, while the subsequent columns show images generated by progressively edited models with increasing tuning strength. Each row shares the same generation seed for consistency.
• Figure 5: The distribution of prompts based on their predicted probabilites for NSFW content using Detoxify Detoxify. The y-axis represents the count of propmts in logarithmic scale.