Deciphering Personalization: Towards Fine-Grained Explainability in Natural Language for Personalized Image Generation Models

TL;DR

This work addresses the lack of fine-grained explainability for personalized image generation by introducing FineXL, a training-free framework that explains how a base model is personalized via distributional divergence $Div\left[p_{personal}(x|t) \| p_{base}(x|t)\right]$ as a linear combination of orthogonal low-level concepts mapped into a shared representation space. The method first quantifies divergence with a high-level image-embedding $V_{div}$, then leverages a vision-language model to extract low-level concepts, maps them into the same space via $f(C)$, enforces orthogonality, and finally decomposes $V_{div}$ as $\mathbf{V}_{div}=\sum_i w_i f(C_i)$ with a residual bound $e_{decomp}$. Across diffusion, GAN, and autoregressive image-generation models and multiple datasets (synthetic, Style Transfer, WikiArt), FineXL achieves up to 56% improvement in explanation accuracy for single-aspect personalization and at least 50% reduction in error for multi-aspect scenarios, demonstrating strong generalizability and practical potential for model selection, debugging, and user understanding. The approach relies on a linear representation hypothesis and careful alignment between text and image encoders, with GPT-4o typically providing the strongest low-level concept discovery among tested VLMs and CLIP-family encoders offering the best alignment performance.

Abstract

Image generation models are usually personalized in practical uses in order to better meet the individual users' heterogeneous needs, but most personalized models lack explainability about how they are being personalized. Such explainability can be provided via visual features in generated images, but is difficult for human users to understand. Explainability in natural language is a better choice, but the existing approaches to explainability in natural language are limited to be coarse-grained. They are unable to precisely identify the multiple aspects of personalization, as well as the varying levels of personalization in each aspect. To address such limitation, in this paper we present a new technique, namely \textbf{FineXL}, towards \textbf{Fine}-grained e\textbf{X}plainability in natural \textbf{L}anguage for personalized image generation models. FineXL can provide natural language descriptions about each distinct aspect of personalization, along with quantitative scores indicating the level of each aspect of personalization. Experiment results show that FineXL can improve the accuracy of explainability by 56\%, when different personalization scenarios are applied to multiple types of image generation models.

Figures (16)

• Figure 1: Importance of explainability for personalized image generation models to be used by human users
• Figure 1: Error (MAE) of identifying the varying levels of personalization in one aspect
• Figure 2: Personalized image generation in multiple aspects with varying levels
• Figure 3: FineXL: fine-grained explanations of a personalized image generation model
• Figure 4: Our design of FineXL: the divergence between pre-trained and personalized models' output distributions is first converted into a high-level representation, which is then linearly decomposed into a set of low-level concepts in natural language about personalization that are suggested by a VLM. More details about this design and each step can be found in Algorithm \ref{['alg:overall']}.