Neural Additive Image Model: Interpretation through Interpolation

TL;DR

This work addresses the challenge of interpreting image-driven predictions in multi-modal settings by introducing the Neural Additive Image Model (NAIM), which couples Neural Additive Models with Diffusion Autoencoders to yield globally interpretable image effects within an additive framework. NAIM encodes images into a semantically meaningful latent space using a Diffusion Autoencoder and models image effects with a dedicated function $f_{img}$ on the latent code $\bm{z}$, preserving additivity for global interpretability. The authors validate the approach on synthetic data, showing accurate recovery of both numerical and image effects, and apply it to Airbnb pricing with host images, achieving higher $R^2$ than baselines and enabling both global and local interpretability through latent-space interpolation and attribute manipulation. This approach enables transparent analysis of image contributions in high-stakes domains and offers a practical path toward bias detection and fairness in multi-modal predictive systems.

Abstract

Understanding how images influence the world, interpreting which effects their semantics have on various quantities and exploring the reasons behind changes in image-based predictions are highly difficult yet extremely interesting problems. By adopting a holistic modeling approach utilizing Neural Additive Models in combination with Diffusion Autoencoders, we can effectively identify the latent hidden semantics of image effects and achieve full intelligibility of additional tabular effects. Our approach offers a high degree of flexibility, empowering us to comprehensively explore the impact of various image characteristics. We demonstrate that the proposed method can precisely identify complex image effects in an ablation study. To further showcase the practical applicability of our proposed model, we conduct a case study in which we investigate how the distinctive features and attributes captured within host images exert influence on the pricing of Airbnb rentals.

Figures (21)

• Figure 1: Effect on the price of an Airbnb listing obtained by interpolating between two input images. A Neural Additive Model that includes an image effect is used to predict the expected price. Image Interpolating is achieved by linearly interpolating between image representations in the embedding space of a Diffusion Autoencoder.
• Figure 2: Model architecture of the base version of the proposed approach not accounting for interaction effects. The tabular data is fit as independent MLPs, the images are encoded with an CNN encoder and fit with a separate MLP. All feature effects are summed up for the model prediction.
• Figure 3: Interpolating between the latent semantic codes $\bm{z}_{int}^{(1)}$ and $\bm{z}_{int}^{(k)}$ in the embedding space can be visualize in pixel-space as the sequence $\left( \bm{\tilde{x}}_{img}^{(1)},\bm{\tilde{x}}_{img}^{(2)}, \ldots, \bm{\tilde{x}}_{img}^{(k)} \right)$ by using the decoder $\tilde{\bm{E}}_{\phi}^{-1}$ on the latent codes. Each latent code $\bm{z}_{int}$, on the other hand, also allows to make the prediction $f_{img}(\bm{z}_{int})$ for a response variable of interest. Plotting the predictions $\left( f_{img}(\bm{z}_{int}^{(1)}), f_{img}(\bm{z}_{int}^{(2)}), \ldots, f_{img}(\bm{z}_{int}^{(k)}) \right)$ against the generated images $\left( \bm{\tilde{x}}_{img}^{(1)},\bm{\tilde{x}}_{img}^{(2)}, \ldots, \bm{\tilde{x}}_{img}^{(k)} \right)$ allows to investigate the image effect.
• Figure 4: Effect of attribute Manipulation for the feature Chubby on the expected rental price. Through (semi) continuous interpolation, an effect trend can be visualized.
• Figure 5: The NAIM models are able to almost perfectly recover the effects of numerical covariates even if an additional image-covariate is present. Effect 1 is a simple linear effect of the form $f_1(x) = 2x$, effect 2 is a power function $f_2(x) = x^2$ and effect three has a sinusoidal form $f_3(x) = \sin (2 \pi x)$. This plot shows the discovered numerical effects for the squares-data $\mathcal{D}_{\text{squares}}$ with $f_{img}(x) = 2x^4$ as the image effect of the x-coordinate of the center of the white square.