Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dealing with Missing Modalities in Multimodal Recommendation: a Feature Propagation-based Approach

Daniele Malitesta, Emanuele Rossi, Claudio Pomo, Fragkiskos D. Malliaros, Tommaso Di Noia

TL;DR

This work addresses missing modalities in multimodal recommendations by recasting the problem as missing node features in a graph and adopting FeatProp as a preprocessing step. The authors project the user-item interaction graph to an item-item co-interaction graph, sparsify and normalize it, then iteratively propagate observed multimodal features to impute missing ones before feeding them into a recommender. They implement a refined FeatProp variant and evaluate it as a preprocessing module on MMSSL and FREEDOM across three Amazon datasets, reporting consistent improvements over zeros, mean, and random imputations, especially with MMSSL. The approach is simple, model-agnostic, and scalable, offering a practical way to bolster multimodal RS performance when multimodal content is incomplete or absent in real-world catalogs.

Abstract

Multimodal recommender systems work by augmenting the representation of the products in the catalogue through multimodal features extracted from images, textual descriptions, or audio tracks characterising such products. Nevertheless, in real-world applications, only a limited percentage of products come with multimodal content to extract meaningful features from, making it hard to provide accurate recommendations. To the best of our knowledge, very few attention has been put into the problem of missing modalities in multimodal recommendation so far. To this end, our paper comes as a preliminary attempt to formalise and address such an issue. Inspired by the recent advances in graph representation learning, we propose to re-sketch the missing modalities problem as a problem of missing graph node features to apply the state-of-the-art feature propagation algorithm eventually. Technically, we first project the user-item graph into an item-item one based on co-interactions. Then, leveraging the multimodal similarities among co-interacted items, we apply a modified version of the feature propagation technique to impute the missing multimodal features. Adopted as a pre-processing stage for two recent multimodal recommender systems, our simple approach performs better than other shallower solutions on three popular datasets.

Dealing with Missing Modalities in Multimodal Recommendation: a Feature Propagation-based Approach

TL;DR

This work addresses missing modalities in multimodal recommendations by recasting the problem as missing node features in a graph and adopting FeatProp as a preprocessing step. The authors project the user-item interaction graph to an item-item co-interaction graph, sparsify and normalize it, then iteratively propagate observed multimodal features to impute missing ones before feeding them into a recommender. They implement a refined FeatProp variant and evaluate it as a preprocessing module on MMSSL and FREEDOM across three Amazon datasets, reporting consistent improvements over zeros, mean, and random imputations, especially with MMSSL. The approach is simple, model-agnostic, and scalable, offering a practical way to bolster multimodal RS performance when multimodal content is incomplete or absent in real-world catalogs.

Abstract

Multimodal recommender systems work by augmenting the representation of the products in the catalogue through multimodal features extracted from images, textual descriptions, or audio tracks characterising such products. Nevertheless, in real-world applications, only a limited percentage of products come with multimodal content to extract meaningful features from, making it hard to provide accurate recommendations. To the best of our knowledge, very few attention has been put into the problem of missing modalities in multimodal recommendation so far. To this end, our paper comes as a preliminary attempt to formalise and address such an issue. Inspired by the recent advances in graph representation learning, we propose to re-sketch the missing modalities problem as a problem of missing graph node features to apply the state-of-the-art feature propagation algorithm eventually. Technically, we first project the user-item graph into an item-item one based on co-interactions. Then, leveraging the multimodal similarities among co-interacted items, we apply a modified version of the feature propagation technique to impute the missing multimodal features. Adopted as a pre-processing stage for two recent multimodal recommender systems, our simple approach performs better than other shallower solutions on three popular datasets.
Paper Structure (20 sections, 5 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 20 sections, 5 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction and motivations
  2. Related work
  3. Multimodal recommendation
  4. Missing information in recommendation
  5. Background
  6. Multimodal recommendation
  7. Missing node features in graph learning
  8. Methodology
  9. Missing modalities in multimodal recommendation
  10. FeatProp for multimodal recommendation
  11. Differences between the original and proposed FeatProp
  12. Experimental settings
  13. Datasets
  14. Multimodal recommender systems
  15. Missing modalities settings
  16. ...and 5 more sections

Figures (2)

  • Figure 1: The two versions of the FeatProp algorithm, as proposed (a) in DBLP:conf/log/RossiK0C0B22 for missing node features, and (b) in this paper for missing modalities in multimodal recommendation. As for (b), the user-item graph is first projected into the item-item co-interaction graph; second, the graph is processed through sparsification and normalisation; then, the FeatProp algorithm is applied; finally, the reconstructed multimodal features are used as inputs to power any multimodal recommender system.
  • Figure 2: Recommendation performance variation (calculated as Recall@20) in various settings of datasets, multimodal recommendation models, and missing modalities imputing strategies. Results of those models with * are not computed via mean and standard deviation over the whole set of 5 random samples due to the excessive computational time.