Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Trustworthy Multimodal Recommendation

Zixuan Li

TL;DR

This paper tackles trustworthiness in multimodal recommender systems by addressing both content and interaction signals. It introduces a plug-and-play Modality-level Rectification Module that learns soft item-modality correspondences and uses anchor-based alignment, sparse affinity, and Sinkhorn-based soft matching to rectify misaligned modality signals without changing backbone architectures. It also analyzes interaction-level trust through stress-testing with collaborative priors and similarity-based graph edits, revealing that such edits can help or harm robustness depending on prior-signal alignment and whether editing targets supervision or propagation. Empirically, the approach improves robustness across multiple datasets and backbones under modality corruption and provides practical guidelines for applying interaction-level edits, underscoring the importance of cautious, alignment-aware deployment in real-world systems.

Abstract

Recent advances in multimodal recommendation have demonstrated the effectiveness of incorporating visual and textual content into collaborative filtering. However, real-world deployments raise an increasingly important yet underexplored issue: trustworthiness. On modern e-commerce platforms, multimodal content can be misleading or unreliable (e.g., visually inconsistent product images or click-bait titles), injecting untrustworthy signals into multimodal representations and making existing recommenders brittle under modality corruption. In this work, we take a step towards trustworthy multimodal recommendation from both a method and an analysis perspective. First, we propose a plug-and-play modality-level rectification component that mitigates untrustworthy modality features by learning soft correspondences between items and multimodal features. Using lightweight projections and Sinkhorn-based soft matching, the rectification suppresses mismatched modality signals while preserving semantic consistency, and can be integrated into existing multimodal recommenders without architectural modifications. Second, we present two practical insights on interaction-level trustworthiness under noisy collaborative signals: (i) training-set pseudo interactions can help or hurt performance under noise depending on prior-signal alignment; and (ii) propagation-graph pseudo edges can also help or hurt robustness, as message passing may amplify misalignment. Extensive experiments on multiple datasets and backbones under varying corruption levels demonstrate improved robustness from modality rectification and validate the above interaction-level observations.

Towards Trustworthy Multimodal Recommendation

TL;DR

This paper tackles trustworthiness in multimodal recommender systems by addressing both content and interaction signals. It introduces a plug-and-play Modality-level Rectification Module that learns soft item-modality correspondences and uses anchor-based alignment, sparse affinity, and Sinkhorn-based soft matching to rectify misaligned modality signals without changing backbone architectures. It also analyzes interaction-level trust through stress-testing with collaborative priors and similarity-based graph edits, revealing that such edits can help or harm robustness depending on prior-signal alignment and whether editing targets supervision or propagation. Empirically, the approach improves robustness across multiple datasets and backbones under modality corruption and provides practical guidelines for applying interaction-level edits, underscoring the importance of cautious, alignment-aware deployment in real-world systems.

Abstract

Recent advances in multimodal recommendation have demonstrated the effectiveness of incorporating visual and textual content into collaborative filtering. However, real-world deployments raise an increasingly important yet underexplored issue: trustworthiness. On modern e-commerce platforms, multimodal content can be misleading or unreliable (e.g., visually inconsistent product images or click-bait titles), injecting untrustworthy signals into multimodal representations and making existing recommenders brittle under modality corruption. In this work, we take a step towards trustworthy multimodal recommendation from both a method and an analysis perspective. First, we propose a plug-and-play modality-level rectification component that mitigates untrustworthy modality features by learning soft correspondences between items and multimodal features. Using lightweight projections and Sinkhorn-based soft matching, the rectification suppresses mismatched modality signals while preserving semantic consistency, and can be integrated into existing multimodal recommenders without architectural modifications. Second, we present two practical insights on interaction-level trustworthiness under noisy collaborative signals: (i) training-set pseudo interactions can help or hurt performance under noise depending on prior-signal alignment; and (ii) propagation-graph pseudo edges can also help or hurt robustness, as message passing may amplify misalignment. Extensive experiments on multiple datasets and backbones under varying corruption levels demonstrate improved robustness from modality rectification and validate the above interaction-level observations.
Paper Structure (51 sections, 20 equations, 5 figures, 2 tables)

This paper contains 51 sections, 20 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Recommendation
  4. Noisy Correspondence and Multimodal Mismatch
  5. Preliminaries
  6. Setting and notation.
  7. Recommendation objective.
  8. Trustworthiness-aware corruptions.
  9. Methodology
  10. Backbone Multimodal Recommender
  11. Offline plug-and-play integration.
  12. Modality-level Rectification Module
  13. Anchor embeddings.
  14. Step 1: Robust modality-to-anchor projection.
  15. Step 2: Sparse affinity construction.
  16. ...and 36 more sections

Figures (5)

  • Figure 1: Illustration of untrustworthy signals in multimodal recommendation. (1) Multimodal semantic inconsistency between product images and textual descriptions. (2) Unreliable user--item interactions, where weakly related but plausible interactions introduce low-confidence edges that may pollute message passing.
  • Figure 2: Robustness under modality misalignment. Recall@10 of FREEDOM and VBPR on Baby and Sports as the modality corruption ratio $\eta_m$ increases. Base. denotes the original backbone using corrupted features, and Rect. applies our Modality-level Rectification as offline preprocessing.
  • Figure 3: Hyperparameter sensitivity of MR under modality corruption ($\eta_m=0.2$) on FREEDOM. We vary the mix ratio $\lambda$ and temperature $\tau$, and report Recall@10 and NDCG@10 on Baby and Sports.
  • Figure 4: Interaction-level stress test on Baby with MGCN. We report Recall@10 as the interaction noise ratio $\eta_e$ varies. Base. is the original backbone; Add. and Prune. apply similarity-based relation completion or edge pruning using a LightGCN prior. Train-only edits BPR positives, Graph-only edits the propagation graph, and Both edits both supervision and propagation.
  • Figure 5: Ablation study of MR under modality misalignment ($\eta_m=20\%$) with FREEDOM on Baby and Sports. We report Recall@10 and NDCG@10. Base: w/o MR; Full: full MR; w/o Sink: replace Sinkhorn matching with row-normalized affinity; w/o SL: remove small-loss selection in projection training.