From Raw Features to Effective Embeddings: A Three-Stage Approach for Multimodal Recipe Recommendation
Jeeho Shin, Kyungho Kim, Kijung Shin
TL;DR
This paper addresses the challenge of leveraging rich multimodal features for recipe recommendation beyond user–recipe interactions. It introduces TESMR, a three-stage framework that first creates content-based summaries via foundation models, then propagates these summaries over the user–recipe interaction graph for relation-based enhancement, and finally aligns content-based and learnable embeddings through cross-view contrastive learning. The approach yields 7–15% gains in Recall@10 over eight baselines on two real-world datasets, with ablations highlighting the critical role of relation-based propagation and user reviews. The work demonstrates a practical path to robust multimodal recipe recommendation and underscores the value of combining foundation-model-derived signals with collaborative learning.
Abstract
Recipe recommendation has become an essential task in web-based food platforms. A central challenge is effectively leveraging rich multimodal features beyond user-recipe interactions. Our analysis shows that even simple uses of multimodal signals yield competitive performance, suggesting that systematic enhancement of these signals is highly promising. We propose TESMR, a 3-stage framework for recipe recommendation that progressively refines raw multimodal features into effective embeddings through: (1) content-based enhancement using foundation models with multimodal comprehension, (2) relation-based enhancement via message propagation over user-recipe interactions, and (3) learning-based enhancement through contrastive learning with learnable embeddings. Experiments on two real-world datasets show that TESMR outperforms existing methods, achieving 7-15% higher Recall@10.