Mitigating Cross-modal Representation Bias for Multicultural Image-to-Recipe Retrieval

TL;DR

The paper addresses bias in cross-modal image-to-recipe retrieval arising from the unequal visual visibility of ingredients and cooking actions. It introduces a causal view and backdoor adjustment to refine similarity by injecting ingredient- and action-derived debiasing terms, implemented via culture-specific dictionaries and plug-in neural modules within multilingual CLIP-based backbones. Key contributions include a formal causal formulation, scalable plug-and-play debiasing components for monolingual and multicultural retrieval, and a newly curated Cookpad-based multicultural dataset with zero-shot evaluation. Results show consistent gains in retrieval metrics, with the largest improvements when both ingredient and action debiasing are applied, especially for low-resource cuisines, highlighting the method's practical impact in multilingual, culturally diverse settings.

Abstract

Existing approaches for image-to-recipe retrieval have the implicit assumption that a food image can fully capture the details textually documented in its recipe. However, a food image only reflects the visual outcome of a cooked dish and not the underlying cooking process. Consequently, learning cross-modal representations to bridge the modality gap between images and recipes tends to ignore subtle, recipe-specific details that are not visually apparent but are crucial for recipe retrieval. Specifically, the representations are biased to capture the dominant visual elements, resulting in difficulty in ranking similar recipes with subtle differences in use of ingredients and cooking methods. The bias in representation learning is expected to be more severe when the training data is mixed of images and recipes sourced from different cuisines. This paper proposes a novel causal approach that predicts the culinary elements potentially overlooked in images, while explicitly injecting these elements into cross-modal representation learning to mitigate biases. Experiments are conducted on the standard monolingual Recipe1M dataset and a newly curated multilingual multicultural cuisine dataset. The results indicate that the proposed causal representation learning is capable of uncovering subtle ingredients and cooking actions and achieves impressive retrieval performance on both monolingual and multilingual multicultural datasets.

Figures (11)

• Figure 1: Left: Causal graph with ingredients and actions as confounders. Right: Backdoor adjustment mitigates spurious correlations by removing incoming edges to the image node.
• Figure 2: The proposed framework for multicultural recipe retrieval. Given a query image of Culture X, the ingredient and action debiasing modules in the culture will derive the embeddings $e_{Act}$ and $e_{Ing}$, respectively. The two embeddings are then added to the image embedding, $e_I$, learnt globally in the cross-modal retrieval module for alleviating representation biases. Please refer to Figure \ref{['fig:subfig_ingre_debiasing']} and Figure \ref{['fig:subfig_action_debiasing']} for the architectures of ingredient debiaisng module and action debiasing module, respectively.
• Figure 3: The ingredient debiasing module takes image embeddings $e_I$ act as keys and values, and ingredient label embeddings as queries. The decoder output is passed to a sigmoid to produce ingredient probabilities $P_{ing}$, which weight the dictionary $D_{ing}$ to yield the debiasing embedding $e_{Ing}$.
• Figure 4: The action debiasing takes the predicted ingredients as input. For each ingredient $e_{Ing_k}$, we generate the sequence of cooking actions, and then retrieve the corresponding action embeddings from the dictionary $D_{act}$. We normalize the action prediction probabilities to weight each action embedding, and then compute a weighted sum of the embeddings to obtain the action embedding $e^k_{Act}$. The final action embedding, $e_{Act}$, used to enhance the image representation, is obtained by first normalizing the ingredient probabilities and then using probabilities to compute a weighted sum of the action embeddings, $e^k_{Act}$, corresponding to each ingredient. Decoders are shared by all ingredients for generation.
• Figure 5: Example showing how the ingredient and action debiasing disambiguates similar recipes. The first row displays the query image, predicted ingredients, and predicted actions. The second row is the retrieved recipe by H-T+ingredient and H-T+action, while the last row is the recipe retrieved by H-T+both. The correctly predicted ingredients and cooking actions are bolded. The predicted ingredients and cooking actions are marked in red and blue, respectively, in the recipes. The ground-truth is boxed in blue.