Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge

Li Zhou, Taelin Karidi, Wanlong Liu, Nicolas Garneau, Yong Cao, Wenyu Chen, Haizhou Li, Daniel Hershcovich

TL;DR

This work investigates cultural biases in Large Language Models (LLMs) within the universally relevant food domain by introducing FmLAMA, a multilingual, Wikidata-derived dataset of dishes, origins, and ingredients. It proposes a Word Prediction probing framework with a MASK-based setup, multiple prompt templates, and codified evaluation metrics ($mAP$, $mWS$, and MES) across six languages and code-switching scenarios. Key findings reveal a pronounced bias toward American culinary knowledge in English prompts, which can be mitigated by incorporating explicit cultural context and multilingual prompts, though results still depend on model architecture and probing language. The study contributes a scalable data-collection pipeline, novel cultural-probing metrics, and actionable insights for reducing cross-cultural biases in LLMs, with potential impact on multicultural deployment and evaluation of AI systems.

Abstract

Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain, a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs' ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.

Does Mapo Tofu Contain Coffee? Probing LLMs for Food-related Cultural Knowledge

TL;DR

This work investigates cultural biases in Large Language Models (LLMs) within the universally relevant food domain by introducing FmLAMA, a multilingual, Wikidata-derived dataset of dishes, origins, and ingredients. It proposes a Word Prediction probing framework with a MASK-based setup, multiple prompt templates, and codified evaluation metrics (, , and MES) across six languages and code-switching scenarios. Key findings reveal a pronounced bias toward American culinary knowledge in English prompts, which can be mitigated by incorporating explicit cultural context and multilingual prompts, though results still depend on model architecture and probing language. The study contributes a scalable data-collection pipeline, novel cultural-probing metrics, and actionable insights for reducing cross-cultural biases in LLMs, with potential impact on multicultural deployment and evaluation of AI systems.

Abstract

Recent studies have highlighted the presence of cultural biases in Large Language Models (LLMs), yet often lack a robust methodology to dissect these phenomena comprehensively. Our work aims to bridge this gap by delving into the Food domain, a universally relevant yet culturally diverse aspect of human life. We introduce FmLAMA, a multilingual dataset centered on food-related cultural facts and variations in food practices. We analyze LLMs across various architectures and configurations, evaluating their performance in both monolingual and multilingual settings. By leveraging templates in six different languages, we investigate how LLMs interact with language-specific and cultural knowledge. Our findings reveal that (1) LLMs demonstrate a pronounced bias towards food knowledge prevalent in the United States; (2) Incorporating relevant cultural context significantly improves LLMs' ability to access cultural knowledge; (3) The efficacy of LLMs in capturing cultural nuances is highly dependent on the interplay between the probing language, the specific model architecture, and the cultural context in question. This research underscores the complexity of integrating cultural understanding into LLMs and emphasizes the importance of culturally diverse datasets to mitigate biases and enhance model performance across different cultural domains.
Paper Structure (49 sections, 8 equations, 28 figures, 14 tables)

This paper contains 49 sections, 8 equations, 28 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cultural knowledge datasets.
  4. Knowledge probing.
  5. FmLAMA Construction
  6. Step #1: Obtain countries set.
  7. Step #2: Acquire food instances.
  8. Cultural Knowledge Probing
  9. Probing task
  10. MASK operation.
  11. Probability acquisition.
  12. Probing template
  13. Cultural information.
  14. Multilingual prompts.
  15. Code-switching.
  16. ...and 34 more sections

Figures (28)

  • Figure 1: Summary of the various aspects of our work.
  • Figure 2: Comparative probing results on FmLAMA-en: incorporating cultural information about the origin of dishes into English prompts can enhance the probing of cultural knowledge. Significance levels are indicated by $*$ for $p < 0.1$, $**$ for $p < 0.05$, and $***$ for $p < 0.01$.
  • Figure 3: Comparison of probing results on Llama3 with prompts in different languages, showing that the English prompt exhibits the best performance.
  • Figure 4: Probing results using code-switching prompts on Qwen2 and LLaMA3.
  • Figure 5: Examples of FmLAMA.
  • ...and 23 more figures