Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multilingual Prompts in LLM-Based Recommenders: Performance Across Languages

Makbule Gulcin Ozsoy

TL;DR

This work highlights the need for diverse language support in LLM-based recommenders and suggests future research on creating evaluation datasets, using newer models and additional languages.

Abstract

Large language models (LLMs) are increasingly used in natural language processing tasks. Recommender systems traditionally use methods such as collaborative filtering and matrix factorization, as well as advanced techniques like deep learning and reinforcement learning. Although language models have been applied in recommendation, the recent trend have focused on leveraging the generative capabilities of LLMs for more personalized suggestions. While current research focuses on English due to its resource richness, this work explores the impact of non-English prompts on recommendation performance. Using OpenP5, a platform for developing and evaluating LLM-based recommendations, we expanded its English prompt templates to include Spanish and Turkish. Evaluation on three real-world datasets, namely ML1M, LastFM, and Amazon-Beauty, showed that usage of non-English prompts generally reduce performance, especially in less-resourced languages like Turkish. We also retrained an LLM-based recommender model with multilingual prompts to analyze performance variations. Retraining with multilingual prompts resulted in more balanced performance across languages, but slightly reduced English performance. This work highlights the need for diverse language support in LLM-based recommenders and suggests future research on creating evaluation datasets, using newer models and additional languages.

Multilingual Prompts in LLM-Based Recommenders: Performance Across Languages

TL;DR

This work highlights the need for diverse language support in LLM-based recommenders and suggests future research on creating evaluation datasets, using newer models and additional languages.

Abstract

Large language models (LLMs) are increasingly used in natural language processing tasks. Recommender systems traditionally use methods such as collaborative filtering and matrix factorization, as well as advanced techniques like deep learning and reinforcement learning. Although language models have been applied in recommendation, the recent trend have focused on leveraging the generative capabilities of LLMs for more personalized suggestions. While current research focuses on English due to its resource richness, this work explores the impact of non-English prompts on recommendation performance. Using OpenP5, a platform for developing and evaluating LLM-based recommendations, we expanded its English prompt templates to include Spanish and Turkish. Evaluation on three real-world datasets, namely ML1M, LastFM, and Amazon-Beauty, showed that usage of non-English prompts generally reduce performance, especially in less-resourced languages like Turkish. We also retrained an LLM-based recommender model with multilingual prompts to analyze performance variations. Retraining with multilingual prompts resulted in more balanced performance across languages, but slightly reduced English performance. This work highlights the need for diverse language support in LLM-based recommenders and suggests future research on creating evaluation datasets, using newer models and additional languages.
Paper Structure (14 sections, 1 figure, 6 tables)

This paper contains 14 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Integration of Recommender Systems and LLMs
  4. LLM-Based Recommenders and Their Challenges
  5. Challenges of LLM-based Recommenders
  6. LLMs and Non-English Content
  7. LLM-based Recommenders Beyond English
  8. Experimental Setup
  9. Impact of Non-English Prompts on a PreTrained LLM-Based Recommender
  10. Observations
  11. Impact of English and Non-English Prompts on LLM Recommender Fine-Tuning
  12. Observations
  13. Analysis and Discussion
  14. Conclusion

Figures (1)

  • Figure 1: Four phases in the evolution of LLM-based recommenders: (i) Initial Phase: Traditional methods generate suggestions based on user actions like clicks. (ii) LLM Integration: LLMs enhance recommendations using prompts. (iii) Prompt Template Interaction: Users request recommendations via natural language templates. (iv) Natural Language Interaction: Direct native language communication with the system.