Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluation of Multilingual LLMs Personalized Text Generation Capabilities Targeting Groups and Social-Media Platforms

Dominik Macko

TL;DR

Multilingual LLMs enable rapid text generation across languages, raising risks of targeted disinformation and manipulation. The authors conduct a large-scale multilingual evaluation of two personalization types, group- and platform-personalization, across 10 languages using 16 open-weight LLMs and 1080 prompt combinations. They find that platform personalization has a stronger impact on detectability across languages, with English exhibiting the highest personalization quality and Ukrainian being the most difficult to detect. The study informs policy, detector development, and the safe deployment of multilingual LLMs in cross-language contexts, and highlights the value of cross-linguistic personalization, evaluation methods, and detector robustness.

Abstract

Capabilities of large language models to generate multilingual coherent text have continuously enhanced in recent years, which opens concerns about their potential misuse. Previous research has shown that they can be misused for generation of personalized disinformation in multiple languages. It has also been observed that personalization negatively affects detectability of machine-generated texts; however, this has been studied in the English language only. In this work, we examine this phenomenon across 10 languages, while we focus not only on potential misuse of personalization capabilities, but also on potential benefits they offer. Overall, we cover 1080 combinations of various personalization aspects in the prompts, for which the texts are generated by 16 distinct language models (17,280 texts in total). Our results indicate that there are differences in personalization quality of the generated texts when targeting demographic groups and when targeting social-media platforms across languages. Personalization towards platforms affects detectability of the generated texts in a higher scale, especially in English, where the personalization quality is the highest.

Evaluation of Multilingual LLMs Personalized Text Generation Capabilities Targeting Groups and Social-Media Platforms

TL;DR

Multilingual LLMs enable rapid text generation across languages, raising risks of targeted disinformation and manipulation. The authors conduct a large-scale multilingual evaluation of two personalization types, group- and platform-personalization, across 10 languages using 16 open-weight LLMs and 1080 prompt combinations. They find that platform personalization has a stronger impact on detectability across languages, with English exhibiting the highest personalization quality and Ukrainian being the most difficult to detect. The study informs policy, detector development, and the safe deployment of multilingual LLMs in cross-language contexts, and highlights the value of cross-linguistic personalization, evaluation methods, and detector robustness.

Abstract

Capabilities of large language models to generate multilingual coherent text have continuously enhanced in recent years, which opens concerns about their potential misuse. Previous research has shown that they can be misused for generation of personalized disinformation in multiple languages. It has also been observed that personalization negatively affects detectability of machine-generated texts; however, this has been studied in the English language only. In this work, we examine this phenomenon across 10 languages, while we focus not only on potential misuse of personalization capabilities, but also on potential benefits they offer. Overall, we cover 1080 combinations of various personalization aspects in the prompts, for which the texts are generated by 16 distinct language models (17,280 texts in total). Our results indicate that there are differences in personalization quality of the generated texts when targeting demographic groups and when targeting social-media platforms across languages. Personalization towards platforms affects detectability of the generated texts in a higher scale, especially in English, where the personalization quality is the highest.
Paper Structure (17 sections, 11 figures, 7 tables)

This paper contains 17 sections, 11 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generation of Personalized Text
  4. Evaluation of Text Personalization
  5. Personalized Texts Detectability
  6. Comparison to SOTA
  7. Dataset
  8. Data Analysis
  9. Evaluation Methodology
  10. Results
  11. Personalization Types Differences (RQ1)
  12. Differences Across Languages (RQ2)
  13. Detectability Analysis (RQ3)
  14. Conclusions
  15. Computational Resources
  16. ...and 2 more sections

Figures (11)

  • Figure 1: Per-generator comparison of stance towards the narrative.
  • Figure 2: Per-generator comparison of personalization capabilities targeting groups (top, not including texts without target-group personalization request) and platforms (bottom).
  • Figure 3: Per-language comparison of personalization capabilities targeting groups (top, not including texts without target-group personalization request) and platforms (bottom).
  • Figure 4: Per-generator comparison of stance towards the narrative without safety-filtered and noisy samples.
  • Figure 5: Per-intended-stance comparison of metaevaluated stance towards the narrative without safety-filtered and noisy samples.
  • ...and 6 more figures