Large Language Models for Mental Health: A Multilingual Evaluation

TL;DR

This study tackles the challenge of applying large language models to multilingual mental health detection by evaluating seven LLMs across eight non-English datasets (depression and suicidal ideation) in six languages, plus their machine-translated equivalents. It systematically examines prompting strategies, including chain-of-thought with emotion infusion, and how instruction fine-tuning on open-source models impacts performance, while comparing against traditional baselines. A core contribution is linking translation quality metrics (LaBSE, BERTScore, BLEU) to downstream MT performance, revealing language-typology effects on MT efficacy and LLM results. The findings show that CoT prompting and fine-tuning substantially boost performance and that MT data generally underperforms original data, with the gaps varying by language and typology, offering practical guidance for multilingual mental health deployments and future research toward translation-robust models.

Abstract

Large Language Models (LLMs) have remarkable capabilities across NLP tasks. However, their performance in multilingual contexts, especially within the mental health domain, has not been thoroughly explored. In this paper, we evaluate proprietary and open-source LLMs on eight mental health datasets in various languages, as well as their machine-translated (MT) counterparts. We compare LLM performance in zero-shot, few-shot, and fine-tuned settings against conventional NLP baselines that do not employ LLMs. In addition, we assess translation quality across language families and typologies to understand its influence on LLM performance. Proprietary LLMs and fine-tuned open-source LLMs achieve competitive F1 scores on several datasets, often surpassing state-of-the-art results. However, performance on MT data is generally lower, and the extent of this decline varies by language and typology. This variation highlights both the strengths of LLMs in handling mental health tasks in languages other than English and their limitations when translation quality introduces structural or lexical mismatches.