MlingConf: A Comprehensive Study of Multilingual Confidence Estimation on Large Language Models

TL;DR

This work introduces MlingConf, a multilingual benchmark to study confidence estimation for large language models across language-agnostic and language-specific tasks. It builds a high-quality multilingual dataset spanning English, Japanese, Chinese, French, and Thai, with LA tasks translated from four English QA datasets and LSQA subsets tailored to linguistic contexts. Key findings show English dominance in LA confidence estimation, while language-specific prompts tied to the query context boost LS-task reliability, motivating a Native-Tone Prompting strategy. The results highlight the importance of language-aware prompting and calibration in multilingual LLM deployments and establish a foundation for expanding multilingual confidence estimation research.

Abstract

The tendency of Large Language Models (LLMs) to generate hallucinations raises concerns regarding their reliability. Therefore, confidence estimations indicating the extent of trustworthiness of the generations become essential. However, current LLM confidence estimations in languages other than English remain underexplored. This paper addresses this gap by introducing a comprehensive investigation of Multilingual Confidence estimation (MlingConf) on LLMs, focusing on both language-agnostic (LA) and language-specific (LS) tasks to explore the performance and language dominance effects of multilingual confidence estimations on different tasks. The benchmark comprises four meticulously checked and human-evaluated high-quality multilingual datasets for LA tasks and one for the LS task tailored to specific social, cultural, and geographical contexts of a language. Our experiments reveal that on LA tasks English exhibits notable linguistic dominance in confidence estimations than other languages, while on LS tasks, using question-related language to prompt LLMs demonstrates better linguistic dominance in multilingual confidence estimations. The phenomena inspire a simple yet effective native-tone prompting strategy by employing language-specific prompts for LS tasks, effectively improving LLMs' reliability and accuracy in LS scenarios.

Figures (15)

• Figure 1: Examples of generations and confidence scores of Llama-3.1 given the same inputs in three languages in LA and LS scenarios derived from SciQ and LSQA datasets respectively.
• Figure 2: Three stages of MlingConf dataset construction.
• Figure 3: Experimental results of Accuracy on four LA datasets on GPT-3.5 and Llama-3.1.
• Figure 4: Experimental results of AUROC and ECE of three confidence estimation methods on five language-specific subset of LSQA using Llama-3.1.
• Figure 5: Experimental results of Accuracy on five language-specific LSQA subsets using Llama-3.1.