Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks

TL;DR

Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks investigates prompt grammaticality's impact on cross-lingual factual knowledge retrieval. The authors compare templated MLAMA prompts, Google Translate sentence translations, and ChatGPT sentence translations across 9 languages using a Llama-2-7b model with fixed distractors to assess $R@1$/$R@n$ performance. They show sentence-level translations yield substantial gains—up to 10% in Slavic and up to 24% in non-Slavic languages—with improvements driven by inflection accuracy and explicitation. They release a curated multilingual dataset and a reusable evaluation pipeline, advocating fluent translations for more interpretable cross-lingual knowledge benchmarking.

Abstract

For multilingual factual knowledge assessment of LLMs, benchmarks such as MLAMA use template translations that do not take into account the grammatical and semantic information of the named entities inserted in the sentence. This leads to numerous instances of ungrammaticality or wrong wording of the final prompts, which complicates the interpretation of scores, especially for languages that have a rich morphological inventory. In this work, we sample 4 Slavic languages from the MLAMA dataset and compare the knowledge retrieval scores between the initial (templated) MLAMA dataset and its sentence-level translations made by Google Translate and ChatGPT. We observe a significant increase in knowledge retrieval scores, and provide a qualitative analysis for possible reasons behind it. We also make an additional analysis of 5 more languages from different families and see similar patterns. Therefore, we encourage the community to control the grammaticality of highly multilingual datasets for higher and more interpretable results, which is well approximated by whole sentence translation with neural MT or LLM systems. The dataset and all related code is published at the Github repository: https://github.com/ZurichNLP/Fluent-mLAMA.

Figures (4)

• Figure 1: Main idea of our experiment. Factual dataset MLAMA uses the templated sentences in each language to prompt LLMs for factual knowledge. This results in many ungrammatical prompts: for Sofya Kovalevskaya, who was the first woman to earn a doctorate in mathematics, the verb "was born" had to be inflected in feminine gender, not in the templated masculine. This leads to poor factual extraction. Sentence-level translation leads to grammatically and lexically coherent sentences and increases factual retrieval of objects (in the correct grammatical forms, in this case, locative case of the word "Stockholm"). The incorrect entities are marked red, the correct but ungrammatical words are in yellow, the correct and grammatical words are in green.
• Figure 2: Correlation between increase in QE scores (x-axis) and increase in $R@1$ performance (y-axis) of the GT and ChatGPT verbalizations compared to templates. The left subplot shows statistics for the first experiment (Slavic languages), the right - for the second experiment (non-Slavic languages). Each point represents the averaged value over a particular language, relation and verbalization type. The color represents the verbalization type; the shape of the point stands for the language. The dotted lines show linear regressions fit by all data points at each subgraph for a corresponding type of verbalization.
• Figure 3: Comparison of $R@n$ scores depending on $n$. Higher score shows better retrieval.
• Figure 4: Distribution of the highest correct ranks for each verbalization in each language; lower score is better. For clarity, the box plots do not show outliers, which explain the skew of the average ranks (shown as green triangles) higher than the third quartiles.