Form and Meaning in Intrinsic Multilingual Evaluations

TL;DR

This work interrogates intrinsic evaluation metrics for multilingual conditional language models, arguing that such metrics measure information content rather than semantic meaning, and may not be comparable across languages or paraphrase variants. It formalizes the problem, analyzes six intrinsic metrics on mono- and multi-parallel data, and tests paraphrase consistency using cross-language datasets like FLORES-200 and EN–DE paraphrase pairs. The study finds substantial inconsistency in metric readings across paraphrases and languages, even with multi-parallel corpora, highlighting limitations of current intrinsic metrics for cross-language evaluation. The results underscore the need for cautious interpretation of intrinsic scores and motivate exploring alternative or adjusted evaluation paradigms that better capture semantics and cross-language information dynamics in multilingual CLMs.

Abstract

Intrinsic evaluation metrics for conditional language models, such as perplexity or bits-per-character, are widely used in both mono- and multilingual settings. These metrics are rather straightforward to use and compare in monolingual setups, but rest on a number of assumptions in multilingual setups. One such assumption is that comparing the perplexity of CLMs on parallel sentences is indicative of their quality since the information content (here understood as the semantic meaning) is the same. However, the metrics are inherently measuring information content in the information-theoretic sense. We make this and other such assumptions explicit and discuss their implications. We perform experiments with six metrics on two multi-parallel corpora both with mono- and multilingual models. Ultimately, we find that current metrics are not universally comparable. We look at the form-meaning debate to provide some explanation for this.

Figures (8)

• Figure 1: Each line represents a row in a parallel dataset, each dot is an individual sentence; four German and one English. Common multilingual evaluations center around comparing intrinsic metrics such as the negative log-likelihood (NLL) or bits per character (BPC) with the assumption that these comparisons are fair since the semantic meaning is consistent. However, these metrics measure information content in the information-theoretic sense. This results in (1) differences within a language ($\text{DE}_i\leftrightarrow\text{DE}_k$), and (2) inconsistency across languages: if the EN sentence falls outside the range of the DE sentences it is consistent. If it falls within the range it is inconsistent, meaning conclusions can flip depending on the DE sentence we choose in our test set.
• Figure 2: Metrics across checkpoints during training when evaluated on FLORES-200. The different lengths of the lines is due to tokenization differences between languages, resulting in shorter or longer sequences. The models have seen the same amount of parallel data when measured in number of lines.
• Figure 3: Paraphrase consistency of NLL values for the EN source and four parallel DE paraphrases using monolingual EuroParl models. On the x-axis, we list the sample index. Each sample consists of the EN sentence (green) and the four DE paraphrases (light red). For the sake of visual clarity, we sort the results by the English NLL. We show the ranking consistency: if all DE paraphrases are above or below the EN source, it means the ranking is consistent. Inconsistent rankings are marked with a red dot inside the green diamond for English. We see similar results for the multilingual model, as well as for the other metrics, see §\ref{['app:sensitivity-results']}.
• Figure 4: Metrics across checkpoints evaluated using FLORES-200 for the multilingual UNPC model.
• Figure 5: Metrics across checkpoints evaluated using FLORES-200 for the multilingual EP model.