Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation

Andrea Piergentili, Beatrice Savoldi, Matteo Negri, Luisa Bentivogli

TL;DR

The paper addresses the challenge of automatically evaluating gender-neutral translation (GNT) by using large language models (LLMs) as evaluators. It proposes an LLM-as-a-judge framework with four prompting schemes, including a two-step phrase-level annotation before a final sentence-level judgment, and evaluates them across Italian, Spanish, and German using mGeNTE and automatic GNTs. Results show that LLMs can effectively evaluate GNT, with phrase-level annotation prompts (especially Cross-P+L) yielding the best accuracy across languages and data conditions, without language-specific fine-tuning data. The approach offers a scalable, cross-language evaluation method that can supplement or replace dedicated classifiers, though limitations remain in granularity, acceptability, and cross-lingual source usage.

Abstract

Gender-neutral translation (GNT) aims to avoid expressing the gender of human referents when the source text lacks explicit cues about the gender of those referents. Evaluating GNT automatically is particularly challenging, with current solutions being limited to monolingual classifiers. Such solutions are not ideal because they do not factor in the source sentence and require dedicated data and fine-tuning to scale to new languages. In this work, we address such limitations by investigating the use of large language models (LLMs) as evaluators of GNT. Specifically, we explore two prompting approaches: one in which LLMs generate sentence-level assessments only, and another, akin to a chain-of-thought approach, where they first produce detailed phrase-level annotations before a sentence-level judgment. Through extensive experiments on multiple languages with five models, both open and proprietary, we show that LLMs can serve as evaluators of GNT. Moreover, we find that prompting for phrase-level annotations before sentence-level assessments consistently improves the accuracy of all models, providing a better and more scalable alternative to current solutions.

An LLM-as-a-judge Approach for Scalable Gender-Neutral Translation Evaluation

TL;DR

The paper addresses the challenge of automatically evaluating gender-neutral translation (GNT) by using large language models (LLMs) as evaluators. It proposes an LLM-as-a-judge framework with four prompting schemes, including a two-step phrase-level annotation before a final sentence-level judgment, and evaluates them across Italian, Spanish, and German using mGeNTE and automatic GNTs. Results show that LLMs can effectively evaluate GNT, with phrase-level annotation prompts (especially Cross-P+L) yielding the best accuracy across languages and data conditions, without language-specific fine-tuning data. The approach offers a scalable, cross-language evaluation method that can supplement or replace dedicated classifiers, though limitations remain in granularity, acceptability, and cross-lingual source usage.

Abstract

Gender-neutral translation (GNT) aims to avoid expressing the gender of human referents when the source text lacks explicit cues about the gender of those referents. Evaluating GNT automatically is particularly challenging, with current solutions being limited to monolingual classifiers. Such solutions are not ideal because they do not factor in the source sentence and require dedicated data and fine-tuning to scale to new languages. In this work, we address such limitations by investigating the use of large language models (LLMs) as evaluators of GNT. Specifically, we explore two prompting approaches: one in which LLMs generate sentence-level assessments only, and another, akin to a chain-of-thought approach, where they first produce detailed phrase-level annotations before a sentence-level judgment. Through extensive experiments on multiple languages with five models, both open and proprietary, we show that LLMs can serve as evaluators of GNT. Moreover, we find that prompting for phrase-level annotations before sentence-level assessments consistently improves the accuracy of all models, providing a better and more scalable alternative to current solutions.

Paper Structure

This paper contains 26 sections, 3 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Gender-inclusivity in language technology
  4. GNT evaluation
  5. LLM-as-a-Judge
  6. GNT evaluation prompts
  7. ❍ Mono-L
  8. ● Mono-P+L
  9. $\lozenge$ Cross-L
  10. $\blacklozenge$ Cross-P+L
  11. Experimental settings
  12. Test data and evaluation metrics
  13. mGeNTE references
  14. Automatic GNTs
  15. Prompting details
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Accuracy of all models in target-only GNT evaluation experiments on mGeNTE references. The Italian experiments include the performance of the gender-neutrality classifier, which is not available for other languages.
  • Figure 2: Precision and recall scores of all models in target-only GNT evaluation of automatic GNTs.
  • Figure 3: Accuracy of all models in source-target GNT evaluation experiments on mGeNTE source-reference pairs. Note that the axes here encompass a wider range of values compared to the target-only chart.