Performance in a dialectal profiling task of LLMs for varieties of Brazilian Portuguese
Raquel Meister Ko Freitag, Túlio Sousa de Gois
TL;DR
The paper investigates dialectal biases in LLMs by testing four models on a Brazilian Portuguese dialect profiling task. It uses a three-stage prompt-based pipeline to generate state-profile text, classify the state via prompts, and clean/analyze data. Key findings show that GPT-4o, GPT-3.5, and Gemini exhibit dialectal sensitivity, while Sabiá-2 lacks; human judges show low agreement; explicit linguistic clues in prompts do not reliably improve performance. The work highlights gaps in linguistic justice in NLP systems and suggests careful design of training data and evaluation for equitable multilingual AI.
Abstract
Different of biases are reproduced in LLM-generated responses, including dialectal biases. A study based on prompt engineering was carried out to uncover how LLMs discriminate varieties of Brazilian Portuguese, specifically if sociolinguistic rules are taken into account in four LLMs: GPT 3.5, GPT-4o, Gemini, and Sabi.-2. The results offer sociolinguistic contributions for an equity fluent NLP technology.