Examining the Behavior of LLM Architectures Within the Framework of Standardized National Exams in Brazil

TL;DR

This work probes how large language models (GPT-3.5, GPT-4, and MariTalk) behave on Brazil's ENEM standardized exam, using SES-stratified human data and 2022 ENEM items to assess biases. It combines MCQ performance analysis with Jensen-Shannon distance over answer distributions and a detailed essay analysis (syntactic metrics and word usage) to determine how closely AI outputs resemble or diverge from human patterns. The study finds no clear SES bias in MC question responses, with similarity largely driven by human accuracy, while ENEM-style essays reveal systematic linguistic differences between AI and human writing; GPT-4 tends to be closest to human patterns, but overall AI texts remain distinguishable. These results underscore that, for Brazilian Portuguese in ENEM, LLM outputs do not map to specific human groups and highlight important distinctions in linguistic style and reasoning that inform bias assessment and safe deployment in non-English contexts.

Abstract

The Exame Nacional do Ensino Médio (ENEM) is a pivotal test for Brazilian students, required for admission to a significant number of universities in Brazil. The test consists of four objective high-school level tests on Math, Humanities, Natural Sciences and Languages, and one writing essay. Students' answers to the test and to the accompanying socioeconomic status questionnaire are made public every year (albeit anonymized) due to transparency policies from the Brazilian Government. In the context of large language models (LLMs), these data lend themselves nicely to comparing different groups of humans with AI, as we can have access to human and machine answer distributions. We leverage these characteristics of the ENEM dataset and compare GPT-3.5 and 4, and MariTalk, a model trained using Portuguese data, to humans, aiming to ascertain how their answers relate to real societal groups and what that may reveal about the model biases. We divide the human groups by using socioeconomic status (SES), and compare their answer distribution with LLMs for each question and for the essay. We find no significant biases when comparing LLM performance to humans on the multiple-choice Brazilian Portuguese tests, as the distance between model and human answers is mostly determined by the human accuracy. A similar conclusion is found by looking at the generated text as, when analyzing the essays, we observe that human and LLM essays differ in a few key factors, one being the choice of words where model essays were easily separable from human ones. The texts also differ syntactically, with LLM generated essays exhibiting, on average, smaller sentences and less thought units, among other differences. These results suggest that, for Brazilian Portuguese in the ENEM context, LLM outputs represent no group of humans, being significantly different from the answers from Brazilian students across all tests.

Figures (4)

• Figure 1: Distance between human and models' probability distributions by SES level by ENEM test subject. The error bars represent the 95% confidence interval.
• Figure 2: Relationship between human accuracy and the overlap with GPT-3.5 answers. Notice how the relationship is almost linear.
• Figure 3: Relationship between human SES Score and the overlap with GPT-3.5 answers.
• Figure 4: Z-score for syntactic metrics. Notice that the GPT-4 model tends to be closer to humans than other models in most features.