On Psychology of AI -- Does Primacy Effect Affect ChatGPT and Other LLMs?

TL;DR

The paper investigates whether the human primacy effect appears in LLMs by adapting Asch's two-candidate task to 200 adjective-pair descriptions across three commercial models. It uses two prompting setups to probe order effects on candidate preference, revealing model-specific biases: ChatGPT shows a primacy-like tilt toward positive-first in the simultaneous task, while Claude often abstains and Gemini remains near-balanced. In the individual-evaluation task, ratings are mostly equal, but when unequal, all models tend to favor the negatively described candidate, with Gemini showing the strongest bias, suggesting a recency-like or negative-first effect under certain conditions. The work highlights significant safety and ethical considerations for AI-assisted decision-making in domains like hiring and resource allocation and advocates for robust, transparent evaluation and governance frameworks.

Abstract

We study the primacy effect in three commercial LLMs: ChatGPT, Gemini and Claude. We do this by repurposing the famous experiment Asch (1946) conducted using human subjects. The experiment is simple, given two candidates with equal descriptions which one is preferred if one description has positive adjectives first before negative ones and another description has negative adjectives followed by positive ones. We test this in two experiments. In one experiment, LLMs are given both candidates simultaneously in the same prompt, and in another experiment, LLMs are given both candidates separately. We test all the models with 200 candidate pairs. We found that, in the first experiment, ChatGPT preferred the candidate with positive adjectives listed first, while Gemini preferred both equally often. Claude refused to make a choice. In the second experiment, ChatGPT and Claude were most likely to rank both candidates equally. In the case where they did not give an equal rating, both showed a clear preference to a candidate that had negative adjectives listed first. Gemini was most likely to prefer a candidate with negative adjectives listed first.