Animate, or Inanimate, That is the Question for Large Language Models

TL;DR

The paper investigates whether large language models (LLMs) process animacy in language similarly to humans, despite being trained solely on text. It adopts a prompting-based psycholinguistic framework, employing typical animacy tasks from BLiMP and BSP as well as atypical animacy paradigms that mimic repetition and contextual adaptation related to N400 studies. Across multiple model families, the study shows that LLMs exhibit human-like preferences for animacy-constrained constructions in typical tasks and adapt to unconventional, animated interpretations in atypical contexts, with OpenAI models often aligning most closely with human baselines. These findings imply that prompting strategies can elicit surprisingly human-like socio-cognitive processing in LLMs, informing both evaluation methods and practical prompt design for more nuanced language understanding.

Abstract

The cognitive essence of humans is deeply intertwined with the concept of animacy, which plays an essential role in shaping their memory, vision, and multi-layered language understanding. Although animacy appears in language via nuanced constraints on verbs and adjectives, it is also learned and refined through extralinguistic information. Similarly, we assume that the LLMs' limited abilities to understand natural language when processing animacy are motivated by the fact that these models are trained exclusively on text. Hence, the question this paper aims to answer arises: can LLMs, in their digital wisdom, process animacy in a similar way to what humans would do? We then propose a systematic analysis via prompting approaches. In particular, we probe different LLMs by prompting them using animate, inanimate, usual, and stranger contexts. Results reveal that, although LLMs have been trained predominantly on textual data, they exhibit human-like behavior when faced with typical animate and inanimate entities in alignment with earlier studies. Hence, LLMs can adapt to understand unconventional situations by recognizing oddities as animated without needing to interface with unspoken cognitive triggers humans rely on to break down animations.

Figures (5)

• Figure 1: Large Language Models seem to be as surprised as humans at the thought of experiencing inanimate entities in animate contexts.
• Figure 2: Large Language Models performances on animate- transitive and passive sub-tasks of BLiMP benchmark warstadt-etal-2020-blimp-benchmark.
• Figure 3: Large Language Models performances on the BSP benchmark VEGAMENDOZA2021107724.
• Figure 4: Average surprise values provided by the LLMs at input-prompt $T_1$, $T_3$ and finally $T_5$.
• Figure 5: Average surprise values.