Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Iconicity in Large Language Models

Anna Marklová, Jiří Milička, Leonid Ryvkin, Ľudmila Lacková Bennet, Libuše Kormaníková

TL;DR

This study investigates whether lexical iconicity can be captured by large language models despite their indirect access to meaning and sound. It generates iconic pseudowords for an artificial language using GPT-4 and tests whether Czech and German participants can guess their meanings, comparing results to natural language baselines. The study further probes whether LLMs themselves can guess meanings for these pseudowords, using GPT-4 and Claude-3.5 Sonnet, and analyzes which cues (such as length and phonological similarity) drive intelligibility. The findings show that LLMs not only generate highly iconically tractable pseudowords but also guess their meanings with high accuracy, often outperforming humans, and that cues shared with human processing underlie these results, suggesting that iconicity can be learned and leveraged by AI systems with practical implications for cross-linguistic experimental design.

Abstract

Lexical iconicity, a direct relation between a word's meaning and its form, is an important aspect of every natural language, most commonly manifesting through sound-meaning associations. Since Large language models' (LLMs') access to both meaning and sound of text is only mediated (meaning through textual context, sound through written representation, further complicated by tokenization), we might expect that the encoding of iconicity in LLMs would be either insufficient or significantly different from human processing. This study addresses this hypothesis by having GPT-4 generate highly iconic pseudowords in artificial languages. To verify that these words actually carry iconicity, we had their meanings guessed by Czech and German participants (n=672) and subsequently by LLM-based participants (generated by GPT-4 and Claude 3.5 Sonnet). The results revealed that humans can guess the meanings of pseudowords in the generated iconic language more accurately than words in distant natural languages and that LLM-based participants are even more successful than humans in this task. This core finding is accompanied by several additional analyses concerning the universality of the generated language and the cues that both human and LLM-based participants utilize.

Iconicity in Large Language Models

TL;DR

This study investigates whether lexical iconicity can be captured by large language models despite their indirect access to meaning and sound. It generates iconic pseudowords for an artificial language using GPT-4 and tests whether Czech and German participants can guess their meanings, comparing results to natural language baselines. The study further probes whether LLMs themselves can guess meanings for these pseudowords, using GPT-4 and Claude-3.5 Sonnet, and analyzes which cues (such as length and phonological similarity) drive intelligibility. The findings show that LLMs not only generate highly iconically tractable pseudowords but also guess their meanings with high accuracy, often outperforming humans, and that cues shared with human processing underlie these results, suggesting that iconicity can be learned and leveraged by AI systems with practical implications for cross-linguistic experimental design.

Abstract

Lexical iconicity, a direct relation between a word's meaning and its form, is an important aspect of every natural language, most commonly manifesting through sound-meaning associations. Since Large language models' (LLMs') access to both meaning and sound of text is only mediated (meaning through textual context, sound through written representation, further complicated by tokenization), we might expect that the encoding of iconicity in LLMs would be either insufficient or significantly different from human processing. This study addresses this hypothesis by having GPT-4 generate highly iconic pseudowords in artificial languages. To verify that these words actually carry iconicity, we had their meanings guessed by Czech and German participants (n=672) and subsequently by LLM-based participants (generated by GPT-4 and Claude 3.5 Sonnet). The results revealed that humans can guess the meanings of pseudowords in the generated iconic language more accurately than words in distant natural languages and that LLM-based participants are even more successful than humans in this task. This core finding is accompanied by several additional analyses concerning the universality of the generated language and the cues that both human and LLM-based participants utilize.
Paper Structure (23 sections, 9 figures, 4 tables)

This paper contains 23 sections, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivation and aims
  3. Methodology
  4. Materials
  5. Generating pseudowords
  6. Creating the word lists
  7. Participants
  8. Human participants
  9. LLM participants
  10. Procedure
  11. Human participants
  12. LLM-based participants
  13. Results and discussion
  14. Correctness overview
  15. Logistic regression
  16. ...and 8 more sections

Figures (9)

  • Figure 1: An example of an experiment trial. The upper pair of words is in Czech (the first word means day, the second one means mountain), and the lower pair are matching Czech-prompted pseudowords.
  • Figure 2: The distribution of the correctness of the answers.
  • Figure 3: The comparison of the correctness of the answers across all datasets.
  • Figure 4: Model of correctness rate (human participants).
  • Figure 5: Model of correctness rate (large language model based participants).
  • ...and 4 more figures