With Ears to See and Eyes to Hear: Sound Symbolism Experiments with Multimodal Large Language Models

TL;DR

This work analyses the ability of VLMs and LLMs to demonstrate sound symbolism as well as their ability to “hear” via the interplay of the language and vision modules of open and closed-source multimodal models.

Abstract

Recently, Large Language Models (LLMs) and Vision Language Models (VLMs) have demonstrated aptitude as potential substitutes for human participants in experiments testing psycholinguistic phenomena. However, an understudied question is to what extent models that only have access to vision and text modalities are able to implicitly understand sound-based phenomena via abstract reasoning from orthography and imagery alone. To investigate this, we analyse the ability of VLMs and LLMs to demonstrate sound symbolism (i.e., to recognise a non-arbitrary link between sounds and concepts) as well as their ability to "hear" via the interplay of the language and vision modules of open and closed-source multimodal models. We perform multiple experiments, including replicating the classic Kiki-Bouba and Mil-Mal shape and magnitude symbolism tasks, and comparing human judgements of linguistic iconicity with that of LLMs. Our results show that VLMs demonstrate varying levels of agreement with human labels, and more task information may be required for VLMs versus their human counterparts for in silico experimentation. We additionally see through higher maximum agreement levels that Magnitude Symbolism is an easier pattern for VLMs to identify than Shape Symbolism, and that an understanding of linguistic iconicity is highly dependent on model size.

Figures (10)

• Figure 1: Illustration of the 3 main experiments we perform. Firstly, Shape Symbolism is a binary choice between two pseudowords to best describe an object that is spiky or rounded. Magnitude Symbolism involves a binary choice between two pseudowords to best describe an object that is small or large. Finally, Iconicity involves rating the perceived iconicity of words, or how much their written/phonetic form is representative of what they describe.
• Figure 2: Examples of "Kiki"-style (spiky) and "Bouba"-style (rounded) generations with DALL-E 3. In total, 50 images were generated, with 25 per condition (the entities remaining constant). The ground truth is taken as the majority human vote.
• Figure 3: Results of the Shape Symbolism experiments per pseudoword pair. Fleiss' $\kappa$fleiss_kappa for inter-annotator agreement between humans is presented next to each pseudoword pair. Arrows indicate the direction of agreement change from the standard prompt to the informed prompt. The dashed line represents 50%, akin to chance-level agreement. Full results in table form are presented in \ref{['tab:kiki_table']} within Appendix \ref{['apx:full_results']}. In all cases, we are comparing with the human majority vote.
• Figure 4: Examples of "Mil"-style (tiny) and "Mal"-style (huge) generations with DALL-E 3. In total, 50 images were generated, with 25 per condition (the entities remaining constant). The ground truth is taken as the majority human vote.
• Figure 5: Results of the Magnitude Symbolism experiments per pseudoword pair. Fleiss' $\kappa$fleiss_kappa for inter-annotator agreement between humans is presented next to each pseudoword pair. Arrows indicate the direction of agreement change from the standard prompt to the informed prompt. The dashed line represents 50%, akin to chance-level agreement. Full results in table form are presented in \ref{['tab:mil_table']} within Appendix \ref{['apx:full_results']}. In all cases, we are comparing with the human majority vote.