Learning the meanings of function words from grounded language using a visual question answering model

TL;DR

This study uses a visually grounded visual-question-answering framework (CLEVR) and the MAC reasoning model to probe how function words like and/or, behind/in front of, and more/fewer can be learned non-symbolically. Across three experiments, the authors demonstrate gradient, context-sensitive interpretations, the emergence of alternative-reasoning effects, and the influence of training frequency on learning order, providing proof-of-concept support for usage-based accounts of word meanings. They show that models can acquire nuanced logical, spatial, and numerical semantics without explicit linguistic prior knowledge, though dataset structure and pragmatic alternatives shape learning trajectories. The findings suggest visually grounded, non-symbolic learning mechanisms can underlie the acquisition of abstract function words, with implications for theories of child language development and the design of AI that reasons about language in grounded contexts.

Abstract

Interpreting a seemingly-simple function word like "or", "behind", or "more" can require logical, numerical, and relational reasoning. How are such words learned by children? Prior acquisition theories have often relied on positing a foundation of innate knowledge. Yet recent neural-network based visual question answering models apparently can learn to use function words as part of answering questions about complex visual scenes. In this paper, we study what these models learn about function words, in the hope of better understanding how the meanings of these words can be learnt by both models and children. We show that recurrent models trained on visually grounded language learn gradient semantics for function words requiring spatial and numerical reasoning. Furthermore, we find that these models can learn the meanings of logical connectives and and or without any prior knowledge of logical reasoning, as well as early evidence that they are sensitive to alternative expressions when interpreting language. Finally, we show that word learning difficulty is dependent on frequency in models' input. Our findings offer proof-of-concept evidence that it is possible to learn the nuanced interpretations of function words in visually grounded context by using non-symbolic general statistical learning algorithms, without any prior knowledge of linguistic meaning.

Figures (32)

• Figure 1: Example images and corresponding questions taken from CLEVR dataset.
• Figure 2: Threshold based interpretation of behind and in front relative to gray cube in CLEVR dataset. (Image from Johnson et al. 2017).
• Figure 3: Example image-question pairs from semantic probes.
• Figure 4: Example probe creation procedure. Given a template, we cycle through every possible variable combination and then sample 10 images and determine their corresponding answers.
• Figure 5: The MAC model initially processes the image and question through CNN and biLSTM units respectively and then through four recurrent MAC cells, each generating output memory and control states. The final output unit takes the final memory state and the question representation to produce a prediction. The model is fully differentiable.