In-Context Analogical Reasoning with Pre-Trained Language Models

TL;DR

This work asks whether large pre-trained language models can perform zero-shot analogical reasoning on Raven's Progressive Matrices by recoding perceptual features as language. It introduces entity-level, layout-level, and structural decomposition abstractions to translate RPM problems into text prompts and shows that PLMs achieve strong relational reasoning, approaching supervised vision-based methods and, in some cases, human performance. The results indicate that model size, in-context learning, and task decompositions critically influence performance, with higher-level abstractions enabling substantial gains, even for small models. The study highlights PLMs’ potential as components of cognitive architectures for analogy and raises questions about prior knowledge and the objectivity of RPM-type tasks when presented as text prompts.

Abstract

Analogical reasoning is a fundamental capacity of human cognition that allows us to reason abstractly about novel situations by relating them to past experiences. While it is thought to be essential for robust reasoning in AI systems, conventional approaches require significant training and/or hard-coding of domain knowledge to be applied to benchmark tasks. Inspired by cognitive science research that has found connections between human language and analogy-making, we explore the use of intuitive language-based abstractions to support analogy in AI systems. Specifically, we apply large pre-trained language models (PLMs) to visual Raven's Progressive Matrices (RPM), a common relational reasoning test. By simply encoding the perceptual features of the problem into language form, we find that PLMs exhibit a striking capacity for zero-shot relational reasoning, exceeding human performance and nearing supervised vision-based methods. We explore different encodings that vary the level of abstraction over task features, finding that higher-level abstractions further strengthen PLMs' analogical reasoning. Our detailed analysis reveals insights on the role of model complexity, in-context learning, and prior knowledge in solving RPM tasks.

Figures (14)

• Figure 1: Raven's Progressive Matrices raven1938ravenzhang2019raven are an analogy-making task where one must infer the missing matrix item based on abstract rules instantiated in the first two rows. To demonstrate the potential analogical reasoning skills in pre-trained language models, we develop language-based abstractions over their key perceptual features, then prompt them to select the completion of the matrix.
• Figure 2: Illustration of the compositional nature of entities, layouts, and component structures in RAVEN, and their unique attributes. We provide example items from sub-tasks each item type appears in.
• Figure 3: Example generated prompts for a complete RPM under entity attribute naming (left) and decomposition (right) abstractions in the Center sub-task.
• Figure 4: Example of generated entity layout encodings when abstracting position and number, and summarizing redundant entity attributes within the layout.
• Figure 5: Quasi-image abstractions for a triangle and pentagon of different size and color.