Learning Visual Abstract Reasoning through Dual-Stream Networks

TL;DR

The paper tackles non-verbal visual abstract reasoning in Raven's Progressive Matrices (RPM) benchmarks by proposing DRNet, a dual-stream network that simulates ventral and dorsal visual streams using a CNN and a ViT as parallel encoders. A learnable LIN fusion combines the two high-level representations, which are then fed to a rule extractor to produce eight abstract rule embeddings subsequently scored by an MLP to select the correct candidate. DRNet reports state-of-the-art average performance across PGM, RAVEN, I-RAVEN, and RAVEN-FAIR and demonstrates strong out-of-distribution generalization, with ablations validating the necessity and complementary roles of the two streams. The work provides insight into how separate local and spatial representations can yield clearer rule representations and robust reasoning, with potential extensions to hippocampal-like processing and multimodal inputs for broader AI capabilities.

Abstract

Visual abstract reasoning tasks present challenges for deep neural networks, exposing limitations in their capabilities. In this work, we present a neural network model that addresses the challenges posed by Raven's Progressive Matrices (RPM). Inspired by the two-stream hypothesis of visual processing, we introduce the Dual-stream Reasoning Network (DRNet), which utilizes two parallel branches to capture image features. On top of the two streams, a reasoning module first learns to merge the high-level features of the same image. Then, it employs a rule extractor to handle combinations involving the eight context images and each candidate image, extracting discrete abstract rules and utilizing an multilayer perceptron (MLP) to make predictions. Empirical results demonstrate that the proposed DRNet achieves state-of-the-art average performance across multiple RPM benchmarks. Furthermore, DRNet demonstrates robust generalization capabilities, even extending to various out-of-distribution scenarios. The dual streams within DRNet serve distinct functions by addressing local or spatial information. They are then integrated into the reasoning module, leveraging abstract rules to facilitate the execution of visual reasoning tasks. These findings indicate that the dual-stream architecture could play a crucial role in visual abstract reasoning.

Figures (6)

• Figure 1: Examples from RAVEN and Procedurally Generated Matrices (PGM) are shown in (a) and (b) respectively. Both types of problems involve presenting participants with eight context images. They are required to select the correct answer (highlighted in red) from the candidate set of eight images to fill in the blank (denoted by ?), in order to satisfy specific rules in the row or column direction of the 3$\times$3 matrix.
• Figure 2: An overview of our DRNet. DRNet consists of a dual encoder module and a reasoning module, where (B$\times$16, 1, 80, 80) represents (batchsize$\times$16, channels, image size, image size). The dual encoder module is used to extract input image features in parallel, after which the features are fed into the reasoning module. The reasoning module first learns to merge the high-level embeddings of the same image. Then, it employs a rule extractor to handle combinations involving the eight context images and each candidate image, extracting abstract rules and utilizing an MLP to make predictions.
• Figure 3: The performance analysis of various operators in DRNet on the RAVEN dataset reveals that the $\mathbf{LIN}$ operator outperforms its counterparts.
• Figure 4: T-SNE visualization of abstract rules in PGM Neutral. The clustered embedding of similar abstract rules indicates that the Reasoning Module in DRNet is adept at discovering rules.
• Figure 5: Illustrative examples from ViT and CNN streams. (a) Demonstrative instances of attention mapping from output tokens to the input space. (b) Visualization of the convolutional layer obtained through the forward process of DRNet.