One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems

TL;DR

This work proposes the unified model for solving Single-Choice Abstract visual Reasoning tasks (SCAR), capable of solving various single-choice AVR tasks, without making any a priori assumptions about the task structure, in particular the number and location of panels.

Abstract

Abstract Visual Reasoning (AVR) comprises a wide selection of various problems similar to those used in human IQ tests. Recent years have brought dynamic progress in solving particular AVR tasks, however, in the contemporary literature AVR problems are largely dealt with in isolation, leading to highly specialized task-specific methods. With the aim of developing universal learning systems in the AVR domain, we propose the unified model for solving Single-Choice Abstract visual Reasoning tasks (SCAR), capable of solving various single-choice AVR tasks, without making any a priori assumptions about the task structure, in particular the number and location of panels. The proposed model relies on a novel Structure-Aware dynamic Layer (SAL), which adapts its weights to the structure of the considered AVR problem. Experiments conducted on Raven's Progressive Matrices, Visual Analogy Problems, and Odd One Out problems show that SCAR (SAL-based models, in general) effectively solves diverse AVR tasks, and its performance is on par with the state-of-the-art task-specific baselines. What is more, SCAR demonstrates effective knowledge reuse in multi-task and transfer learning settings. To our knowledge, this work is the first successful attempt to construct a general single-choice AVR solver relying on self-configurable architecture and unified solving method. With this work we aim to stimulate and foster progress on task-independent research paths in the AVR domain, with the long-term goal of development of a general AVR solver.

Figures (8)

• Figure 1: STL vs MTL. In this paper, we consider a diverse set of AVR tasks including RPMs, VAPs, and O3 tests (top). Existing literature deals with AVR tasks in isolation, which leads to the development of task-specific methods with limited applicability to other, even related problems (bottom left). Instead, we propose a single self-configurable model capable of dealing with diverse AVR tasks, thus intrinsically facilitating MTL settings (bottom right).
• Figure 2: Structure-Aware dynamic Layer. SAL enables processing of AVR tasks with diverse structures by adapting its weights to the problem instance. The figure illustrates two separate forward passes through the layer for an RPM (top) and a VAP matrix (bottom). From left, the panels $X_i$ belonging to a single problem instance are processed with an encoder $\mathcal{E}$. The resultant embeddings $\{h_{i,j}\}$ are arranged into $A_t$ groups $\{H_{i,k}\}$. Next, the figure illustrates further processing of a single group $H_{i,1}$ for both matrices. Thanks to SAL's adaptability, the resultant vector $v_{i,1} \in \mathbb{R}^{d_v}$ has the same dimensions irrespectively of the considered task, thus enabling uniform processing in the subsequent model layers.
• Figure 3: SCAR architecture. Each input panel $x_{i,j}$ is embedded separately with the panel encoder $\mathcal{E}$ to a latent representation $h_{i,j}$. The embeddings are arranged into $A_t$ groups $\{ H_{i,k} \}_{k=1}^{A_t}$. The figure depicts processing of a single group $H_{i,k}$ corresponding to answer $k$. Vectors belonging to this group are fused and processed with the reasoning module $\mathcal{G}$. The resultant representation $g_{i,k}$ is used to predict the index of the correct answer and optionally the encoded rules.
• Figure 4: RPMs from I-RAVEN zhang2019ravenhu2021stratified. The $3 \times 3$ grid of context images has to be completed with the appropriate answer panel (A -- H). In I-RAVEN each matrix belongs to one out of seven possible visual configurations. Examples of the three selected configurations (Up-Down, Out-InCenter, Out-InGrid) are presented in the figure. Each problem instance has up to 8 rules, which can be applied separately to each matrix hierarchy. The matrices are governed by the following rules: (a) in each row there is a circle, a pentagon, and a triangle in the upper part of exactly one image, while the lower parts of the images in each row contain the same shape with a gradually darker colour from left to right; (b) in each row there is a circle, a square, and a hexagon as the outer shape with a gradually increasing size from left to right, and a circle, a square, and a pentagon in the same colour as the inner shape, resp.; (c) in each row there is a circle, a triangle, and a hexagon with the same size as the outer shape, and circles, pentagons, and triangles as the inner shapes, resp., whose count in the third column is the sum of the shape counts in the preceding columns. The correct answers are F, C, and H, respectively.
• Figure 5: RPMs from G-set mandziuk2019deepiq. The $3 \times 3$ grid of context images has to be completed with the appropriate answer panel (A -- E). The matrices are governed by the following rules: (a) and (b) progression applied to object rotation; (c) in each row there are the same three object shapes. The correct answers are B, A, and B, respectively.