Multi-Label Contrastive Learning for Abstract Visual Reasoning

TL;DR

The paper addresses abstract visual reasoning in RPM tasks, highlighting how humans identify underlying rules versus how DL models often overfit to patterns or dataset biases. It introduces Multi-Label Contrastive Learning (MLCL), casting RPM solving as a multi-label classification over abstract rules and extending Noise Contrastive Estimation to multi-label samples via the loss $L^{\mathrm{mlc}}$, with a temperature parameter $\tau$. A sparse rule-encoding scheme and an auxiliary rule-discovery pathway are integrated, combining contrastive and auxiliary training and enabling effective linear evaluation after pre-training. Experiments on Balanced-RAVEN and PGM demonstrate state-of-the-art performance on Balanced-RAVEN, with robust generalization and improved interpretability through explicit rule signaling, while maintaining efficiency without requiring large batch sizes or heavy augmentation.

Abstract

For a long time the ability to solve abstract reasoning tasks was considered one of the hallmarks of human intelligence. Recent advances in application of deep learning (DL) methods led, as in many other domains, to surpassing human abstract reasoning performance, specifically in the most popular type of such problems - the Raven's Progressive Matrices (RPMs). While the efficacy of DL systems is indeed impressive, the way they approach the RPMs is very different from that of humans. State-of-the-art systems solving RPMs rely on massive pattern-based training and sometimes on exploiting biases in the dataset, whereas humans concentrate on identification of the rules / concepts underlying the RPM (or generally a visual reasoning task) to be solved. Motivated by this cognitive difference, this work aims at combining DL with human way of solving RPMs and getting the best of both worlds. Specifically, we cast the problem of solving RPMs into multi-label classification framework where each RPM is viewed as a multi-label data point, with labels determined by the set of abstract rules underlying the RPM. For efficient training of the system we introduce a generalisation of the Noise Contrastive Estimation algorithm to the case of multi-label samples. Furthermore, we propose a new sparse rule encoding scheme for RPMs which, besides the new training algorithm, is the key factor contributing to the state-of-the-art performance. The proposed approach is evaluated on two most popular benchmark datasets (Balanced-RAVEN and PGM) and on both of them demonstrates an advantage over the current state-of-the-art results. Contrary to applications of contrastive learning methods reported in other domains, the state-of-the-art performance reported in the paper is achieved with no need for large batch sizes or strong data augmentation.

Figures (14)

• Figure 1: Solving RPMs requires to identify abstract relationships hidden behind random visual distractors and contrast possible answers to select the one which fits best. Left RPM, although has more visual details, is governed by only a single rule (row-wise AND applied to shape position), whereas perceptually simpler right RPM contains 8 distinct relations applied to both outer and inner structures. The examples come from the PGM and Balanced-RAVEN datasets, respectively and in both cases the correct answer, which is to be placed in the bottom-right panel, is A.
• Figure 2: Augmentation of RPMs from the Balanced-RAVEN dataset. Selected transformation is applied in the same way to all images in a given RPM. The data augmentation module applies a randomly selected combination of presented methods with additional random rotation and transposition, using implementation from buslaev2020albumentations. For clarity, we only depict different views of a single row for two RPMs belonging to configurations 2x2Grid (left part) and 3x3Grid (right part), respectively.
• Figure 3: In all training setups, we start by filling in the context panels with each choice panel. In each case, we generate an embedding with an abstract reasoning encoder network (SCL, HriNet or CoPINet). The embeddings are used as an input to 1) a scoring module which predicts the answer (supervised training, denoted as CE), 2) a rule prediction module which predicts rules as encoded meta-target (auxiliary training, denoted as AUX) and 3) a projection network which maps them into a lower-dimensional representation suitable for computing contrastive loss. The proposed approach combines both auxiliary and contrastive training into a joint learning framework -- MLCL.
• Figure A.1: RPM example from the PGM dataset presented in the main paper. The matrix is governed by a single rule -- AND applied row-wise to octagon shape position. The correct answer is A.
• Figure B.1: Additional ablation studies on the Balanced-RAVEN dataset. The figures present variations in the final classification performance averaged across 4 random seeds depending on: 1) a batch size used for the contrastive pre-training stage of MLCL a) without and b) with data augmentation; 2) balancing factors in the definition of Multi-Label Contrastive Loss c) without and d) with data augmentation.