Compositional Attention Networks for Machine Reasoning

TL;DR

The paper introduces the Memory, Attention and Composition (MAC) network for explicit, multi-step reasoning in visual question answering. Each MAC cell maintains separate control and memory states to perform one reasoning operation, with three units (control, read, write) that attend over the question and the image-based knowledge base, enabling iterative, transparent inference. The network achieves state-of-the-art results on CLEVR (e.g., ${\text{accuracy}}=98.94\%$) and demonstrates strong data efficiency, learning effectively from an order of magnitude less data and offering interpretable attention maps that reveal its reasoning path. The approach generalizes to human-collected questions (CLEVR-Humans) and shows robustness through ablations, highlighting the value of explicit structural priors for compositional reasoning and potential applicability to other multi-step inference tasks.

Abstract

We present the MAC network, a novel fully differentiable neural network architecture, designed to facilitate explicit and expressive reasoning. MAC moves away from monolithic black-box neural architectures towards a design that encourages both transparency and versatility. The model approaches problems by decomposing them into a series of attention-based reasoning steps, each performed by a novel recurrent Memory, Attention, and Composition (MAC) cell that maintains a separation between control and memory. By stringing the cells together and imposing structural constraints that regulate their interaction, MAC effectively learns to perform iterative reasoning processes that are directly inferred from the data in an end-to-end approach. We demonstrate the model's strength, robustness and interpretability on the challenging CLEVR dataset for visual reasoning, achieving a new state-of-the-art 98.9% accuracy, halving the error rate of the previous best model. More importantly, we show that the model is computationally-efficient and data-efficient, in particular requiring 5x less data than existing models to achieve strong results.

Figures (18)

• Figure 1: A CLEVR example. Color added for illustration.
• Figure 2: Model Overview. The MAC network consists of an input unit, a core recurrent network and an output unit. (1) The input unit transforms the raw image and question into distributed vector representations. (2) The core recurrent network reasons sequentially over the question by decomposing it into a series of operations (control) that retrieve information from the image (knowledge base) and aggregate the results into a recurrent memory. (3) The output classifier computes the final answer using the question and the final memory state.
• Figure 3: The MAC cell architecture. The MAC recurrent cell consists of a control unit, read unit, and write unit, that operate over dual control and memory hidden states. The control unit successively attends to different parts of the task description (question), updating the control state to represent at each timestep the reasoning operation the cell intends to perform. The read unit extracts information out of a knowledge base (here, image), guided by the control state. The write unit integrates the retrieved information into the memory state, yielding the new intermediate result that follows from applying the current reasoning operation.
• Figure 4: The Control Unit (CU) architecture. The control unit attends at each iteration to some part of the question, by applying soft attention over the question words, and updates the control state accordingly. The unit's inputs and outputs are in bold. See section \ref{['sec:CU']} for details.
• Figure 5: The Read Unit (RU) architecture. The read unit retrieves information from the knowledge base that is necessary for performing the current reasoning operation (control) and potentially related to previously obtained intermediate results (memory). It extracts the information by performing a two-stage attention process over the knowledge base elements. See section \ref{['sec:RU']} for details.