Learning by Abstraction: The Neural State Machine

TL;DR

The paper tackles visual reasoning by unifying neural and symbolic approaches through the Neural State Machine (NSM). NSM first learns a probabilistic scene graph from images and then reasoned over it as a differentiable state machine guided by a semantically grounded sequence of instructions derived from the question. It achieves state-of-the-art single-model results on GQA and VQA-CP, and demonstrates strong generalization to unseen concept compositions and linguistic structures. By representing both modalities in a shared concept space and performing sequential graph traversal, NSM showcases the benefits of abstraction and modularity for robust, interpretable reasoning in vision-language tasks.

Abstract

We introduce the Neural State Machine, seeking to bridge the gap between the neural and symbolic views of AI and integrate their complementary strengths for the task of visual reasoning. Given an image, we first predict a probabilistic graph that represents its underlying semantics and serves as a structured world model. Then, we perform sequential reasoning over the graph, iteratively traversing its nodes to answer a given question or draw a new inference. In contrast to most neural architectures that are designed to closely interact with the raw sensory data, our model operates instead in an abstract latent space, by transforming both the visual and linguistic modalities into semantic concept-based representations, thereby achieving enhanced transparency and modularity. We evaluate our model on VQA-CP and GQA, two recent VQA datasets that involve compositionality, multi-step inference and diverse reasoning skills, achieving state-of-the-art results in both cases. We provide further experiments that illustrate the model's strong generalization capacity across multiple dimensions, including novel compositions of concepts, changes in the answer distribution, and unseen linguistic structures, demonstrating the qualities and efficacy of our approach.

Figures (8)

• Figure 1: The Neural State Machine is a graph network that simulates the computation of an automaton. For the task of VQA, the model constructs a probabilistic scene graph to capture the semantics of a given image, which it then treats as a state machine, traversing its states as guided by the question to perform sequential reasoning.
• Figure 2: Question examples along with answers predicted by the NSM. The questions involve diverse reasoning skills such as multi-step inference, relational and spatial reasoning, logic and comparisons.
• Figure 3: A visualization of object masks from the inferred scene graphs, which form the basis for our model.
• Figure 4: A visualization of a graph traversal step, where attention is being shifted from one node to its neighbor along the most relevant edge.
• Figure 5: A visualization of the NSM's reasoning process: given an image and a question (left side), the model first builds a probabilistic scene graph (the blue box and the image on the right), and translates the question into a series of instructions (the green and purple boxes, where for each instruction we present its closest concept (or word) in vector space (section 3.1)). The model then performs sequential reasoning over the graph, attending to relevant object nodes in the image's scene as guided by the instructions, to iteratively compute the answer.