VSA4VQA: Scaling a Vector Symbolic Architecture to Visual Question Answering on Natural Images

TL;DR

VSA4VQA presents a cognitive-inspired, four-dimensional vector symbolic architecture to perform visual question answering on natural images. By encoding objects with position and size in a unified SSP memory and learning 37 spatial query masks, the approach enables complex spatial reasoning in a zero-shot setting, augmented by CLIP for attribute queries. On the GQA dataset, it achieves competitive zero-shot accuracy (~46.5%), with capacity analyses showing 1,024 dimensions provide a favorable balance between decoding fidelity and efficiency. The work demonstrates that cognitively plausible image representations can scale to natural scenes and complex relational queries, offering a pathway toward hardware-efficient, interpretable VQA systems and highlighting error modes for future improvement.

Abstract

While Vector Symbolic Architectures (VSAs) are promising for modelling spatial cognition, their application is currently limited to artificially generated images and simple spatial queries. We propose VSA4VQA - a novel 4D implementation of VSAs that implements a mental representation of natural images for the challenging task of Visual Question Answering (VQA). VSA4VQA is the first model to scale a VSA to complex spatial queries. Our method is based on the Semantic Pointer Architecture (SPA) to encode objects in a hyperdimensional vector space. To encode natural images, we extend the SPA to include dimensions for object's width and height in addition to their spatial location. To perform spatial queries we further introduce learned spatial query masks and integrate a pre-trained vision-language model for answering attribute-related questions. We evaluate our method on the GQA benchmark dataset and show that it can effectively encode natural images, achieving competitive performance to state-of-the-art deep learning methods for zero-shot VQA.

Figures (4)

• Figure 1: Example question from GQA (Hudson & Manning, 2019). Our VSA4VQA method performs three steps: 1. select the lamp, 2. find items to the right of the lamp with a spatial query mask encoded in SSPs, and 3. filter the positive proposals to find furniture, yielding the correct answer "bed".
• Figure 2: Overview of VSA4VQA. Object bounding boxes' $(x, y)$-location, width $w$, and height $h$ are encoded into SSP memory. The program generator from Chen et al. (2020) maps the question to functions. Our method then implements these functions with (1) SSP unbinding, (2) CLIP, (3) SSP query masks, and finally queries the name of the resulting object (4) to answer the question. For implementation details on all functions see Table 1.
• Figure 3: Query mask generation for relation to the right of. For all samples of the relation (1): create object masks (2), normalise masks to same scale and center position (3), and add them to obtain final relation mask (4).
• Figure 4: Error analysis by type of question. Percentage on top indicates total share of questions that include the respective query in their program. Wrong answers are further split to indicate when no answer was given.