Exploring the Effectiveness of Object-Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models

TL;DR

The paper tackles how object-centric representations fare in visual reasoning compared to large foundation models, specifically for VQA. It introduces a large-scale empirical framework that trains 684 downstream VQA models across 15 upstream representations on three synthetic and two real-world multi-object datasets, using a unified downstream transformer approach. Key findings show that foundation models can match OC models without fine-tuning but require more compute, while applying OC biases to foundation models (e.g., DINOSAURv2) yields efficient, explicit representations and strong performance. The work highlights the value of downstream evaluation as a practical measure of representation quality and suggests jointly leveraging OC inductive biases with foundation models as a promising path. It also identifies limitations and directions for future research, including extensions to video, fine-tuning studies, and causal generalization.

Abstract

Object-centric (OC) representations, which model visual scenes as compositions of discrete objects, have the potential to be used in various downstream tasks to achieve systematic compositional generalization and facilitate reasoning. However, these claims have yet to be thoroughly validated empirically. Recently, foundation models have demonstrated unparalleled capabilities across diverse domains, from language to computer vision, positioning them as a potential cornerstone of future research for a wide range of computational tasks. In this paper, we conduct an extensive empirical study on representation learning for downstream Visual Question Answering (VQA), which requires an accurate compositional understanding of the scene. We thoroughly investigate the benefits and trade-offs of OC models and alternative approaches including large pre-trained foundation models on both synthetic and real-world data, ultimately identifying a promising path to leverage the strengths of both paradigms. The extensiveness of our study, encompassing over 600 downstream VQA models and 15 different types of upstream representations, also provides several additional insights that we believe will be of interest to the community at large.

Figures (28)

• Figure 1: An overview of our framework. Starting from an image and a question, we first extract image and question representations, by applying the upstream model and the text embedding module, respectively. The obtained representations are then passed to the projection layer and then, a positional encoding is applied to the text representations. Next, both are concatenated and a transformer model is applied to the combined sequence. Finally, The answer to the question is obtained by an MLP that takes the transformed value of the CLS token and produces a probability vector over all possible answers.
• Figure 2: Average accuracies on the VQA downstream task for different upstream representation models on synthetic datasets, when using T-15 as the downstream model. The bars indicate means and 95% confidence intervals with 3 random seeds, when available.
• Figure 3: Average accuracies of different models vs. downstream GFLOPs across different datasets. Points along the x-axis represent T-2, T-5, and T-15, respectively. For pre-trained models, only 1 seed is available. For other models, the results are averaged over 3 random seeds and the shaded areas indicate 95% confidence intervals.
• Figure 4: Left: Average accuracies of different models w.r.t. downstream GFLOPs on GQA. Points along the x-axis represent T-2, T-5, and T-15, respectively. For pre-trained models, only one seed is available. For other models, the results are averaged over 3 random seeds and the shaded areas indicate 95% confidence intervals. Middle & Right: Average accuracies on GQA and VQA-v2 for different upstream models, with T-15 and T-2 as the downstream models, respectively. The bars indicate means and 95% confidence intervals with 3 random seeds, when available.
• Figure 5: Correlation between property prediction accuracy (reported separately by object property) and overall VQA accuracy.