Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Attention Mechanism based Cognition-level Scene Understanding

Xuejiao Tang, Wenbin Zhang

TL;DR

The paper tackles cognition-level scene understanding in Visual Commonsense Reasoning by predicting answers and rationales from image-question pairs. It introduces PAVCR, a parallel-attention, memory-augmented architecture that fuses visual and textual information and encodes commonsense in parallel to mitigate long-sequence information loss. Empirical results on the VCR dataset show that PAVCR achieves state-of-the-art performance across subtasks while offering improved efficiency over BiLSTM-based predecessors. The work demonstrates effective high-level reasoning capability and provides a foundation for interpretable, cognition-oriented visual reasoning in practical applications.

Abstract

Given a question-image input, the Visual Commonsense Reasoning (VCR) model can predict an answer with the corresponding rationale, which requires inference ability from the real world. The VCR task, which calls for exploiting the multi-source information as well as learning different levels of understanding and extensive commonsense knowledge, is a cognition-level scene understanding task. The VCR task has aroused researchers' interest due to its wide range of applications, including visual question answering, automated vehicle systems, and clinical decision support. Previous approaches to solving the VCR task generally rely on pre-training or exploiting memory with long dependency relationship encoded models. However, these approaches suffer from a lack of generalizability and losing information in long sequences. In this paper, we propose a parallel attention-based cognitive VCR network PAVCR, which fuses visual-textual information efficiently and encodes semantic information in parallel to enable the model to capture rich information for cognition-level inference. Extensive experiments show that the proposed model yields significant improvements over existing methods on the benchmark VCR dataset. Moreover, the proposed model provides intuitive interpretation into visual commonsense reasoning.

Attention Mechanism based Cognition-level Scene Understanding

TL;DR

The paper tackles cognition-level scene understanding in Visual Commonsense Reasoning by predicting answers and rationales from image-question pairs. It introduces PAVCR, a parallel-attention, memory-augmented architecture that fuses visual and textual information and encodes commonsense in parallel to mitigate long-sequence information loss. Empirical results on the VCR dataset show that PAVCR achieves state-of-the-art performance across subtasks while offering improved efficiency over BiLSTM-based predecessors. The work demonstrates effective high-level reasoning capability and provides a foundation for interpretable, cognition-oriented visual reasoning in practical applications.

Abstract

Given a question-image input, the Visual Commonsense Reasoning (VCR) model can predict an answer with the corresponding rationale, which requires inference ability from the real world. The VCR task, which calls for exploiting the multi-source information as well as learning different levels of understanding and extensive commonsense knowledge, is a cognition-level scene understanding task. The VCR task has aroused researchers' interest due to its wide range of applications, including visual question answering, automated vehicle systems, and clinical decision support. Previous approaches to solving the VCR task generally rely on pre-training or exploiting memory with long dependency relationship encoded models. However, these approaches suffer from a lack of generalizability and losing information in long sequences. In this paper, we propose a parallel attention-based cognitive VCR network PAVCR, which fuses visual-textual information efficiently and encodes semantic information in parallel to enable the model to capture rich information for cognition-level inference. Extensive experiments show that the proposed model yields significant improvements over existing methods on the benchmark VCR dataset. Moreover, the proposed model provides intuitive interpretation into visual commonsense reasoning.
Paper Structure (17 sections, 12 equations, 9 figures, 3 tables)

This paper contains 17 sections, 12 equations, 9 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notations and Problem Formulation
  4. Proposed Framework
  5. Feature Representation Layer
  6. Multimodal Feature Fusion Layer
  7. Commonsense Encoder
  8. Prediction Layer
  9. Experimental Results
  10. Dataset
  11. Metric
  12. Models for Baseline Comparisons
  13. Experimental Results
  14. Ablation Studies
  15. Case Study
  16. ...and 2 more sections

Figures (9)

  • Figure S1: Proposed framework.
  • Figure S2: An example of VCR running.
  • Figure S3: Multimodal Feature Fusion Layer.
  • Figure S4: Co-attention.
  • Figure S5: Overview of the types of inference required by questions in VCR.
  • ...and 4 more figures