MacVQA: Adaptive Memory Allocation and Global Noise Filtering for Continual Visual Question Answering

TL;DR

MacVQA addresses continual visual question answering by combining Global Noise Filtering and Adaptive Memory Allocation to produce robust, memory-efficient multimodal representations. The approach integrates region-aware denoising, global feature fusion, prototype-based memory with gated fusion, and a cross-attention decoder, all optimized via a composite loss that balances representation quality, memory stability, and decoding accuracy. Empirically, MacVQA achieves state-of-the-art performance across ten continual VQA tasks on VQA v2, demonstrating strong retention (low forgetting) and superior generalization to novel compositions, with clear gains from both GonF and AMA. The framework offers a practical path toward scalable, robust continual VQA in dynamic, multimodal environments, with potential extensions to additional modalities and larger memory systems.

Abstract

Visual Question Answering (VQA) requires models to reason over multimodal information, combining visual and textual data. With the development of continual learning, significant progress has been made in retaining knowledge and adapting to new information in the VQA domain. However, current methods often struggle with balancing knowledge retention, adaptation, and robust feature representation. To address these challenges, we propose a novel framework with adaptive memory allocation and global noise filtering called MacVQA for visual question answering. MacVQA fuses visual and question information while filtering noise to ensure robust representations, and employs prototype-based memory allocation to optimize feature quality and memory usage. These designs enable MacVQA to balance knowledge acquisition, retention, and compositional generalization in continual VQA learning. Experiments on ten continual VQA tasks show that MacVQA outperforms existing baselines, achieving 43.38% average accuracy and 2.32% average forgetting on standard tasks, and 42.53% average accuracy and 3.60% average forgetting on novel composition tasks.

Figures (6)

• Figure 1: Illustration with continual learning methods for VQA. (a) Continual Learning-based VQA: The system learns new tasks while retaining prior knowledge. (b) Global Noise Filtering: Refines visual features by removing noise for better task performance. (c) Adaptive Memory Allocation: Combines learned and memorized knowledge for question answering.
• Figure 2: The proposed framework consists of three key modules: (a) Global Noise Filtering, which refines visual features by removing noise through scoring, fusion, and a Denoising Autoencoder (DAE); (b) Memory Pool, storing visual and textual prototypes as dynamic references; (c) Adaptive Memory Allocation, which dynamically allocates prototype features from the memory pool to enhance adaptability and generalization.
• Figure 3: Similarity score radar plots for two examples (the k-th data from task i-1 and task i). Left: question memory; right: visual memory. Top-3 prototypes are selected by similarity.
• Figure 4: Memory size sensitivity analysis for VQA v2 under standard and novel composition testing paradigms
• Figure 5: Task-Specific Hyperparameter Sensitivity of MacVQA.