Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Tasks

Susmit Agrawal, Krishn Vishwas Kher, Saksham Mittal, Swarnim Maheshwari, Vineeth N. Balasubramanian

TL;DR

The paper introduces Memory-Integrated Reconfigurable Adapters (MIRA), a memory-augmented framework that uses Hopfield-style associative memory to retrieve per-sample, task-specific adapter mixtures on a ViT backbone. Adapters are stored as memory values and retrieval keys are learned post-hoc, enabling unified handling of Domain Generalization, Domain Incremental Learning, and Class Incremental Learning with minimal architectural changes. Empirically, MIRA achieves state-of-the-art or strong results across DG benchmarks and outperforms specialized CL baselines on several datasets, with modest memory and latency overhead. This biologically inspired approach demonstrates rapid task switching and enduring knowledge retention, highlighting a path toward memory-centric artificial intelligence.

Abstract

Organisms constantly pivot between tasks such as evading predators, foraging, traversing rugged terrain, and socializing, often within milliseconds. Remarkably, they preserve knowledge of once-learned environments sans catastrophic forgetting, a phenomenon neuroscientists hypothesize, is due to a singular neural circuitry dynamically overlayed by neuromodulatory agents such as dopamine and acetylcholine. In parallel, deep learning research addresses analogous challenges via domain generalization (DG) and continual learning (CL), yet these methods remain siloed, despite the brains ability to perform them seamlessly. In particular, prior work has not explored architectures involving associative memories (AMs), which are an integral part of biological systems, to jointly address these tasks. We propose Memory-Integrated Reconfigurable Adapters (MIRA), a unified framework that integrates Hopfield-style associative memory modules atop a shared backbone. Associative memory keys are learned post-hoc to index and retrieve an affine combination of stored adapter updates for any given task or domain on a per-sample basis. By varying only the task-specific objectives, we demonstrate that MIRA seamlessly accommodates domain shifts and sequential task exposures under one roof. Empirical evaluations on standard benchmarks confirm that our AM-augmented architecture significantly enhances adaptability and retention: in DG, MIRA achieves SoTA out-of-distribution accuracy, and in incremental learning settings, it outperforms architectures explicitly designed to handle catastrophic forgetting using generic CL algorithms. By unifying adapter-based modulation with biologically inspired associative memory, MIRA delivers rapid task switching and enduring knowledge retention in a single extensible architecture, charting a path toward more versatile and memory-augmented AI systems.

Memory-Integrated Reconfigurable Adapters: A Unified Framework for Settings with Multiple Tasks

TL;DR

The paper introduces Memory-Integrated Reconfigurable Adapters (MIRA), a memory-augmented framework that uses Hopfield-style associative memory to retrieve per-sample, task-specific adapter mixtures on a ViT backbone. Adapters are stored as memory values and retrieval keys are learned post-hoc, enabling unified handling of Domain Generalization, Domain Incremental Learning, and Class Incremental Learning with minimal architectural changes. Empirically, MIRA achieves state-of-the-art or strong results across DG benchmarks and outperforms specialized CL baselines on several datasets, with modest memory and latency overhead. This biologically inspired approach demonstrates rapid task switching and enduring knowledge retention, highlighting a path toward memory-centric artificial intelligence.

Abstract

Organisms constantly pivot between tasks such as evading predators, foraging, traversing rugged terrain, and socializing, often within milliseconds. Remarkably, they preserve knowledge of once-learned environments sans catastrophic forgetting, a phenomenon neuroscientists hypothesize, is due to a singular neural circuitry dynamically overlayed by neuromodulatory agents such as dopamine and acetylcholine. In parallel, deep learning research addresses analogous challenges via domain generalization (DG) and continual learning (CL), yet these methods remain siloed, despite the brains ability to perform them seamlessly. In particular, prior work has not explored architectures involving associative memories (AMs), which are an integral part of biological systems, to jointly address these tasks. We propose Memory-Integrated Reconfigurable Adapters (MIRA), a unified framework that integrates Hopfield-style associative memory modules atop a shared backbone. Associative memory keys are learned post-hoc to index and retrieve an affine combination of stored adapter updates for any given task or domain on a per-sample basis. By varying only the task-specific objectives, we demonstrate that MIRA seamlessly accommodates domain shifts and sequential task exposures under one roof. Empirical evaluations on standard benchmarks confirm that our AM-augmented architecture significantly enhances adaptability and retention: in DG, MIRA achieves SoTA out-of-distribution accuracy, and in incremental learning settings, it outperforms architectures explicitly designed to handle catastrophic forgetting using generic CL algorithms. By unifying adapter-based modulation with biologically inspired associative memory, MIRA delivers rapid task switching and enduring knowledge retention in a single extensible architecture, charting a path toward more versatile and memory-augmented AI systems.

Paper Structure

This paper contains 18 sections, 3 theorems, 12 equations, 2 figures, 12 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Method
  4. Experiments and Results
  5. Conclusion
  6. Acknowledgment and Funding Transparency Statement
  7. Appendix
  8. MIRA: A Multi-Perspective Analysis
  9. Meta-learning (Hypernetworks)
  10. Functional Interpolation/Extrapolation
  11. Test Time Adaptation
  12. Biological Perspective
  13. Implications.
  14. Additional Experiments
  15. Experimental Details
  16. ...and 3 more sections

Key Result

Lemma 1

Let $\mathcal{H}_k$ denote the reproducing-kernel Hilbert space induced by the kernel $k(\cdot, \cdot)$, and assume an optimal solution to Eqn. eqn:test-time-opt$\{\alpha^{*}_{t,l}(x)\}^{T,L}$ admits a representation in a finite eigenbasis of the integral operator associated with $k$. Then, for any

Figures (2)

  • Figure 1: Associative memories can enable networks to quickly adapt to diverse tasks, by storing and recalling task-specific weights on-demand. MIRA proposes a framework for such an approach.
  • Figure 2: Overview of MIRA for Domain Generalization and Continual Learning scenarios. In DG, all training tasks are provided together to both the Adaptation and the Consolidation stages. In the CL scenarios, the dataset for each task arrives sequentially, and each dataset is passed to both stages. The Adaptation stage trains adapters for each task, while the Consolidation stage learns the associated keys for the stored adapters.

Theorems & Definitions (6)

  • Lemma 1
  • proof
  • Lemma 2: DualGPM--Hebbian Gradient Subspace Equivalence
  • proof
  • Theorem 1
  • proof