Distributed Dynamic Associative Memory via Online Convex Optimization

TL;DR

The paper tackles memory recall in distributed, time-varying multi-agent settings by formulating distributed dynamic associative memory (DDAM) and introducing a tree-based online gradient descent algorithm (DDAM-TOGD). It provides regret guarantees, showing sublinear static regret and path-length dependent dynamic regret that account for communication delays, and introduces a combinatorial routing-tree design to minimize delays and improve performance. Empirical results on synthetic and real wireless data demonstrate DDAM-TOGD's superior accuracy and robustness compared to consensus-based baselines. The work offers principled design guidelines for online, distributed memory systems in dynamic networks and lays groundwork for extending DDAM to time-varying connectivity and non-linear associative memories.

Abstract

An associative memory (AM) enables cue-response recall, and it has recently been recognized as a key mechanism underlying modern neural architectures such as Transformers. In this work, we introduce the concept of distributed dynamic associative memory (DDAM), which extends classical AM to settings with multiple agents and time-varying data streams. In DDAM, each agent maintains a local AM that must not only store its own associations but also selectively memorize information from other agents based on a specified interest matrix. To address this problem, we propose a novel tree-based distributed online gradient descent algorithm, termed DDAM-TOGD, which enables each agent to update its memory on the fly via inter-agent communication over designated routing trees. We derive rigorous performance guarantees for DDAM-TOGD, proving sublinear static regret in stationary environments and a path-length dependent dynamic regret bound in non-stationary environments. These theoretical results provide insights into how communication delays and network structure impact performance. Building on the regret analysis, we further introduce a combinatorial tree design strategy that optimizes the routing trees to minimize communication delays, thereby improving regret bounds. Numerical experiments demonstrate that the proposed DDAM-TOGD framework achieves superior accuracy and robustness compared to representative online learning baselines such as consensus-based distributed optimization, confirming the benefits of the proposed approach in dynamic, distributed environments.

Figures (9)

• Figure 1: (a) In a distributed dynamic associative memory (DDAM) system, each agent maintains a local memory, and corresponding AM mechanism, by processing local, streaming, data, as well as by interacting with other agents over a physical network. Each agent $n$ collects streaming data in the form of key $\mathbf{k}_{n,t}$ and value $\mathbf{v}_{n,t}$ over discrete time $t=1,2,...,T$. The blue region represents the subset $\mathcal{N}_n$ of physical neighbors that can directly exchange information with agent $n$ over the network. (b) The goal of DDAM is for the memory at each agent not only to recall associations from its own data, but also from a subset of other agents. The purple region in the figure represents the logical subset $\mathcal{W}_n$ of agents whose data is relevant for agent $n$. The nodes in subset $\mathcal{W}_n$ may not be physical neighbors of agent $n$. Through inter-agent communication over the physical network, DDAM aims at recalling information from both local and logically related agents in an online fashion.
• Figure 2: Physical topology considered in the experiments.
• Figure 3: Regret versus time horizon $T$ ($\rho = 0.75$, $y_0 = 2$, and $y_1 = 10$).
• Figure 4: Regret at $T = 2500$ versus memory correlation parameter $\rho$ ($y_0 = 6$, and $y_1 = 10$).
• Figure 5: Regret at $T = 2500$ versus Dirichlet parameter $y_0$ ($\rho = 0.75$, and $y_1 = 10$).

Theorems & Definitions (8)

• Theorem 1
• Corollary 1
• Theorem 2
• Theorem 3
• Corollary 2
• Lemma 1
• Theorem 4
• Lemma 2