Learning Memory Mechanisms for Decision Making through Demonstrations

TL;DR

This work introduces AttentionTuner to leverage memory dependency pairs in Transformers and finds significant improvements across several tasks compared to standard Transformers when evaluated on Memory Gym and the Long-term Memory Benchmark.

Abstract

In Partially Observable Markov Decision Processes, integrating an agent's history into memory poses a significant challenge for decision-making. Traditional imitation learning, relying on observation-action pairs for expert demonstrations, fails to capture the expert's memory mechanisms used in decision-making. To capture memory processes as demonstrations, we introduce the concept of memory dependency pairs $(p, q)$ indicating that events at time $p$ are recalled for decision-making at time $q$. We introduce AttentionTuner to leverage memory dependency pairs in Transformers and find significant improvements across several tasks compared to standard Transformers when evaluated on Memory Gym and the Long-term Memory Benchmark. Code is available at https://github.com/WilliamYue37/AttentionTuner.

Figures (6)

• Figure 1: The red arrows indicate episodic memory dependencies labeled by an expert. The correct action to take in $o_2$ depends on $o_0$ and the correct action to take in $o_3$ depends on $o_1$. These memory dependency pairs are used to create the expert self-attention matrix $E \in \{0,1\}^{n \times n}$ where $n$ is the length of the sequence and $E_{ij} = 1$ only if the expert has indicated that observation $o_j$ should be recalled from memory at observation $o_i$, and $E_{ij} = 0$ otherwise. A binary cross entropy loss is taken between $E$ and the learner's self-attention matrix $\sigma(QK^\top)$ to form the memory loss $\mathcal{L}(\sigma(QK^\top), E)$ that encourages the learner to learn the expert's memory mechanism. The memory loss is scaled by $\lambda$ to match the magnitude of $\mathcal{L}_{\mathrm{IL}}$ and then added to form the final loss used during training.
• Figure 2: Architecture of the causal Transformer for sequential decision making modeling used in all AttentionTuner and vanilla Transformer experiments. The model embeds observations $o_t$ using a Convolutional Neural Network (CNN) and actions $a_t$ using a Multilayer Perceptron (MLP) network. Positional encodings are added to these embeddings to preserve positional context. Subsequently, they are fed into a causal Transformer decoder, which predicts future actions $a_{t}, a_{t+1}, \ldots$, conditioned on past events. Complete architectural details can be found in Appendix \ref{['architecture']}.
• Figure 3: Overview of LTMB and MemGym tasks, each represented by a sequence of three images: the initial observation (left), a subsequent observation that must be recalled (middle), and a decision-making state that depends on memory of the middle state (right). Detailed task descriptions are available in Appendix \ref{['app:descrip']}.
• Figure 4: Median learning curves with interquartile range for Memory Gym and LTMB tasks are presented. The median is used rather than the mean due to the influence of outliers. Detailed mean learning curves are accessible in Appendix \ref{['mean_curves_section']} Figure \ref{['mean_curves']}. The top row displays the training loss while the bottom row shows the test accuracy on action prediction (not success rate). For a direct comparison with the vanilla Transformer, only the imitation learning loss (Equation \ref{['ILloss']}) is plotted for AttentionTuner.
• Figure 5: Success rates and 90% confidence intervals for AttentionTuner training on 'Mortar Mayhem' and 'Hallway' tasks with missing annotations. Numerical values are presented in Appendix \ref{['sec:annotation']} Table \ref{['tbl:annotations_ablation']}.