Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Method of Selective Attention for Reservoir Based Agents

Kevin McKee

TL;DR

In reservoir-based reinforcement learning, training efficiency is hindered by high-dimensional, uninformative inputs. The paper compares Layer Normalization, a simple vector filter, and a novel Excessively Parameterized Input Concealment (EPIC) that uses reward-driven gradients to suppress inputs, with EPIC achieving the largest speedups. Empirically, input masking speeds training substantially, up to about four-fold versus a no-masking baseline and around two-fold over LayerNorm, sustaining gains across different distraction sizes. The work supports reward-driven selective attention as a practical, low-cost technique to enhance training efficiency for memory-augmented RL agents, with implications for designing attention mechanisms in sequential decision tasks.

Abstract

Training of deep reinforcement learning agents is slowed considerably by the presence of input dimensions that do not usefully condition the reward function. Existing modules such as layer normalization can be trained with weight decay to act as a form of selective attention, i.e. an input mask, that shrinks the scale of unnecessary inputs, which in turn accelerates training of the policy. However, we find a surprising result that adding numerous parameters to the computation of the input mask results in much faster training. A simple, high dimensional masking module is compared with layer normalization and a model without any input suppression. The high dimensional mask resulted in a four-fold speedup in training over the null hypothesis and a two-fold speedup in training over the layer normalization method.

A Method of Selective Attention for Reservoir Based Agents

TL;DR

In reservoir-based reinforcement learning, training efficiency is hindered by high-dimensional, uninformative inputs. The paper compares Layer Normalization, a simple vector filter, and a novel Excessively Parameterized Input Concealment (EPIC) that uses reward-driven gradients to suppress inputs, with EPIC achieving the largest speedups. Empirically, input masking speeds training substantially, up to about four-fold versus a no-masking baseline and around two-fold over LayerNorm, sustaining gains across different distraction sizes. The work supports reward-driven selective attention as a practical, low-cost technique to enhance training efficiency for memory-augmented RL agents, with implications for designing attention mechanisms in sequential decision tasks.

Abstract

Training of deep reinforcement learning agents is slowed considerably by the presence of input dimensions that do not usefully condition the reward function. Existing modules such as layer normalization can be trained with weight decay to act as a form of selective attention, i.e. an input mask, that shrinks the scale of unnecessary inputs, which in turn accelerates training of the policy. However, we find a surprising result that adding numerous parameters to the computation of the input mask results in much faster training. A simple, high dimensional masking module is compared with layer normalization and a model without any input suppression. The high dimensional mask resulted in a four-fold speedup in training over the null hypothesis and a two-fold speedup in training over the layer normalization method.

Paper Structure

This paper contains 10 sections, 3 equations, 1 figure, 1 table.

Table of Contents

  1. Distracting Multi-armed Bandit
  2. Models
  3. Layer Normalization
  4. Simple vector filter
  5. Excessively Parameterized Input Concealment
  6. Results
  7. Discussion
  8. Limitations
  9. Conclusion
  10. Appendix A: Hyperparameters

Figures (1)

  • Figure 1: Results on multi-armed bandit with 32 and 64-length distraction noise. Trained input masking by regularized layer normalization methods doubled training efficiency. Over-parameterizing the mask quadrupled training efficiency. For EPIC, the length of random vector $u$ is given in parentheses.