Improvements of Dark Experience Replay and Reservoir Sampling towards Better Balance between Consolidation and Plasticity

TL;DR

This work tackles the trade-off between memory consolidation and plasticity in continual learning by augmenting two established components: DER and RS. It introduces A2ER, comprising adaptation, block, and correction strategies to DER, enabling automatic weight tuning, selective replay of erroneous past data, and correction of past outputs. It also proposes O2S for RS, featuring a q-logarithm-based generalized acceptance, plural buffers for multi-timescale memory, and data omission to remove unnecessary samples. Across regression, classification, and reinforcement learning benchmarks, the proposed methods yield steady performance gains, demonstrating improved stability and generalization by balancing consolidation with plasticity in dynamic data distributions.

Abstract

Continual learning is the one of the most essential abilities for autonomous agents, which can incrementally learn daily-life skills. For this ultimate goal, a simple but powerful method, dark experience replay (DER), has been proposed recently. DER mitigates catastrophic forgetting, in which the skills acquired in the past are unintentionally forgotten, by stochastically storing the streaming data in a reservoir sampling (RS) buffer and by relearning them or retaining the past outputs for them. However, since DER considers multiple objectives, it will not function properly without appropriate weighting of them. In addition, the ability to retain past outputs inhibits learning if the past outputs are incorrect due to distribution shift or other effects. This is due to a tradeoff between memory consolidation and plasticity. The tradeoff is hidden even in the RS buffer, which gradually stops storing new data for new skills in it as data is continuously passed to it. To alleviate the tradeoff and achieve better balance, this paper proposes improvement strategies to each of DER and RS. Specifically, DER is improved with automatic adaptation of weights, block of replaying erroneous data, and correction of past outputs. RS is also improved with generalization of acceptance probability, stratification of plural buffers, and intentional omission of unnecessary data. These improvements are verified through multiple benchmarks including regression, classification, and reinforcement learning problems. As a result, the proposed methods achieve steady improvements in learning performance by balancing the memory consolidation and plasticity.

Figures (8)

• Figure 1: A2ER with three strategies for improving DER
• Figure 2: Updated $\Delta_\tau$ using $\eta_\tau$
• Figure 3: O2S with three strategies for improving RS
• Figure 4: Example of $f_q(n)$ with $N^{\mathrm{RS}}=100$
• Figure 5: Data processing in plural strategy with $L$ RS buffers