Ferret: An Efficient Online Continual Learning Framework under Varying Memory Constraints

TL;DR

Ferret tackles real-time online continual learning under varying memory budgets by marrying fine-grained pipeline parallelism with an iterative gradient compensation scheme. It jointly optimizes model partitioning and pipeline planning through a bi-level framework, enabling memory-aware throughput while mitigating gradient staleness with a Taylor-series-based estimator and diagonal Fisher approximation. Empirical results across 20 benchmarks and 5 integrated OCL algorithms show Ferret achieving up to $3.7\times$ lower memory overhead to reach the same online accuracy and robust performance across diverse memory budgets, with the Iter-Fisher method providing automatic, data-adaptive compensation. This framework advances practical, scalable OCL in real-time environments by balancing latency, throughput, and memory usage, delivering significant gains in online adaptation without sacrificing stability.

Abstract

In the realm of high-frequency data streams, achieving real-time learning within varying memory constraints is paramount. This paper presents Ferret, a comprehensive framework designed to enhance online accuracy of Online Continual Learning (OCL) algorithms while dynamically adapting to varying memory budgets. Ferret employs a fine-grained pipeline parallelism strategy combined with an iterative gradient compensation algorithm, ensuring seamless handling of high-frequency data with minimal latency, and effectively counteracting the challenge of stale gradients in parallel training. To adapt to varying memory budgets, its automated model partitioning and pipeline planning optimizes performance regardless of memory limitations. Extensive experiments across 20 benchmarks and 5 integrated OCL algorithms show Ferret's remarkable efficiency, achieving up to 3.7$\times$ lower memory overhead to reach the same online accuracy compared to competing methods. Furthermore, Ferret consistently outperforms these methods across diverse memory budgets, underscoring its superior adaptability. These findings position Ferret as a premier solution for efficient and adaptive OCL framework in real-time environments.

Figures (11)

• Figure 1: The overall workflow of Ferret. In A, based on the optimal model partition scheme $L^*$ and pipeline configuration $C^*$, $N$ workers are spawned to initiate fine-grained pipeline parallelism that consumes streaming data interleavedly, and update the same model asynchronously by iteratively compensating stale gradients. In B, $L^*$ and $C^*$ are obtained by optimizing Eq. \ref{['eq:main-obj']}.
• Figure 1: Online Accuracy Gain per unit of Memory ($agm_{\mathcal{B}}(\mathcal{A}, T)$) of different algorithms, where $\mathcal{B}$ is the 1-Skip. "M-", "M", "M+" refer to the ferret method with minimal, medium and maximal memory footprint, respectively.
• Figure 2: To adapt to different levels of staleness in fine-grained pipeline parallelism, $\nabla \mathcal{L}(D^t, \theta^{t+\tau})$ is iteratively approximated by $\nabla \mathcal{L}(D^t, \theta^{t})$.
• Figure 3: To further reduce approximation errors, we optimize $\lambda$ automatically by comparing historical approximations ($\nabla \mathcal{L}(D^t, \theta^t)$, etc.) and observations ($\nabla \mathcal{L}(D^{t-1}, \theta^t)$, etc.)
• Figure 4: Consumed memory of different stream learning algorithms. Ferret achieves rapid adaptation across varying memory constraints.

Theorems & Definitions (1)

• Definition 4.1: Adaptation Rate of A OCL framework