Mitigating Task-Order Sensitivity and Forgetting via Hierarchical Second-Order Consolidation

TL;DR

This paper tackles task-order sensitivity in continual learning by introducing Hierarchical Taylor Series-based Continual Learning (HTCL), a framework that couples fast local adaptation with Hessian-regularized, second-order consolidation across a hierarchical structure. By partitioning tasks into small groups and optimizing intra-group orderings, HTCL uses a Taylor-series based update to consolidate across groups, and extends to multi-level hierarchies to capture long-horizon dependencies. Empirical results across image, graph, and text domains show HTCL reduces final-performance variance across task permutations by up to 68% and improves mean accuracy while substantially reducing forgetting, with deeper hierarchies offering stronger gains for long sequences. The method is model-agnostic, scalable with practical curvature approximations, and yields a robust, reusable consolidation layer that can augment a wide range of continual learning baselines.

Abstract

We introduce $\textbf{Hierarchical Taylor Series-based Continual Learning (HTCL)}$, a framework that couples fast local adaptation with conservative, second-order global consolidation to address the high variance introduced by random task ordering. To address task-order effects, HTCL identifies the best intra-group task sequence and integrates the resulting local updates through a Hessian-regularized Taylor expansion, yielding a consolidation step with theoretical guarantees. The approach naturally extends to an $L$-level hierarchy, enabling multiscale knowledge integration in a manner not supported by conventional single-level CL systems. Across a wide range of datasets and replay and regularization baselines, HTCL acts as a model-agnostic consolidation layer that consistently enhances performance, yielding mean accuracy gains of $7\%$ to $25\%$ while reducing the standard deviation of final accuracy by up to $68\%$ across random task permutations.

Figures (8)

• Figure 1: Task-order sensitivity in CL and its mitigation via HTCL. Left: Standard CL methods exhibit high sensitivity to task ordering. Right: HTCL achieves consistent performance regardless of task ordering, producing a narrow accuracy distribution.
• Figure 2: (top) The HTCL framework for a sequence of six tasks partitioned into three groups ($\mathrm{G}_1, \mathrm{G}_2, \mathrm{G}_3$). Within each group, all $k!$ (here $2!$) intra-group permutations are evaluated through neural network training to provide $\mathbf{w}_{\ell}^{(1)}, \mathbf{w}_{\ell}^{(2)}, \mathbf{w}_{\ell}^{(3)}.$ These three results are successively integrated into the hierarchical model $\mathbf{w}_{1}^{(t)}$. (bottom) Illustration of the update rule on a loss landscape. The current hierarchical model $\mathbf{w}_{1}^{(t-1)}$ (blue) integrates knowledge from the local model $\mathbf{w}_{\ell}^{(t)}$ (brown). Rather than naively minimizing $\mathcal{J}(\mathbf{w})$, HTCL approximates $\mathcal{J}(\mathbf{w}_{1}^{(t-1)}+\Delta\mathbf{w})$ (blue cross) using a second-order Taylor expansion of the loss around $\mathbf{w}_{1}^{(t-1)}$ with a regularization term that penalizes deviation from $\mathbf{w}_{\ell}^{(t)}$. This yields the updated hierarchical model $\mathbf{w}_{1}^{(t)}$ (red star).
• Figure 3: Task-order robustness and memory retention of HTCL with SER on SplitMNIST. Accuracy distributions are computed over all $120$ permutations of the five-task sequence. (a) Mean accuracy across all tasks after sequential training, comparing SER alone against SER augmented with 2-, 3-, 4-, and 5-level HTCL hierarchies. Error bars indicate standard deviation. (b) Per-task standard deviation across permutations. HTCL consistently achieves lower or comparable variance for most tasks across all hierarchy depths. (c) Overall standard deviation across methods, demonstrating improved task-order robustness with HTCL. (d) Mean forgetting across methods. HTCL reduces forgetting and improves memory retention. (e) Computation time per permutation. HTCL incurs moderate overhead relative to the baseline, reflecting a practical trade-off between stability and computational cost.
• Figure 4: Scalability of HTCL under long task sequences. We extend the two-level local–global structure to a three-level hierarchy as described in Eq. \ref{['eq:multi_hierarchy']}, using DER as a baseline for illustration. The three-layer HTCL demonstrates higher memory retention and lower variance when handling long task sequences. (a) Mean accuracy per task: DER + HTCL-L3 achieves $45.2\%$ compared to $40.6\%$ for DER alone. (b) Standard deviation per task: DER + HTCL-L3 maintains a lower standard deviation ($12.0$) compared to DER alone ($17.4$).
• Figure 5: Impact of the number of tasks on forgetting in CL across different domains. Forgetting increases as the number of tasks grows, with models experiencing up to 20% accuracy drop in a 10-task setting.

Theorems & Definitions (12)

• Remark 3.1: Time and space complexity
• Theorem 1.1
• proof
• Lemma 1.2: Recursive formulation
• proof
• Theorem 1.3: Performance Bound
• proof
• Proposition 1.4: Exact computational cost
• proof : Proof sketch
• Corollary 1.5: Hierarchy cost