Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning

TL;DR

This work addresses catastrophic forgetting in continual learning by introducing FOPNG, a Fisher-space optimization that projects updates onto the Fisher-orthogonal complement of past task gradients. By solving a constrained, closed-form update in information geometry—$v^* = \epsilon \frac{F_{new}^{-1/2} P F_{new}^{-1/2} g}{\sqrt{g^T F_{new}^{-1/2} P^T P F_{new}^{-1/2} g}}$ with $P = I - M(M^T M)^{-1} M^T$ and $M = F_{new}^{-1/2} F_{old} G$—the method ensures progress on the new task while controlling changes in prior tasks through Fisher-norm constraints. It unifies natural gradient descent with orthogonal gradient ideas within a principled information-geometric framework and demonstrates strong empirical performance on Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100 using efficient diagonal Fisher approximations. The contributions include a theoretical framework, a closed-form projected update, and practical implementations with diagonal Fisher, supported by ablations showing robustness to hyperparameters. Overall, FOPNG offers a reparameterization-invariant, stable, and scalable approach to continual learning that reduces forgetting without significant data replay or architectural changes, enabling safer learning in data-restricted settings.

Abstract

Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tasks. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks. Unlike existing methods that operate in Euclidean parameter space, FOPNG projects gradients onto the Fisher-orthogonal complement of previous task gradients. This approach unifies natural gradient descent with orthogonal gradient methods within an information-geometric framework. The resulting update direction is invariant under reparameterization, guarantees descent in the Fisher metric, and helps preserve prior task outputs. We provide theoretical analysis establishing the properties of the projected update, describe efficient and practical implementations using the diagonal Fisher, and demonstrate strong results on standard continual learning benchmarks such as Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100.

Figures (4)

• Figure 1: Comparisons across benchmarks. Shaded regions demonstrate 68% confidence intervals across 5 independent training runs.
• Figure 2: In general, larger batch sizes increase performance and reduce variance. Shaded regions show 95% confidence intervals across 5 independent runs.
• Figure 3: Performance is generally stable in intermediate ranges of $\lambda$. Shaded regions show 95% confidence intervals across 5 independent runs.
• Figure 4: Performance is extremely stable across all values of $\alpha$. Shaded regions show 95% confidence intervals across 5 independent runs

Theorems & Definitions (15)

• Theorem 2.1: Reparameterization Invariance of Fisher Norm
• proof
• Theorem 2.2: Local Quadratic Approximation of KL Divergence
• proof
• Theorem 3.2: FOPNG Update
• proof
• Theorem 3.3: Descent Direction
• proof
• Theorem 3.4: FNG Closed-Form Solution
• proof