Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Continual Learning as Shared-Manifold Continuation Under Compatible Shift

Henry J. Kobs

Abstract

Continual learning methods usually preserve old behavior by regularizing parameters, matching old outputs, or replaying previous examples. These strategies can reduce forgetting, but they do not directly specify how the latent representation should evolve. We study a narrower geometric alternative for the regime where old and new data should remain on the same latent support: continual learning as continuation of a shared manifold. We instantiate this view within Support-Preserving Manifold Assimilation (SPMA) and evaluate a geometry-preserving variant, SPMA-OG, that combines sparse replay, output distillation, relational geometry preservation, local smoothing, and chart-assignment regularization on old anchors. On representative compatible-shift CIFAR10 and Tiny-ImageNet runs, SPMA-OG improves over sparse replay baselines in old-task retention and representation-preservation metrics while remaining competitive on new-task accuracy. On a controlled synthetic atlas-manifold benchmark, it achieves near-perfect anchor-geometry preservation while also improving new-task accuracy over replay. These results provide evidence that geometry-aware anchor regularization is a useful inductive bias when continual learning should preserve a shared latent support rather than create a new one.

Continual Learning as Shared-Manifold Continuation Under Compatible Shift

Abstract

Continual learning methods usually preserve old behavior by regularizing parameters, matching old outputs, or replaying previous examples. These strategies can reduce forgetting, but they do not directly specify how the latent representation should evolve. We study a narrower geometric alternative for the regime where old and new data should remain on the same latent support: continual learning as continuation of a shared manifold. We instantiate this view within Support-Preserving Manifold Assimilation (SPMA) and evaluate a geometry-preserving variant, SPMA-OG, that combines sparse replay, output distillation, relational geometry preservation, local smoothing, and chart-assignment regularization on old anchors. On representative compatible-shift CIFAR10 and Tiny-ImageNet runs, SPMA-OG improves over sparse replay baselines in old-task retention and representation-preservation metrics while remaining competitive on new-task accuracy. On a controlled synthetic atlas-manifold benchmark, it achieves near-perfect anchor-geometry preservation while also improving new-task accuracy over replay. These results provide evidence that geometry-aware anchor regularization is a useful inductive bias when continual learning should preserve a shared latent support rather than create a new one.
Paper Structure (19 sections, 10 equations, 4 figures, 4 tables)

This paper contains 19 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem setup
  5. Local chart memory
  6. SPMA objective family
  7. The SPMA-OG instantiation
  8. Experimental Setup
  9. Benchmarks
  10. Models and protocol
  11. Baselines
  12. Metrics
  13. Results
  14. Compatible-shift benchmarks
  15. Representation preservation
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Overview of the SPMA viewpoint. Frozen teacher features define a local chart memory. Old anchors preserve function, geometry, and chart structure; the student is then adapted to the new task using cross-entropy together with anchor-based geometric regularization.
  • Figure 2: Synthetic atlas-manifold benchmark. The true latent surface is shown at left, followed by teacher, plain fine-tuning, replay-anchor, old-geometry, and SPMA-OG feature views. SPMA-OG preserves the old manifold most faithfully while still absorbing the new task into the same shared support.
  • Figure 3: Old/new tradeoff on the two compatible-shift benchmarks used in the paper. SPMA-OG occupies the best or near-best region while preserving the strongest latent geometry.
  • Figure 4: Representation preservation summary. Compared with replay-only and plain fine-tuning, SPMA-OG yields markedly stronger CKA and pairwise-distance correlation, especially on Tiny-ImageNet compatible shift.