Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning

TL;DR

Inspired by the fly olfactory circuit, Fly-CL is proposed, a bio-inspired framework compatible with a wide range of pretrained backbones that substantially reduces training time while achieving performance comparable to or exceeding that of current state-of-the-art methods.

Abstract

Using a nearly-frozen pretrained model, the continual representation learning paradigm reframes parameter updates as a similarity-matching problem to mitigate catastrophic forgetting. However, directly leveraging pretrained features for downstream tasks often suffers from multicollinearity in the similarity-matching stage, and more advanced methods can be computationally prohibitive for real-time, low-latency applications. Inspired by the fly olfactory circuit, we propose Fly-CL, a bio-inspired framework compatible with a wide range of pretrained backbones. Fly-CL substantially reduces training time while achieving performance comparable to or exceeding that of current state-of-the-art methods. We theoretically show how Fly-CL progressively resolves multicollinearity, enabling more effective similarity matching with low time complexity. Extensive simulation experiments across diverse network architectures and data regimes validate Fly-CL's effectiveness in addressing this challenge through a biologically inspired design. Code is available at https://github.com/gfyddha/Fly-CL.

Figures (8)

• Figure 1: Schematic of the Fly Olfactory Circuit. Odors are first detected and pre-processed by olfactory receptor neurons (ORNs) in the antennal lobe, where feature extraction and normalization take place, before being transmitted to projection neurons (PNs). Expansion coding occurs as signals move from PNs to Kenyon cells (KCs), with an expansion ratio of approximately $40$. Each KC connects to a fixed number of PNs (about $6$). Lateral inhibition, mediated by an anterior paired lateral (APL) neuron, suppresses most weakly activated KCs, exemplifying a winner-take-all strategy. Finally, signals from KCs to mushroom body output neurons (MBONs) involve a dense down-projection that reduces dimensionality to select specific actions.
• Figure 2: Schematic of the Fly-CL Framework.Left: Our framework extracts image embeddings using a frozen pre-trained model, projects them into a higher-dimensional space via a fixed sparse random projection, and filters them through a top-$k$ operation (PNs $\rightarrow$ KCs). Then we utilize a learned down-projection for similarity matching during inference time (KCs $\rightarrow$ MBONs). Right: During the training phase, the parameter $\bm{C}_t$ is learned via a streaming ridge classification scheme.
• Figure 3: Pearson Correlation Coefficients of Prototypes at Different Decorrelation Stages in Fly-CL. Heatmaps display Pearson correlation coefficients for $10$ randomly selected class prototypes at each stage of our pipeline (consistent across visualizations).
• Figure 4: Accuracy Curves from Ablation Studies on Three Datasets. We report average accuracy ($A_t$) for each stage. w/o refers to the removal of the specific component.
• Figure 5: Sensitivity Analysis for Expanded dim $m$, Weight Sparsity $p$, and Activation Sparsity $k$ on CUB-200-2011. We report average accuracy in last task ($A_T$) and overall accuracy ($\bar{A}$). The dots denote the default values we use across experiments.

Theorems & Definitions (9)

• Theorem 4.1
• Theorem 4.2
• Theorem B.1
• proof
• Theorem B.4
• proof
• Theorem B.5
• proof
• Theorem B.6