Transferring Core Knowledge via Learngenes

TL;DR

This work introduces Genetic Transfer Learning (GTL), a framework that evolves neural networks to extract learngenes—core neural circuits encoded as channel subsets within convolutional kernels. By simulating natural evolution (training populations, tournaments, mutations, and inheritance), GTL condenses essential knowledge into about 20% of parameters, which significantly enhances accuracy on few-shot tasks (e.g., CIFAR-FS, miniImageNet) and enables fast adaptation with limited data. The learngenes exhibit instincts-like behavior and demonstrate strong learning ability, scalability, and cross-architecture transfer, outperforming many baselines when transferred to varied network depths, widths, and architectures. The approach provides an efficient, bio-inspired alternative to traditional pre-training by focusing on core knowledge transfer rather than whole-network replication, with practical implications for rapid adaptation in diverse downstream tasks.

Abstract

The pre-training paradigm fine-tunes the models trained on large-scale datasets to downstream tasks with enhanced performance. It transfers all knowledge to downstream tasks without discriminating which part is necessary or unnecessary, which may lead to negative transfer. In comparison, knowledge transfer in nature is much more efficient. When passing genetic information to descendants, ancestors encode only the essential knowledge into genes, which act as the medium. Inspired by that, we adopt a recent concept called ``learngene'' and refine its structures by mimicking the structures of natural genes. We propose the Genetic Transfer Learning (GTL) -- a framework to copy the evolutionary process of organisms into neural networks. GTL trains a population of networks, selects superior learngenes by tournaments, performs learngene mutations, and passes the learngenes to next generations. Finally, we successfully extract the learngenes of VGG11 and ResNet12. We show that the learngenes bring the descendant networks instincts and strong learning ability: with 20% parameters, the learngenes bring 12% and 16% improvements of accuracy on CIFAR-FS and miniImageNet. Besides, the learngenes have the scalability and adaptability on the downstream structure of networks and datasets. Overall, we offer a novel insight that transferring core knowledge via learngenes may be sufficient and efficient for neural networks.

Figures (10)

• Figure 1: (a) Transferring the entire knowledge may be redundant or negative. (b) Leveraging the learngenes to transfer the core knowledge to descendant networks inspired by the genes in nature.
• Figure 2: Structure of the learngenes, which are several complete neural circuits in the unit of channels (colored blue) within kernels.
• Figure 3: The framework of Genetic Transfer Learning (GTL), which are used to condense core knowledge and extract the learngenes.
• Figure 4: The parameter quantity of the learngenes (i.e., blue bars) and the average (with max and min) accuracy on validation classes of networks in population (i.e., red curves) during evolution. Black lines are accuracy (with the number of transferred parameters) of the models trained from scratch and pre-trained on training classes.
• Figure 5: The visualization of core knowledge in the learngenes. All networks have not undergone any learning or fine-tuning.$\quad$$\dag$ ResNet50 from Pytorch Official was pre-trained on ImageNet, which has included these classes.