Cross-Domain Few-Shot Learning by Representation Fusion

TL;DR

This paper tackles cross-domain few-shot learning under domain shifts by proposing representation fusion, which merges information from multiple network layers to capture both low- and high-level features. It introduces CHEF, an ensemble of Hebbian learners that operate on different backbone layers and yield logits summed for final prediction, avoiding backward passes through the backbone. The approach achieves state-of-the-art results across large cross-domain benchmarks and demonstrates strong performance on standard datasets (miniImagenet, tieredImagenet) as well as real-world drug discovery toxicity tasks. The work highlights the importance of multi-level feature fusion for robust transfer under domain shift and offers a fast, parameter-efficient alternative to backpropagation-based adaptation.

Abstract

In order to quickly adapt to new data, few-shot learning aims at learning from few examples, often by using already acquired knowledge. The new data often differs from the previously seen data due to a domain shift, that is, a change of the input-target distribution. While several methods perform well on small domain shifts like new target classes with similar inputs, larger domain shifts are still challenging. Large domain shifts may result in high-level concepts that are not shared between the original and the new domain, whereas low-level concepts like edges in images might still be shared and useful. For cross-domain few-shot learning, we suggest representation fusion to unify different abstraction levels of a deep neural network into one representation. We propose Cross-domain Hebbian Ensemble Few-shot learning (CHEF), which achieves representation fusion by an ensemble of Hebbian learners acting on different layers of a deep neural network. Ablation studies show that representation fusion is a decisive factor to boost cross-domain few-shot learning. On the few-shot datasets miniImagenet and tieredImagenet with small domain shifts, CHEF is competitive with state-of-the-art methods. On cross-domain few-shot benchmark challenges with larger domain shifts, CHEF establishes novel state-of-the-art results in all categories. We further apply CHEF on a real-world cross-domain application in drug discovery. We consider a domain shift from bioactive molecules to environmental chemicals and drugs with twelve associated toxicity prediction tasks. On these tasks, that are highly relevant for computational drug discovery, CHEF significantly outperforms all its competitors. Github: https://github.com/ml-jku/chef

Figures (5)

• Figure 1: Working principle of CHEF. An ensemble of Hebbian learners is applied to the upper layers of a trained neural network. Distilling information from different layers of abstraction is called representation fusion. Each Hebbian learner is iteratively optimized and the results are combined. CHEF does not require backpropagation of error signals through the entire backbone network, only the parameters of the individual learners need adjustment. Hebbian learners can easily be exchanged with other supervised techniques.
• Figure 2: Ablation study of 5-shot and 50-shot top-1 accuracies (along with $95\%$ confidence intervals) of different residual blocks and the output layer of a PyTorch Imagenet-pretrained ResNet-18 and the ensemble result (orange, "ens") on the four different datasets of the cross-domain few-shot learning benchmark. For comparison, also the ResNet-10 ensemble results (green) are included. The ResNet-18 is pretrained on the whole Imagenet dataset.
• Figure 3: 20-shot top-1 accuracies (along with $95\%$ confidence intervals) of different residual blocks and the output layer of an Imagenet-pretrained ResNet-18 and the ensemble result (orange, "ens") on the four different datasets of the cross-domain few-shot learning benchmark. For comparison, also the ResNet-10 ensemble results (green) are included.
• Figure 4: Ablation study of the Conv-4 architecture on the miniImagenet and tieredImagenet datasets for 1-shot and 5-shot. The plots show the individual performances of Hebbian learners acting on single layers and their ensemble performance along with 95% confidence intervals. The labels on the $x$-axis indicate how far the respective layer is from the output layer.
• Figure 5: Ablation study of the ResNet-12 architecture on the miniImagenet and tieredImagenet datasets for 1-shot and 5-shot. The plots show the individual performances of Hebbian learners acting on single layers and their ensemble performance along with 95% confidence intervals. The labels on the $x$-axis indicate how far the respective layer is from the output layer.