Learning to Generate Parameters of ConvNets for Unseen Image Data

TL;DR

PudNet reframes ConvNet training as a parameter-prediction task by learning a dataset-to-parameter hyper-mapping. It uses dataset sketches, an adaptive hyper-recurrent unit to capture cross-layer parameter dependencies, and per-layer weight generators to synthesize ConvNet weights for unseen data in a single forward pass, trained via meta-learning with auxiliary objectives. Empirical results show PudNet achieves competitive top-1 accuracy on intra- and inter-dataset tasks (including ImageNet-1K) with orders of magnitude fewer GPU seconds than conventional training, and extends to image denoising with similar efficiency gains. This approach offers scalable, rapid adaptation of networks to new domains and hints at broader applicability to adapters in large language and vision-language models.

Abstract

Typical Convolutional Neural Networks (ConvNets) depend heavily on large amounts of image data and resort to an iterative optimization algorithm (e.g., SGD or Adam) to learn network parameters, which makes training very time- and resource-intensive. In this paper, we propose a new training paradigm and formulate the parameter learning of ConvNets into a prediction task: given a ConvNet architecture, we observe there exist correlations between image datasets and their corresponding optimal network parameters, and explore if we can learn a hyper-mapping between them to capture the relations, such that we can directly predict the parameters of the network for an image dataset never seen during the training phase. To do this, we put forward a new hypernetwork based model, called PudNet, which intends to learn a mapping between datasets and their corresponding network parameters, and then predicts parameters for unseen data with only a single forward propagation. Moreover, our model benefits from a series of adaptive hyper recurrent units sharing weights to capture the dependencies of parameters among different network layers. Extensive experiments demonstrate that our proposed method achieves good efficacy for unseen image datasets on two kinds of settings: Intra-dataset prediction and Inter-dataset prediction. Our PudNet can also well scale up to large-scale datasets, e.g., ImageNet-1K. It takes 8967 GPU seconds to train ResNet-18 on the ImageNet-1K using GC from scratch and obtain a top-5 accuracy of 44.65%. However, our PudNet costs only 3.89 GPU seconds to predict the network parameters of ResNet-18 achieving comparable performance (44.92%), more than 2,300 times faster than the traditional training paradigm.

Figures (8)

• Figure 1: Correlation coefficients between training datasets and the network parameters on the Fashion-MNIST xiao2017fashion, CIFAR-100 krizhevsky2009learning, Mini-ImageNet vinyals2016matching datasets, respectively. 'learned’ depicts correlations between training datasets and the corresponding optimal network parameters. 'random’ denotes correlations between training datasets and the network parameters selected randomly from 1000 groups.
• Figure 2: Overview of our PudNet. PudNet compresses each image dataset into a sketch with a fixed size and utilizes the designed hypernetwork to generate parameters of a target ConvNet using this sketch. Specifically, PudNet exploits the adaptive hyper-recurrent units (AHRU) to process the dataset sketch, capturing the dependencies of parameters among different network layers for predicting the parameter representations. Subsequently, the weight generators are employed to produce the dataset-specific parameters of the target ConvNet.
• Figure 3: An illustration of capturing context relations via AHRU.
• Figure 4: Architecture of the weight generator.
• Figure 5: Effect of different groups of datasets in terms of the target network ResNet-18.