A Brief Review of Hypernetworks in Deep Learning

TL;DR

This paper addresses the lack of a comprehensive review of hypernetworks, which generate the weights of a target network to confer data adaptivity, dynamic architectures, and parameter efficiency. It introduces a five-criterion taxonomy (inputs, outputs, input variability, output variability, and hypernet architecture) and provides an illustrative training example to ground the discussion. The survey covers applications across continual and federated learning, causal inference, domain adaptation, RL, NLP, computer vision, AutoML/NAS, and uncertainty quantification, highlighting how task- and data-conditioned hypernets enable soft weight sharing and flexible parameter generation. It also discusses key challenges—initialization, scalability, numerical stability, and theoretical understanding—and outlines directions to advance the field, aiming to spur practical adoption of hypernetworks in diverse domains.

Abstract

Hypernetworks, or hypernets for short, are neural networks that generate weights for another neural network, known as the target network. They have emerged as a powerful deep learning technique that allows for greater flexibility, adaptability, dynamism, faster training, information sharing, and model compression. Hypernets have shown promising results in a variety of deep learning problems, including continual learning, causal inference, transfer learning, weight pruning, uncertainty quantification, zero-shot learning, natural language processing, and reinforcement learning. Despite their success across different problem settings, there is currently no comprehensive review available to inform researchers about the latest developments and to assist in utilizing hypernets. To fill this gap, we review the progress in hypernets. We present an illustrative example of training deep neural networks using hypernets and propose categorizing hypernets based on five design criteria: inputs, outputs, variability of inputs and outputs, and the architecture of hypernets. We also review applications of hypernets across different deep learning problem settings, followed by a discussion of general scenarios where hypernets can be effectively employed. Finally, we discuss the challenges and future directions that remain underexplored in the field of hypernets. We believe that hypernetworks have the potential to revolutionize the field of deep learning. They offer a new way to design and train neural networks, and they have the potential to improve the performance of deep learning models on a variety of tasks. Through this review, we aim to inspire further advancements in deep learning through hypernetworks.

Figures (2)

• Figure 1: An overview of the architectures and gradient flows for a standard DNN $\mathcal{F}(X; \Theta)$ and the same DNN implemented with hypernets, referred to as HyperDNN $\mathcal{F}(X; \Theta)=\mathcal{F}(X; \mathcal{H}(C; \Phi))$. For the DNN, gradients flow through the DNN, and DNN weights $\Theta$ are learned during training. For the HyperDNN, gradients flow through the hypernet, and hypernet weights $\Phi$ are learned during training to produce DNN weights $\Theta$ as outputs.
• Figure 2: Proposed categorization of hypernets based on five design criteria.