A Survey of Weight Space Learning: Understanding, Representation, and Generation

TL;DR

This survey provides the first unified taxonomy of Weight Space Learning, which categorizes existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetry of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models.

Abstract

Neural network weights are typically viewed as the end product of training, while most deep learning research focuses on data, features, and architectures. However, recent advances show that the set of all possible weight values (weight space) itself contains rich structure: pretrained models form organized distributions, exhibit symmetries, and can be embedded, compared, or even generated. Understanding such structures has tremendous impact on how neural networks are analyzed and compared, and on how knowledge is transferred across models, beyond individual training instances. This emerging research direction, which we refer to as Weight Space Learning (WSL), treats neural weights as a meaningful domain for analysis and modeling. This survey provides the first unified taxonomy of WSL. We categorize existing methods into three core dimensions: Weight Space Understanding (WSU), which studies the geometry and symmetries of weights; Weight Space Representation (WSR), which learns embeddings over model weights; and Weight Space Generation (WSG), which synthesizes new weights through hypernetworks or generative models. We further show how these developments enable practical applications, including model retrieval, continual and federated learning, neural architecture search, and data-free reconstruction. By consolidating fragmented progress under a coherent framework, this survey highlights weight space as a learnable, structured domain with growing impact across model analysis, transferring, and weight generation. We release an accompanying resource at https://github.com/Zehong-Wang/Awesome-Weight-Space-Learning.

Figures (6)

• Figure 1: Overview of the survey. We organize weight space learning into three dimensions: understanding weight-space properties, learning compact representations, generating weights via auxiliary models. In addition, it consolidates applications to related domains and benchmarks, paving the way for future research.
• Figure 2: Conceptual overview of weight space learning. (a) Weight space understanding analyzes symmetries in weight space and introduces quotient spaces to reduce redundancy and clarify functional equivalence. (b) Weight space representation learns compact embeddings of neural weights, discriminating symmetry in weight space well. (c) Weight space generation synthesizes network weights, supporting efficient model creation and adaptation.
• Figure 3: Taxonomy of foundational components and representative works for weight space learning.
• Figure 4: Weight space symmetry and its induced functional manifestations. (a) illustrates the invariance under group action: applying a transformation $\rho_{in}$ to weights $\theta$ in weight space does not alter the induced function in quotient space. (b) illustrates the equivariance under group action: transforming weights $\theta$ by $\rho_{in}$ in weight space induces a corresponding transformation $\rho_{out}$ in quotient space.
• Figure 5: Weight Space Representation Methods: Overview and Comparison. (a) Model-based approaches. All take raw weight space $\theta$ as input but differ in how they handle symmetry. Symmetry-agnostic methods learn representations directly from weight distributions without explicitly encoding neuron permutation symmetry. Symmetry-aware methods incorporate dedicated equivariant modules to respect permutation invariance. Graph-based symmetry approaches leverage GNNs to automatically capture structural symmetries among neurons. (b) Model-free approaches. Probe-based behavioral learning infers representations from functional responses to designed probes, enabling embeddings that encode behavioral properties and functional symmetries of the underlying network.