Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How to Train Your Metamorphic Deep Neural Network

Thomas Sommariva, Simone Calderara, Angelo Porrello

TL;DR

This work expands Neural Metamorphosis (NeuMeta) from metamorphosing only the final blocks to full-network metamorphosis by introducing a block-wise, incremental training strategy that leverages Implicit Neural Representations (INRs) to model a continuous weight manifold. Key innovations include INR pre-initialization, learnable residual scaling to replace Batch Normalization, gradient accumulation over multiple configurations, and disentangled INRs for weights and biases, enabling stable training across varying compression ratios denoted by the factor $\\gamma$. Empirical results on ResNet-56 with CIFAR-100 show substantial accuracy gains over NeuMeta and pruning baselines, with near-baseline performance under aggressive compression and clear evidence of improved scalability with network width. The approach demonstrates the feasibility of deploying dynamically adaptable metamorphic networks, potentially enabling efficient, on-device inference across heterogeneous environments. The work also discusses generalization and trade-offs between in-distribution performance and flexibility to unseen configurations, outlining future work to extend metamorphism to other architectures like QRNNs and ViT.

Abstract

Neural Metamorphosis (NeuMeta) is a recent paradigm for generating neural networks of varying width and depth. Based on Implicit Neural Representation (INR), NeuMeta learns a continuous weight manifold, enabling the direct generation of compressed models, including those with configurations not seen during training. While promising, the original formulation of NeuMeta proves effective only for the final layers of the undelying model, limiting its broader applicability. In this work, we propose a training algorithm that extends the capabilities of NeuMeta to enable full-network metamorphosis with minimal accuracy degradation. Our approach follows a structured recipe comprising block-wise incremental training, INR initialization, and strategies for replacing batch normalization. The resulting metamorphic networks maintain competitive accuracy across a wide range of compression ratios, offering a scalable solution for adaptable and efficient deployment of deep models. The code is available at: https://github.com/TSommariva/HTTY_NeuMeta.

How to Train Your Metamorphic Deep Neural Network

TL;DR

This work expands Neural Metamorphosis (NeuMeta) from metamorphosing only the final blocks to full-network metamorphosis by introducing a block-wise, incremental training strategy that leverages Implicit Neural Representations (INRs) to model a continuous weight manifold. Key innovations include INR pre-initialization, learnable residual scaling to replace Batch Normalization, gradient accumulation over multiple configurations, and disentangled INRs for weights and biases, enabling stable training across varying compression ratios denoted by the factor . Empirical results on ResNet-56 with CIFAR-100 show substantial accuracy gains over NeuMeta and pruning baselines, with near-baseline performance under aggressive compression and clear evidence of improved scalability with network width. The approach demonstrates the feasibility of deploying dynamically adaptable metamorphic networks, potentially enabling efficient, on-device inference across heterogeneous environments. The work also discusses generalization and trade-offs between in-distribution performance and flexibility to unseen configurations, outlining future work to extend metamorphism to other architectures like QRNNs and ViT.

Abstract

Neural Metamorphosis (NeuMeta) is a recent paradigm for generating neural networks of varying width and depth. Based on Implicit Neural Representation (INR), NeuMeta learns a continuous weight manifold, enabling the direct generation of compressed models, including those with configurations not seen during training. While promising, the original formulation of NeuMeta proves effective only for the final layers of the undelying model, limiting its broader applicability. In this work, we propose a training algorithm that extends the capabilities of NeuMeta to enable full-network metamorphosis with minimal accuracy degradation. Our approach follows a structured recipe comprising block-wise incremental training, INR initialization, and strategies for replacing batch normalization. The resulting metamorphic networks maintain competitive accuracy across a wide range of compression ratios, offering a scalable solution for adaptable and efficient deployment of deep models. The code is available at: https://github.com/TSommariva/HTTY_NeuMeta.
Paper Structure (17 sections, 3 equations, 7 figures, 2 tables, 2 algorithms)

This paper contains 17 sections, 3 equations, 7 figures, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction and Related Works
  2. Background
  3. Neural Metamorphosis
  4. Training Procedure of NeuMeta
  5. Method
  6. Incremental Training
  7. Substitution of Batch Normalization Layers
  8. INR Pre-Initialization
  9. Gradient Accumulation
  10. Disentangling Weights and Biases
  11. Experiments
  12. Implementation Details
  13. Baselines
  14. Results and Ablative Studies
  15. Generalization and Fine-Tuning Analysis
  16. ...and 2 more sections

Figures (7)

  • Figure 1: 2D visualization of weight manifold smoothing via permutation alignment
  • Figure 2: Ablation study on the last block of the third layer of ResNet56. $^\dagger \gamma=0.75$ wasn’t applied in training.
  • Figure 3: Accuracy across varying numbers of metamorphic blocks for different $\gamma$.
  • Figure 4: Accuracy comparison with structural pruning techniques.
  • Figure 5: Accuracy and loss at different $\gamma$
  • ...and 2 more figures