Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Intermediate Neural Collapse with Simplex ETFs for Efficient Deep Neural Networks

Emily Liu

TL;DR

This work investigates neural collapse and its link to simplex equiangular tight frames (ETFs) as a mechanism to enable memory-efficient training. It introduces Adaptive-ETF, which locks intermediate layers to ETF structure once NCC separability is achieved, and ETF-Transformer, which extends ETF constraints to transformer feedforward blocks. Through experiments on multilayer perceptrons and Vision Transformers, the authors show comparable accuracy to baselines while significantly reducing learnable parameters, and observe nuanced NCC dynamics under heavy ETF constraining. The findings suggest practical training benefits and offer insights into the role of NC2/NC3 in both fully connected and transformer architectures, with potential cross-modal applicability and broader implications for regularization and interpretability.

Abstract

Neural collapse is a phenomenon observed during the terminal phase of neural network training, characterized by the convergence of network activations, class means, and linear classifier weights to a simplex equiangular tight frame (ETF), a configuration of vectors that maximizes mutual distance within a subspace. This phenomenon has been linked to improved interpretability, robustness, and generalization in neural networks. However, its potential to guide neural network training and regularization remains underexplored. Previous research has demonstrated that constraining the final layer of a neural network to a simplex ETF can reduce the number of trainable parameters without sacrificing model accuracy. Furthermore, deep fully connected networks exhibit neural collapse not only in the final layer but across all layers beyond a specific effective depth. Using these insights, we propose two novel training approaches: Adaptive-ETF, a generalized framework that enforces simplex ETF constraints on all layers beyond the effective depth, and ETF-Transformer, which applies simplex ETF constraints to the feedforward layers within transformer blocks. We show that these approaches achieve training and testing performance comparable to those of their baseline counterparts while significantly reducing the number of learnable parameters.

Leveraging Intermediate Neural Collapse with Simplex ETFs for Efficient Deep Neural Networks

TL;DR

This work investigates neural collapse and its link to simplex equiangular tight frames (ETFs) as a mechanism to enable memory-efficient training. It introduces Adaptive-ETF, which locks intermediate layers to ETF structure once NCC separability is achieved, and ETF-Transformer, which extends ETF constraints to transformer feedforward blocks. Through experiments on multilayer perceptrons and Vision Transformers, the authors show comparable accuracy to baselines while significantly reducing learnable parameters, and observe nuanced NCC dynamics under heavy ETF constraining. The findings suggest practical training benefits and offer insights into the role of NC2/NC3 in both fully connected and transformer architectures, with potential cross-modal applicability and broader implications for regularization and interpretability.

Abstract

Neural collapse is a phenomenon observed during the terminal phase of neural network training, characterized by the convergence of network activations, class means, and linear classifier weights to a simplex equiangular tight frame (ETF), a configuration of vectors that maximizes mutual distance within a subspace. This phenomenon has been linked to improved interpretability, robustness, and generalization in neural networks. However, its potential to guide neural network training and regularization remains underexplored. Previous research has demonstrated that constraining the final layer of a neural network to a simplex ETF can reduce the number of trainable parameters without sacrificing model accuracy. Furthermore, deep fully connected networks exhibit neural collapse not only in the final layer but across all layers beyond a specific effective depth. Using these insights, we propose two novel training approaches: Adaptive-ETF, a generalized framework that enforces simplex ETF constraints on all layers beyond the effective depth, and ETF-Transformer, which applies simplex ETF constraints to the feedforward layers within transformer blocks. We show that these approaches achieve training and testing performance comparable to those of their baseline counterparts while significantly reducing the number of learnable parameters.

Paper Structure

This paper contains 17 sections, 6 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Contribution
  3. Background: Neural Collapse
  4. (NC1) Variability collapse
  5. (NC2) Convergence to Simplex Equiangular Tight Frame (ETF)
  6. (NC3) Convergence to self-duality
  7. (NC4) Simplfication to nearest class center (NCC)
  8. Related work: Simplex ETF and Effective Depth
  9. Effective Depth
  10. Simplex ETF for neural network training
  11. Combining Effective Depth and Simplex ETF
  12. Experiments
  13. Training
  14. Intermediate Simplex ETF in Multilayered Perceptrons
  15. New training scheme: Adaptive ETF
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Training progress on baseline MLP compared to replacing the final layer, final two layers, and final three layers of the neural network with simplex ETFs.
  • Figure 2: Training progress on baseline network compared to replacing feedforward layers with simplex ETFs beyond the effective depth.
  • Figure 3: Comparison of training progress on vision transformers with 1 block, 2 blocks, 3 blocks, and 4 blocks.
  • Figure 4: NCC for FC layers in transformer ETF architectures.