Physics-Inspired Binary Neural Networks: Interpretable Compression with Theoretical Guarantees

TL;DR

The paper introduces Physics-Inspired Binary Neural Networks (PIBiNNs) that fuse one-bit quantization with physics-driven sparsity embedded in algorithm-unrolled solvers to compress networks dramatically while preserving essential operator geometry. It advances with two-stage quantization-aware training using a single global scale $\λ$, constructs sparse, block-diagonal weight matrices aligned with problem structure, and provides convergence guarantees for one-bit unrolling along with a local-Rademacher-based generalization bound. Empirically, PIBiNN achieves extreme compression (up to ≈$99.9\%$ reduction in parameters) with competitive accuracy on synthetic tasks and real datasets (BSD500, EEGdenoiseNet), and scales to large block-sparse configurations, significantly reducing memory and metadata overhead. The work offers a principled pathway to memory-efficient, physics-aware neural architectures, and highlights practical considerations for deployment and future system-level optimizations.

Abstract

Why rely on dense neural networks and then blindly sparsify them when prior knowledge about the problem structure is already available? Many inverse problems admit algorithm-unrolled networks that naturally encode physics and sparsity. In this work, we propose a Physics-Inspired Binary Neural Network (PIBiNN) that combines two key components: (i) data-driven one-bit quantization with a single global scale, and (ii) problem-driven sparsity predefined by physics and requiring no updates during training. This design yields compression rates below one bit per weight by exploiting structural zeros, while preserving essential operator geometry. Unlike ternary or pruning-based schemes, our approach avoids ad-hoc sparsification, reduces metadata overhead, and aligns directly with the underlying task. Experiments suggest that PIBiNN achieves advantages in both memory efficiency and generalization compared to competitive baselines such as ternary and channel-wise quantization.

Figures (7)

• Figure 1: General DNNs vs DUNs. DUNs appear to be an excellent tool in large-scale machine learning applications due to their inherent physics-inspired sparsity.
• Figure 2: The impact of the number of layers on "Train" and "Test" NMSE in one-bit DUN.
• Figure 3: Comparison of per-layer error across both training and test stages for three models: (a) DUN with high-resolution weights, (b) one-bit DUN utilizing lazy projection, and (c) one-bit DUN employing $\ell_1$-regularization.
• Figure 4: Spectral plots of the HT update process for (a) DUN with $K=5,\delta=1$, (b) one-bit DUN with $K=5,\delta=1,\lambda=0.02$, (c) DUN with $K=10,\delta=1$, and (d) one-bit DUN with $K=10,\delta=1,\lambda=0.02$. These results illustrate that the training process naturally learns binary weights from the set defined in \ref{['ht1']}.
• Figure 5: Spectral plots of the HT update process for the one-bit DUN with (a) $K=5,\delta=1$, (b) $K=5,\delta=0.95$, (c) $K=5,\delta=0.9$, (d) $K=5,\delta=0.85$, (e) $K=10,\delta=1$, (f) $K=10,\delta=0.95$, (g) $K=10,\delta=0.9$, and (h) $K=10,\delta=0.85$. All results are obtained with a fixed scale value of $\lambda=0.02$.

Theorems & Definitions (6)

• Definition 1
• Lemma 1
• Definition 2
• Theorem 1
• Lemma 2
• proof