Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization

Ibne Farabi Shihab, Sanjeda Akter, Anuj Sharma

TL;DR

The paper presents a differentiable surrogate for range-partition entropy to directly regulate representation complexity and bound downstream runtime, enabling end-to-end optimization of efficiency. The method, entropy regularization, acts as a plug-in loss term that complements existing optimizations and can induce structured sparsity, including block-sparse attention in transformers, with theoretical guarantees in geometric regimes. Empirically, it achieves 4–5× speedups in convex-hull and triangulation tasks with <0.2% error, and boosts throughput on ViT-Base/ImageNet (1.60× standalone, up to 2.07× with FlashAttention) and several open LMs at high sparsity with modest accuracy penalties, while also improving robustness (IoU, CIFAR-100-C). The approach is strongest for geometry and vision tasks, with more modest gains in large-language-model settings, suggesting a principled pathway to joint efficiency and robustness via complexity-aware representation learning.

Abstract

We introduce the first differentiable approximation of range-partition entropy, a complexity measure from computational geometry that directly bounds algorithmic runtime. Unlike architectural modifications, our method is a complementary regularizer that provides orthogonal efficiency gains when combined with existing optimizations. We establish theoretical guarantees in computational geometry, achieving 4--5$\times$ provable speedups on convex hull and triangulation with $<$0.2\% error. On ImageNet-1K with ViT-Base, entropy regularization achieves 80.1\% top-1 accuracy at 80\% sparsity (1.60$\times$ standalone speedup), and when combined with FlashAttention yields 2.07$\times$ speedup versus 1.63$\times$ for FlashAttention alone. On large language models (LLaMA-2 7B, Mistral-7B, Phi-2), we achieve 1.48--1.60$\times$ inference speedups at 70--75\% sparsity with minimal quality degradation (ROUGE-L drops of 0.3--0.4 points, perplexity increase of 0.9). Unlike prior regularization methods that target output distributions, we directly minimize representation complexity, yielding both efficiency gains and improved robustness through semantically structured sparsity patterns (IoU 0.73 vs 0.41 for magnitude pruning, CIFAR-100-C mCE 48.7 vs 55.4). Benefits are strongest for geometry and vision transformers, with more modest but measurable gains on LLMs, demonstrating that complexity regularization offers a principled pathway to joint efficiency-robustness optimization.

Differentiable Entropy Regularization: A Complexity-Aware Approach for Neural Optimization

TL;DR

The paper presents a differentiable surrogate for range-partition entropy to directly regulate representation complexity and bound downstream runtime, enabling end-to-end optimization of efficiency. The method, entropy regularization, acts as a plug-in loss term that complements existing optimizations and can induce structured sparsity, including block-sparse attention in transformers, with theoretical guarantees in geometric regimes. Empirically, it achieves 4–5× speedups in convex-hull and triangulation tasks with <0.2% error, and boosts throughput on ViT-Base/ImageNet (1.60× standalone, up to 2.07× with FlashAttention) and several open LMs at high sparsity with modest accuracy penalties, while also improving robustness (IoU, CIFAR-100-C). The approach is strongest for geometry and vision tasks, with more modest gains in large-language-model settings, suggesting a principled pathway to joint efficiency and robustness via complexity-aware representation learning.

Abstract

We introduce the first differentiable approximation of range-partition entropy, a complexity measure from computational geometry that directly bounds algorithmic runtime. Unlike architectural modifications, our method is a complementary regularizer that provides orthogonal efficiency gains when combined with existing optimizations. We establish theoretical guarantees in computational geometry, achieving 4--5 provable speedups on convex hull and triangulation with 0.2\% error. On ImageNet-1K with ViT-Base, entropy regularization achieves 80.1\% top-1 accuracy at 80\% sparsity (1.60 standalone speedup), and when combined with FlashAttention yields 2.07 speedup versus 1.63 for FlashAttention alone. On large language models (LLaMA-2 7B, Mistral-7B, Phi-2), we achieve 1.48--1.60 inference speedups at 70--75\% sparsity with minimal quality degradation (ROUGE-L drops of 0.3--0.4 points, perplexity increase of 0.9). Unlike prior regularization methods that target output distributions, we directly minimize representation complexity, yielding both efficiency gains and improved robustness through semantically structured sparsity patterns (IoU 0.73 vs 0.41 for magnitude pruning, CIFAR-100-C mCE 48.7 vs 55.4). Benefits are strongest for geometry and vision transformers, with more modest but measurable gains on LLMs, demonstrating that complexity regularization offers a principled pathway to joint efficiency-robustness optimization.

Paper Structure

This paper contains 63 sections, 9 theorems, 45 equations, 10 figures, 27 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Experiments
  5. Discussion and Conclusion
  6. Practical Recipe
  7. Comprehensive Related Work
  8. Deployment and Amortization Details
  9. Quantitative Amortization Analysis
  10. Broader Exploratory Experiments
  11. Task: 3D Maxima Set Identification
  12. Task: Comprehensive GLUE Benchmark Evaluation
  13. Task: Scaling to Larger Models
  14. Task: Large-Scale 3D Shape Processing
  15. Scalability: Large Models and Long Sequences
  16. ...and 48 more sections

Key Result

Theorem 1

Let $S\subset\mathbb{R}^d$ be finite and suppose a hard partition is induced by $m^\star$ halfspaces with $\gamma$-margin on $S$. For any $\tau \le \gamma/4$, there exist parameters $\Theta$ with $m\le m^\star$ such that, with probability at least $1-\delta$, where $\varepsilon = ( e^{-\gamma/(4\tau)} + c\sqrt{\frac{d\log m^\star+\log(2K/\delta)}{n}} )\! \log\!\frac{K}{e^{-\gamma/(4\tau)} + c\sqr

Figures (10)

  • Figure 1: Conceptual comparison of differentiable optimization approaches. (a) NeuralSort grover2019neuralsort learns permutations for sorting operations. (b) Sinkhorn Networks cuturi2013sinkhorn learn optimal transport plans between distributions. (c) Our method minimizes structural complexity via range-partition entropy, discovering clustered representations (bottom) versus dispersed distributions (top). The top panel shows high-entropy, scattered points leading to slow algorithmic runtime; the bottom panel shows low-entropy, clustered points enabling fast instance-optimal algorithms. Unlike prior work targeting specific operations, we optimize a general complexity measure that transfers across computational structures.
  • Figure 2: Empirical validation shows that as the empirical margin $\hat{\gamma}(S)$ increases during training, the approximation error $|H_{\text{true}} - H_{\mathrm{soft}}|$ decreases.
  • Figure 3: Learned attention patterns under different regularization schemes.
  • Figure 4: Validation of surrogate entropy approximation.
  • Figure 5: Hyperparameter sensitivity analysis for anchor count and temperature.
  • ...and 5 more figures

Theorems & Definitions (13)

  • Theorem 1: Halfspace-aware consistency, finite-sample version
  • Lemma 1: Bi-Lipschitz Metric Stability
  • Corollary 1: JL-Preserving Entropy
  • Proposition 1: Entropy--tail–mass bound
  • proof : Proof sketch
  • Lemma 2: Attention truncation error
  • proof
  • Theorem 2: Attention complexity vs. partition entropy
  • proof
  • Remark 1: Relation to subquadratic attention
  • ...and 3 more