Effective Sparsity: A Unified Framework via Normalized Entropy and the Effective Number of Nonzeros

Abstract

Classical sparsity promoting methods rely on the l0 norm, which treats all nonzero components as equally significant. In practical inverse problems, however, solutions often exhibit many small amplitude components that have little effect on reconstruction but lead to an overestimation of signal complexity. We address this limitation by shifting the paradigm from discrete cardinality to effective sparsity. Our approach introduces the effective number of nonzeros (ENZ), a unified class of normalized entropy-based regularizers, including Shannon and Renyi forms, that quantifies the concentration of significant coefficients. We show that, unlike the classical l0 norm, the ENZ provides a stable and continuous measure of effective sparsity that is insensitive to negligible perturbations. For noisy linear inverse problems, we establish theoretical guarantees under the Restricted Isometry Property (RIP), proving that ENZ based recovery is unique and stable. We also derive a decomposition showing that the ENZ equals the support cardinality times a distributional efficiency term, thereby linking entropy with l0 regularization. Numerical experiments show that this effective sparsity framework outperforms traditional cardinality based methods in robustness and accuracy.

Figures (9)

• Figure 1: Effective sparsity in natural images. Sorted magnitude decay of wavelet coefficients for a collection of standard test images (including cameraman, peppers, and related benchmarks). Both grayscale images and individual color channels (RGB) are analyzed. Each image is normalized by its maximum pixel value and transformed using a Daubechies-4 (db4) wavelet basis with four decomposition levels. Coefficient magnitudes are sorted in descending order and normalized by their maximum value. Thin curves show individual decay profiles across images and channels. The blue solid curve denotes the mean decay, and the black solid curve denotes the median. Dashed curves indicate the 5%, 25%, 80%, and 95% percentile envelopes. The horizontal axis represents the normalized coefficient index (from largest to smallest), and the vertical axis shows the normalized coefficient magnitude on a logarithmic scale. The rapid decay demonstrates that natural images are not strictly sparse but exhibit pronounced effective sparsity, with most coefficients being nonzero yet negligible in magnitude.
• Figure 2: Effective sparsity in spatial variation of natural images. Sorted magnitude decay of directional total variation (TV) coefficients for the same collection of standard test images as in Figure \ref{['fig.test.image']}. Horizontal ($\mathrm{TV}_x$) and vertical ($\mathrm{TV}_y$) finite differences are computed for each image; for color images, TV is evaluated independently on each RGB channel. All TV magnitudes are normalized by their maximum value and sorted in descending order. Thin curves correspond to individual decay profiles across images, channels, and directions. The blue solid curve denotes the mean decay, and the black solid curve denotes the median. Dashed curves indicate the 5%, 25%, 80%, and 95% percentile envelopes. The horizontal axis represents the normalized coefficient index (from largest to smallest), and the vertical axis shows the normalized TV magnitude on a logarithmic scale. The rapid and nearly identical decay observed in both spatial directions indicates that natural images possess effective sparsity in their spatial gradients: most variations are nonzero but contribute negligibly, while a small subset of dominant transitions captures the essential geometric structure.
• Figure 3: Effective sparsity in textual data (Spectral Decay). Aggregated spectral energy profiles of term-document matrices from the 20 Newsgroups dataset. For each text category, singular values $\sigma_k$ were computed using TF-IDF vectorization and SVD; the curves plot the normalized energy $\sigma_k^2 / \sigma_{\max}^2$. The blue solid line represents the mean decay across all categories, the black line represents the median, and the dashed lines indicate the 5%, 25%, 80%, and 95% percentile envelopes. The steep initial descent demonstrates that textual information within a single topic is low-rank, as meaningful semantic content is concentrated in a low-dimensional subspace (effective sparsity).
• Figure 4: Visualization of level sets in $\mathbb{R}^3$. The ENZ ball provides an unbounded, magnitude-aware geometry that directly generalizes the $\ell_0$ norm while avoiding the shrinkage bias inherent in bounded $\ell_p$ balls.
• Figure 5: Structural decomposition of the ENZ. The metric factors the discrete support cardinality into a product with a continuous distributional efficiency term, bridging the gap between combinatorial and informational sparsity.

Theorems & Definitions (6)

• Proposition 1
• Theorem 2: Stability of the Effective Nonzeros under Noise
• Theorem 3: Decomposition of Effective Nonzeros
• Remark 4: ENZ as a Multiplicative Discount
• Remark 5: Hartley Entropy and the $\ell_0$ Limit
• Theorem 6: Generalized Rényi Decomposition