Sparsity vs. Statistical Independence in Adaptive Signal Representations: A Case Study of the Spike Process

TL;DR

The paper analyzes how sparsity and statistical independence guide basis selection for representing stochastic data, using the spike process as a tractable testbed. It defines the Best Sparsifying Basis (BSB) via the expected ${\ell^p}$ norm and the Least Statistically-Dependent Basis (LSDB) via coordinate-wise entropy, then derives exact LSDB/BSB results across dictionaries: the standard basis often dominates for orthonormal bases when $n\ge 5$, the Haar–Walsh dictionary favors the Walsh basis only in small dimensions, and even within ${\rm O}(n)$ a Householder reflection can yield the same independence level as the standard basis; however, no linear transform in ${\rm GL}(n,{\bf R})$ achieves true independence for $n>2$. These findings reveal a fundamental separation between sparsity and independence, highlight the limitations of linear transforms for complete independence, and motivate considering nonlinear representations for more complex signals. The results have implications for understanding neural coding and for designing efficient adaptive representations in data processing and computational neuroscience.

Abstract

Finding a basis/coordinate system that can efficiently represent an input data stream by viewing them as realizations of a stochastic process is of tremendous importance in many fields including data compression and computational neuroscience. Two popular measures of such efficiency of a basis are sparsity (measured by the expected $\ell^p$ norm, $0 < p \leq 1$) and statistical independence (measured by the mutual information). Gaining deeper understanding of their intricate relationship, however, remains elusive. Therefore, we chose to study a simple synthetic stochastic process called the spike process, which puts a unit impulse at a random location in an $n$-dimensional vector for each realization. For this process, we obtained the following results: 1) The standard basis is the best both in terms of sparsity and statistical independence if $n \geq 5$ and the search of basis is restricted within all possible orthonormal bases in $R^n$; 2) If we extend our basis search in all possible invertible linear transformations in $R^n$, then the best basis in statistical independence differs from the one in sparsity; 3) In either of the above, the best basis in statistical independence is not unique, and there even exist those which make the inputs completely dense; 4) There is no linear invertible transformation that achieves the true statistical independence for $n > 2$.

Figures (8)

• Figure 1: Sparsity and statistical independence prefer the different coordinates.
• Figure 2: Ten realizations of the spike process ($n=256$).
• Figure 3: A table of dictionary coefficients are organized as the binary tree structured table.
• Figure 4: A plot of $f:x \rightarrow - \left[ x \log{x} + (1-x)\log(1-x) \right]$.
• Figure 5: A plot of $g:x \rightarrow - \left[ x \log{\frac{x}{2}} + (1-x)\log(1-x) \right]$.