Sensing Theorems for Unsupervised Learning in Linear Inverse Problems

Abstract

Solving an ill-posed linear inverse problem requires knowledge about the underlying signal model. In many applications, this model is a priori unknown and has to be learned from data. However, it is impossible to learn the model using observations obtained via a single incomplete measurement operator, as there is no information about the signal model in the nullspace of the operator, resulting in a chicken-and-egg problem: to learn the model we need reconstructed signals, but to reconstruct the signals we need to know the model. Two ways to overcome this limitation are using multiple measurement operators or assuming that the signal model is invariant to a certain group action. In this paper, we present necessary and sufficient sensing conditions for learning the signal model from measurement data alone which only depend on the dimension of the model and the number of operators or properties of the group action that the model is invariant to. As our results are agnostic of the learning algorithm, they shed light into the fundamental limitations of learning from incomplete data and have implications in a wide range set of practical algorithms, such as dictionary learning, matrix completion and deep neural networks.

Figures (14)

• Figure 1: Model identification and signal recovery regimes as a function of the number of partial observations $m$ per signal, model dimension $k$, ambient dimension $n$ and number of measurement operators $|\mathcal{G}|$ (when it is possible to access multiple independent operators) or maximum multiplicity of an invariant subspace $\max_j c_j/s_j$ (when the signal is group invariant or the operators are related via a group action).
• Figure 2: Learning a one-dimensional model from incomplete measurement data. Each column shows a different sample from the signal set. The sensing mask is shown in red.
• Figure 3: Examples of low-dimensional signal sets. a) the intersection of a union of $k$-dimensional subspaces with the unit sphere has box-counting dimension $k-1$. b) the box-counting dimension of a generative model with a Lipschitz continuous decoder equals the dimension of the latent space.
• Figure 4: Toy example of a 1-dimensional subspace embedded in $\mathbb{R}^{3}$. If we only observe the projection of the signal set into the plane $x(3)=0$, then there are infinite possible lines that are consistent with the measurements. Adding the projection into the $x(1)=0$ plane, allows us to uniquely identify the signal model.
• Figure 5: Learning a one-dimensional shift invariant model from masked observations. Each column shows a different observation from the signal set. The sensing mask is shown in red. Following \ref{['eq:virtual']}, the samples can be reinterpreted as having measured another valid signal $x_{i'}$ through a different operator $A_{g_i}$.

Theorems & Definitions (33)

• Proposition 1
• proof
• Example 3.1: One-dimensional toy model
• Proposition 2
• proof
• Example 3.2
• Proposition 3: Theorem 1 in chen2021equivariant
• proof
• Theorem 4
• Example 4.1: Shift invariant toy model