Graphical model for tensor factorization by sparse sampling

TL;DR

The paper develops a Bayes-optimal framework for tensor factorization from sparse measurements on a random graph, analyzed in a dense-limit regime where $N\gg M$ and $N,M\to\infty$. It derives exact replica-based MMSE characterizations and constructs scalable G-AMP algorithms with corresponding state evolution that agree with the replica predictions across Ising and Gaussian priors and various output channels (additive noise and sign). It reveals phase transitions and hard/easy regimes for inference, and demonstrates the impact of the spreading factor $F$ and model choices on algorithmic convergence. The results offer theoretical insight and practical algorithms for highly incomplete tensor data, with potential applications to recommender systems and high-rank tensor completion, and indicate avenues for extending to multi-species factor models.

Abstract

We consider tensor factorizations based on sparse measurements of the tensor components. The measurements are designed in a way that the underlying graph of interactions is a random graph. The setup will be useful in cases where a substantial amount of data is missing, as in recommendation systems heavily used in social network services. In order to obtain theoretical insights on the setup, we consider statistical inference of the tensor factorization in a high dimensional limit, which we call as dense limit, where the graphs are large and dense but not fully connected. We build message-passing algorithms and test them in a Bayes optimal teacher-student setting. We also develop a replica theory, which becomes exact in the dense limit,to examine the performance of statistical inference.

Figures (26)

• Figure 1: Graphical representations of our model. a) The black-squares $\blacksquare$ represent the function nodes to each of which a $p$-plet $(i_{1},i_{2},\ldots,i_{p})$ is assigned. Each function node has $p$ arms. b) The gray circles represent the variable nodes $(i=1,2,\ldots,N)$ to each of which a $M$-component vectorial variable ${\bm x}_{i}$ (arrow) is assigned. Each of the variable nodes has $c=\alpha M$ arms. Equivalently, we may also represent a variable node as a collection of sub-variable nodes (green circles) to each of which a component of the vector $x_{i\mu}$ is assigned. c) a factor graph is created by joining the variable and function nodes.
• Figure 2: Some representative diagrams which appear in the cumulant expansion.
• Figure 3: Left: Order parameter $m=q$ in the Bayes optimal case for Ising prior, additive Gaussian noise and $p=2$. Dashed lines indicates a metastable magnetized state associated to a first-order transition. For $\alpha=1.6$ three vertical lines are shown: the central one $\lambda_c$ is the thermodynamic first-order transition where the difference in free energy between the two branches of solutions changes sign; the leftmost one is the spinodal point $\lambda_{\rm d}(\alpha)$ for the high-magnetization solution (corresponding to the red line in the right panel); the rightmost one is the spinodal point for the low-magnetization solution (green line in the right panel). Right: phase diagram in the $\alpha-\lambda$ plane. The black solid line is the spinodal line (stability limit) of the paramagnetic (PM) solution $\lambda^{*}(\alpha)$; for $\alpha<1$ the PM solution is stable $\forall \lambda$. Region I: coexistence of PM and a high-magnetization solution, separated by a first-order transition (not shown for clarity). Region II: coexistence of low and high-magnetization phases, separated by a first-order transition (not shown). Region III: only the high-magnetization phase survives beyond the spinodal point for the low-magnetization solution. Regions IV and V: as in region III, there is a unique magnetized phase, but decreasing $\lambda$ it transitions continuously to the PM phase.
• Figure 4: Order parameter $m=q$ in the Bayes optimal case for Ising prior, additive Gaussian noise and $p=2$. The order parameter is presented as a function of $\alpha$ for different values of $\lambda$. These figures present the same information as in Fig. \ref{['fig:ising_gauss_p2']}, but shown in a different, complementary way. The first panel shows a unique branch of solutions, transitioning continuously from the paramagnetic one $m=0$. Increasing $\lambda$, in the second and third panels, also a high overlap branch (shown in green) appears discontinuously. Last panel shows the behaviour of the order parameter in the $\lambda\rightarrow\infty$ limit. The vertical dashed line coincides with the easy-to-hard threshold $\alpha_P$ for perfect reconstruction in the $\lambda\rightarrow\infty$ limit, while the boundary for the existence of the perfect recovery solution is given by $\alpha_s=0$.
• Figure 5: Evolution of the order parameters in the G-AMP algorithm for $\alpha=1.6$, $\alpha=1.2$ and $\alpha=0.9$ on random instances of the problem. The horizontal axis $t$ is for the time step. Data is averaged over 5 instances. Solid lines represent the State Evolution predictions. On the smaller panels, the convergence parameter is plotted.

Theorems & Definitions (9)

• Lemma 1
• Lemma 2
• Lemma 3
• Corollary 1
• Lemma 4
• Lemma 5
• proof
• Lemma 6
• Lemma 7