Nonlinear Multiple Response Regression and Learning of Latent Spaces

TL;DR

The paper presents a unified nonlinear latent space learning framework that handles both unlabeled and labeled data by casting latent space discovery as nonlinear multiple-response regression in an index model. It leverages generalized Stein's lemma to estimate the latent coefficient space B without requiring explicit forms of the nonlinear link functions, effectively generalizing PCA beyond linearity. Two estimators are developed: a first-order score-based method and a second-order score-based method, with theoretical guarantees showing consistency and convergence rates under mild assumptions. Semi-supervised extensions enable latent space learning from unlabeled data alongside labeled data, improving downstream tasks such as reconstruction and classification. Real data experiments on MNIST and M1 Patch-seq corroborate the method's advantages in interpretability, computational efficiency, and robustness over classical PCA, autoencoders, and some neural network approaches.

Abstract

Identifying low-dimensional latent structures within high-dimensional data has long been a central topic in the machine learning community, driven by the need for data compression, storage, transmission, and deeper data understanding. Traditional methods, such as principal component analysis (PCA) and autoencoders (AE), operate in an unsupervised manner, ignoring label information even when it is available. In this work, we introduce a unified method capable of learning latent spaces in both unsupervised and supervised settings. We formulate the problem as a nonlinear multiple-response regression within an index model context. By applying the generalized Stein's lemma, the latent space can be estimated without knowing the nonlinear link functions. Our method can be viewed as a nonlinear generalization of PCA. Moreover, unlike AE and other neural network methods that operate as "black boxes", our approach not only offers better interpretability but also reduces computational complexity while providing strong theoretical guarantees. Comprehensive numerical experiments and real data analyses demonstrate the superior performance of our method.

Figures (10)

• Figure 1: The data we consider could be unlabeled features only, or feature-label pairs, for example, CT and the corresponding diagnoses, or images of suspects and their positions on the surveillance screen, etc. The feature can be linearly embedded in a low-dimensional latent space, and the feature itself and possible labels can be generated from the embeddings through link functions. The goal of the work is to learn the latent space without knowing link functions.
• Figure 2: The figure demonstrates finite sample performances of competing methods when $p = 30$. From left to right, $\boldsymbol{x} \sim \mathcal{N}(0, \boldsymbol{\Sigma}_{\mathcal{N}})$, $\mathcal{H}_{\chi,\psi}(0, \boldsymbol{\Sigma}_{\mathcal{H}})$ and $t_{\nu}(0, \boldsymbol{\Sigma}_{t})$, respectively.
• Figure 3: Comparison of qualities of learned latent spaces of competing methods based on three different metrics. The values are calculated empirically on the testing set and medians over 100 repetitions are reported. From Left to right, the metrics are RMSE, SSIM and classification accuracy, respectively.
• Figure 4: PMSE on M1 Patch-seq Dataset, from left to right, $n_{2}$ equals to 50, 75 and 100, respectively. From left to right, the label size $n_{2}$ are 50, 75, 100, respectively.
• Figure S1: The figure demonstrates finite sample performances of competing methods when $p = 30$. From left to right, $\boldsymbol{x} \sim \mathcal{N}(0, \boldsymbol{\Sigma}_{\mathcal{N}})$, $\mathcal{H}_{\chi,\psi}(0, \boldsymbol{\Sigma}_{\mathcal{H}})$ and $t_{\nu}(0, \boldsymbol{\Sigma}_{t})$, respectively.

Theorems & Definitions (17)

• Definition 2.1
• Lemma 2.2
• Lemma 2.3
• Theorem 2.4
• Theorem 3.5
• Remark 3.6
• Theorem 3.8
• Theorem 3.13
• Remark I.1
• Lemma IV.1