Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Out-of-Sample Embedding with Proximity Data: Projection versus Restricted Reconstruction

Michael W. Trosset, Kaiyi Tan, Minh Tang, Carey E. Priebe

TL;DR

This work analyzes how proximity data can be used for out-of-sample embedding by distinguishing two core strategies: projection, which preserves the original representation space while adding a point, and restricted reconstruction, which preserves the original configuration but may change the representation space. It shows that many kernel-based out-of-sample methods arise from either projection or restricted reconstruction, and derives concrete, computationally tractable formulations for each, including a univariate ridge-like optimization for restricted reconstruction. The paper also clarifies how kernel and distance-based methods relate, discusses PSD versus non-PSD kernels, and offers practical guidance on when to use each strategy. The results provide a unified perspective on CMDS, kernel PCA, and Landmark MDS extensions, with implications for embedding large proximity datasets efficiently.

Abstract

The problem of using proximity (similarity or dissimilarity) data for the purpose of "adding a point to a vector diagram" was first studied by J.C. Gower in 1968. Since then, a number of methods -- mostly kernel methods -- have been proposed for solving what has come to be called the problem of *out-of-sample embedding*. We survey the various kernel methods that we have encountered and show that each can be derived from one or the other of two competing strategies: *projection* or *restricted reconstruction*. Projection can be analogized to a well-known formula for adding a point to a principal component analysis. Restricted reconstruction poses a different challenge: how to best approximate redoing the entire multivariate analysis while holding fixed the vector diagram that was previously obtained. This strategy results in a nonlinear optimization problem that can be simplified to a unidimensional search. Various circumstances may warrant either projection or restricted reconstruction.

Out-of-Sample Embedding with Proximity Data: Projection versus Restricted Reconstruction

TL;DR

This work analyzes how proximity data can be used for out-of-sample embedding by distinguishing two core strategies: projection, which preserves the original representation space while adding a point, and restricted reconstruction, which preserves the original configuration but may change the representation space. It shows that many kernel-based out-of-sample methods arise from either projection or restricted reconstruction, and derives concrete, computationally tractable formulations for each, including a univariate ridge-like optimization for restricted reconstruction. The paper also clarifies how kernel and distance-based methods relate, discusses PSD versus non-PSD kernels, and offers practical guidance on when to use each strategy. The results provide a unified perspective on CMDS, kernel PCA, and Landmark MDS extensions, with implications for embedding large proximity datasets efficiently.

Abstract

The problem of using proximity (similarity or dissimilarity) data for the purpose of "adding a point to a vector diagram" was first studied by J.C. Gower in 1968. Since then, a number of methods -- mostly kernel methods -- have been proposed for solving what has come to be called the problem of *out-of-sample embedding*. We survey the various kernel methods that we have encountered and show that each can be derived from one or the other of two competing strategies: *projection* or *restricted reconstruction*. Projection can be analogized to a well-known formula for adding a point to a principal component analysis. Restricted reconstruction poses a different challenge: how to best approximate redoing the entire multivariate analysis while holding fixed the vector diagram that was previously obtained. This strategy results in a nonlinear optimization problem that can be simplified to a unidimensional search. Various circumstances may warrant either projection or restricted reconstruction.
Paper Structure (6 sections, 1 theorem, 44 equations, 2 figures)

This paper contains 6 sections, 1 theorem, 44 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Two Fundamental Strategies
  3. Projection with Proximity Data
  4. Restricted Reconstruction with Proximity Data
  5. Computing Restricted Reconstructions
  6. Discussion

Key Result

Theorem 1

Let $M$ denote an $n \times q$ data matrix and let denote the corresponding centered data matrix with pairwise inner products $B = \tilde{M} \tilde{M}^t$. The $d$-dimensional principal component representation of $M$ is the $n \times d$ configuration matrix $U_d \Sigma_d$, with pairwise inner products $\bar{B} = U_d \Sigma_d^2 U_d^t$. Let $Z$ denote

Figures (2)

  • Figure 1: Two fundamental strategies for out-of-sample embedding. Both strategies fix the original configuration of points. The projection strategy also fixes the representation space. Unlike the projection strategy, the restricted reconstruction strategy depends on the method used to construct the original configuration.
  • Figure 2: Out-of-sample embedding by restricted reconstruction for CMDS. The original objects $\xi_1,\ldots,\xi_n$ are embedded as $x_1,\ldots,x_n \in \Re^d$. The out-of-sample object $\eta$ is then embedded as $y_* \in \Re^d$.

Theorems & Definitions (3)

  • Example 1
  • Theorem 1
  • Example 2