LCIP: Loss-Controlled Inverse Projection of High-Dimensional Image Data

TL;DR

LCIP introduces a loss-controlled inverse projection framework that separates information preserved by a projection from information lost during projection, enabling a user-driven sweep of the high-dimensional data space. By training an encoder $Enc$, a decoder $Dec$, and a discriminator $Dis$ with the objective $J = L_{rec}(X,X') - \lambda L_{adv}(Y,Y')$, LCIP achieves disentanglement of $\mathbf{y}$ and $\mathbf{z}$ while reconstructing data from $Y$ and $Z$. It then interpolates $\mathbf{z}$ for unseen 2D points and applies an interactive control mechanism that combines $\mathbf{y}$ and $\mathbf{z}$ to produce a controllable $P^{-1}$, enabling exploration beyond a fixed surface and yielding smoother, more plausible samples. Evaluations against state-of-the-art inverse projections show comparable quality with added benefits in gap areas, and user studies confirm the practicality and smoothness of LCIP’s interactive control. The method generalizes across projections like t-SNE and UMAP and holds promise for applications in style transfer, data augmentation, and decision-map analysis.

Abstract

Projections (or dimensionality reduction) methods $P$ aim to map high-dimensional data to typically 2D scatterplots for visual exploration. Inverse projection methods $P^{-1}$ aim to map this 2D space to the data space to support tasks such as data augmentation, classifier analysis, and data imputation. Current $P^{-1}$ methods suffer from a fundamental limitation -- they can only generate a fixed surface-like structure in data space, which poorly covers the richness of this space. We address this by a new method that can `sweep' the data space under user control. Our method works generically for any $P$ technique and dataset, is controlled by two intuitive user-set parameters, and is simple to implement. We demonstrate it by an extensive application involving image manipulation for style transfer.

Figures (19)

• Figure 1: LCIP workflow. $\oplus$ denotes concatenation. (a) Training $Dis$ and $Dec$. $P$ is a user-selected DR method that projects $X$ to $Y$. $Enc$ encodes $X$ into $Z$. $Dis$ uses $Z$ to predict $Y$. $Dec$ ($P^{-1}$) uses $Y$ and $Z$ to reconstruct $X$. The adversarial network $Dis$ enforces disentanglement (Sec. \ref{['sec:design']}). (b) Inversely projecting a 2D point $\mathbf{p}$ to data sample $\mathbf{q}$ (Sec. \ref{['sec:initialize_z']}). (c) Users can refine the inverse projection by maneuvering the controls marked in red: a source point $\mathbf{p}_s$, a target point $\mathbf{x}_t$, a pull factor $\alpha$, and a kernel radius $\sigma$. (Sec. \ref{['sec:control_mechanism']}).
• Figure 2: Controlling the inverse projection. User parameters are marked in red.
• Figure 3: Showing disentanglement on the MNIST dataset. (a) 2D t-SNE projection $Y$. (b) UMAP projection of $Z$ using $WithDis$. (c) Inverse projections of the linear interpolation between two data points in the $Z$ and $Y$ spaces, using $WithDis$ (c) and $NoDis$ (d).
• Figure 4: MSE of the studied inverse projections.
• Figure 5: Visual comparison of inverse projection, MNIST dataset. (a) 13 points A-M are selected in the projection space at various distance from projected samples. (b) The inverse projections at these locations using the tested inverse projection techniques. (c) Results obtained by inversely projecting points along a line between the locations U and V in the projection.