Manifold Decoders: A Framework for Generative Modeling from Nonlinear Embeddings
Riddhish Thakare, Kingdom Mutala Akugri
TL;DR
This work tackles the reconstruction gap in nonlinear dimensionality reduction by introducing a systematic neural decoder framework for prominent NLDR methods and extending it with diffusion in the learned manifold spaces. By evaluating on CelebA and comparing against autoencoder and standard diffusion baselines, the study reveals that while the NLDR-based decoders can reconstruct data, end-to-end autoencoders currently deliver superior reconstruction quality, and manifold-based diffusion yields poorly generated samples due to the discrete, sparse nature of classical embeddings. The results highlight fundamental challenges in retrofitting generative capabilities onto NLDR methods, suggesting that effective generative manifold learning will require architectures that jointly optimize geometric structure and reconstruction, or diffusion processes designed to respect the manifold geometry. The work points to hybrid and differentiable approaches as promising directions to bridge geometry-preserving embeddings with high-fidelity generation for practical applications in generative modeling and data synthesis.
Abstract
Classical nonlinear dimensionality reduction (NLDR) techniques like t-SNE, Isomap, and LLE excel at creating low-dimensional embeddings for data visualization but fundamentally lack the ability to map these embeddings back to the original high-dimensional space. This one-way transformation limits their use in generative applications. This paper addresses this critical gap by introducing a system- atic framework for constructing neural decoder architectures for prominent NLDR methods, enabling bidirectional mapping for the first time. We extend this framework by implementing a diffusion-based generative process that operates directly within these learned manifold spaces. Through experiments on the CelebA dataset, we evaluate the reconstruction and generative performance of our approach against autoencoder and standard diffusion model baselines. Our findings reveal a fundamental trade- off: while the decoders successfully reconstruct data, their quality is surpassed by end-to-end optimized autoencoders. Moreover, manifold-constrained diffusion yields poor-quality samples, suggesting that the discrete and sparse nature of classical NLDR embeddings is ill-suited for the continuous inter- polation required by generative models. This work highlights the inherent challenges in retrofitting generative capabilities onto NLDR methods designed primarily for visualization and analysis.