Latent Space Imaging

TL;DR

Latent Space Imaging (LSI) introduces a paradigm for ultra-low-bandwidth imaging by encoding image information directly into the latent space of a pretrained generative model. It couples a linear optical encoder with a nonlinear digital encoder to produce a latent representation that can be used for reconstruction or directly for downstream tasks via linear projections, dramatically reducing sensor data while maintaining essential semantics. The work provides a proof-of-concept hardware prototype based on single-pixel imaging, demonstrates extensive downstream capabilities (face attributes, landmarks, segmentation) on face data, and shows compelling compression gains (up to 1:16384 for certain tasks). The results suggest LSI enables domain-specific, high-speed, and hardware-light imaging, with applications spanning privacy-preserving cameras and ultra-low-power sensing, while highlighting dependency on the training distribution of the generative model and the potential need for alternative physical encodings for broader domains.

Abstract

Digital imaging systems have traditionally relied on brute-force measurement and processing of pixels arranged on regular grids. In contrast, the human visual system performs significant data reduction from the large number of photoreceptors to the optic nerve, effectively encoding visual information into a low-bandwidth latent space representation optimized for brain processing. Inspired by this, we propose a similar approach to advance artificial vision systems. Latent Space Imaging introduces a new paradigm that combines optics and software to encode image information directly into the semantically rich latent space of a generative model. This approach substantially reduces bandwidth and memory demands during image capture and enables a range of downstream tasks focused on the latent space. We validate this principle through an initial hardware prototype based on a single-pixel camera. By implementing an amplitude modulation scheme that encodes into the generative model's latent space, we achieve compression ratios ranging from 1:100 to 1:1000 during imaging, and up to 1:16384 for downstream applications. This approach leverages the model's intrinsic linear boundaries, demonstrating the potential of latent space imaging for highly efficient imaging hardware, adaptable future applications in high-speed imaging, and task-specific cameras with significantly reduced hardware complexity.

Figures (24)

• Figure 1: We propose an extremely-compressed imaging paradigm called Latent Space Imaging (LSI). The optical encoder (O) projects the real signal into a compressed set of measurements. A digital encoder ($\mathcal{D}_{\theta}\xspace$) then maps this signal to the latent space (L) of a frozen generative model ($\mathcal{G}\xspace$), enabling image reconstruction. The L can also be linearly projected ($P\xspace$) to perform downstream tasks directly—such as facial segmentation ($P\xspace_{S}$), landmark detection ($P\xspace_{L}$), and attribute classification ($P\xspace_{A}$)—without requiring image reconstruction or a complex new model.
• Figure 2: We illustrate one possible implementation of the Latent Space Imaging technique using a single-pixel framework. An objective lens focuses the image onto the Digital Micromirror Device (DMD), which is responsible for implementing the learned mask to spatially modulate the incoming signal. This is followed by a relay lens, which focuses the modulated signal onto a photodiode (SPD) responsible for integrating the signal. Utilizing time multiplexing, we can retrieve the necessary measurements.
• Figure 3: Displays optimized pixel forms for 1:1024 compression ratio.
• Figure 4: Illustration of reconstructions achieved through the LSI pipeline with bounded and quantized pixels, using compression ratios from 1:128 to 1:4096 in the simulation setting.
• Figure 5: Face reconstructions from the experimental setup across different compression ratios.