EigenGS Representation: From Eigenspace to Gaussian Image Space

TL;DR

This work introduces EigenGS, a method that connects eigenspace PCA with 2D Gaussian Splatting to create a fast, high-quality image representation. By learning eigen-Gaussians from PCA and then mapping test images to an image-space Gaussian set, EigenGS provides an instant, robust initialization that accelerates convergence while enabling subsequent refinement through image-space reconstruction loss. A frequency-aware learning scheme partitions Gaussians to model distinct spatial frequencies, mitigating penny-round-tile artifacts and improving reconstruction across resolutions. Cross-dataset generalization experiments, including an ImageNet-trained universal EigenGS, demonstrate robust transfer and potential for universal Gaussian-based representations in real-time image synthesis and rendering tasks.

Abstract

Principal Component Analysis (PCA), a classical dimensionality reduction technique, and 2D Gaussian representation, an adaptation of 3D Gaussian Splatting for image representation, offer distinct approaches to modeling visual data. We present EigenGS, a novel method that bridges these paradigms through an efficient transformation pipeline connecting eigenspace and image-space Gaussian representations. Our approach enables instant initialization of Gaussian parameters for new images without requiring per-image optimization from scratch, dramatically accelerating convergence. EigenGS introduces a frequency-aware learning mechanism that encourages Gaussians to adapt to different scales, effectively modeling varied spatial frequencies and preventing artifacts in high-resolution reconstruction. Extensive experiments demonstrate that EigenGS not only achieves superior reconstruction quality compared to direct 2D Gaussian fitting but also reduces necessary parameter count and training time. The results highlight EigenGS's effectiveness and generalization ability across images with varying resolutions and diverse categories, making Gaussian-based image representation both high-quality and viable for real-time applications.

Figures (15)

• Figure 1: Overview of our method: (a) Our approach learns a set of eigen-Gaussian representations that characterize the principal components in eigenspace. (b) For a given test image, we can instantly derive its image-based Gaussian representation in image space from the eigen-Gaussians. (c) The initial image-space Gaussian representation serves as a robust starting point, enabling rapid convergence to reconstruct the input image. (d) We further optimize this representation by minimizing the image-space Gaussian reconstruction loss between the rendered output and the test image to obtain a high-quality final result. Here, we use red arrows to represent the gradient flow from the reconstruction loss.
• Figure 2: Fast convergence speed on Cats dataset. Our method achieves perceptually significant improvements within the first 100 iterations. The rapid visual convergence, exhibited by $\widetilde{I}^{(0)} \rightarrow \widetilde{I}^{(10)} \rightarrow \widetilde{I}^{(100)}$, demonstrates the effectiveness of our PCA-based initialization, particularly on well-aligned datasets with moderate resolution. Additionally, our result also indicates that traditional PCA reconstruction, which is basically similar to our result at iteration zero, can be further improved by our approach.
• Figure 3: The "penny-round-tile" artifact that emerges when Gaussians converge to uniformly small sizes during optimization. Our frequency-aware learning scheme addresses this issue by modeling different spatial frequencies with appropriately sized Gaussians.
• Figure 4: We sample the scores every 50 iterations on all datasets to exhibit the fast convergence capability, with particularly strong performance on smaller datasets like CelebA and Cats. The advantage is less pronounced but still evident on the more challenging Stanford Cars dataset due to its varied viewpoints and larger dimensions.
• Figure 5: Cross-dataset initialization comparison. Left Column: initialization using EigenGS from the corresponding dataset. Middle column: initialization using Cats-trained EigenGS for CelebA dataset. Right Column: initialization using ImageNet-trained EigenGS for CelebA dataset. Despite some domain-specific artifacts in cross-dataset results, it still demonstrates effective color and structure preservation. Therefore, our method can still achieve high-quality reconstruction even with cross-dataset initialization, which aligns with the quantitative evaluations shown in \ref{['tab:transfer_result']}.