PAPR in Motion: Seamless Point-level 3D Scene Interpolation

TL;DR

This work tackles point-level 3D scene interpolation between two observed states using multi-view images, aiming to reconstruct intermediate geometry and renderings without intermediate supervision. It introduces PAPR in Motion, which extends Proximity Attention Point Rendering to a Lagrangian, point-based representation and learns a progression of geometry and appearance from the start to the end state. Two regularizers—local distance preservation ($L_{rigid}$) and a local displacement averaging step (LDAS)—enforce plausible, cohesive motion and temporal smoothness, enabling non-rigid deformations to be captured robustly. Across synthetic and real scenes, PAPR in Motion outperforms the leading neural renderer for dynamic scenes on Scene Interpolation metrics (SI-CD, SI-EMD, SI-FID), establishing a new baseline for point-based dynamic interpolation and offering a practical tool for animation and visual effects with minimal supervision.

Abstract

We propose the problem of point-level 3D scene interpolation, which aims to simultaneously reconstruct a 3D scene in two states from multiple views, synthesize smooth point-level interpolations between them, and render the scene from novel viewpoints, all without any supervision between the states. The primary challenge is on achieving a smooth transition between states that may involve significant and non-rigid changes. To address these challenges, we introduce "PAPR in Motion", a novel approach that builds upon the recent Proximity Attention Point Rendering (PAPR) technique, which can deform a point cloud to match a significantly different shape and render a visually coherent scene even after non-rigid deformations. Our approach is specifically designed to maintain the temporal consistency of the geometric structure by introducing various regularization techniques for PAPR. The result is a method that can effectively bridge large scene changes and produce visually coherent and temporally smooth interpolations in both geometry and appearance. Evaluation across diverse motion types demonstrates that "PAPR in Motion" outperforms the leading neural renderer for dynamic scenes. For more results and code, please visit our project website at https://niopeng.github.io/PAPR-in-Motion/ .

Figures (8)

• Figure 1: We introduce the novel task of point-level 3D scene interpolation: Given multi-view RGB images of a scene at two distinct states, the start state and the end state, our goal is to achieve a seamless point-level interpolation between these states without any intermediate supervision. Our method, PAPR in Motion, can effectively synthesize a plausible geometry for any intermediate state, represented by a point cloud (top row), and renders the corresponding scene appearance (bottom row). Additionally, we provide a visualization of the point interpolation trajectories (right), which illustrate the coherent motion synthesized within the scene.
• Figure 2: Qualitative comparison of 3D scene interpolation from start to end state using synthetic scenes. Both methods start by training a static model for the start state and subsequently finetune it towards the end state, all without any intermediate supervision. Dynamic Gaussian Luiten2023Dynamic3G struggles to maintain consistent scene geometry and appearance, resulting in unnatural motion, as shown by the point trajectories. On the other hand, PAPR in Motion creates smooth and plausible interpolations between states. It successfully handles a variety of transformations, from the rigid motion observed in the Lego man's standing pose to the non-rigid, fluid motion of the dolphin's tail.
• Figure 3: Qualitative comparison of 3D scene interpolation from start to end state using real-world scenes. Both methods start by training a static model for the start state and subsequently finetune it towards the end state, all without any intermediate supervision. Dynamic Gaussian Luiten2023Dynamic3G struggles with significant scene changes, as shown by both scenes where only a portion of the points move to the right place. In contrast, PAPR in Motion successfully handles these challenging scenarios, producing smooth and realistic interpolations between states.
• Figure 4: Ablation study on the impact of proposed regularization terms, where we incrementally remove the local displacement averaging step (LDAS) and the local distance preserving loss ($\mathcal{L}_{rigid}$). The results show that removing LDAS leads to unwanted deformations in the part geometry. When $\mathcal{L}_{rigid}$ is also removed, there is a further detrimental effect, with points drifting away from the object surface. These findings validates the critical role of both regularization techniques in maintaining the integrity and the quality of our model's interpolations.
• Figure 5: Sensitivity analysis on the effect of $k$, the number of nearest neighbour used in regularization calculations. The results show that larger values of $k$ tend to increase the rigidity of moving parts while smaller values of $k$ result in more flexible part movements. In this example, $k=300$ exhibits the best surface continuity and smoothness throughout the interpolation process.