Neural Implicit Representations for Physical Parameter Inference from a Single Video

TL;DR

The paper addresses the challenge of extracting interpretable physical parameters from a single video by coupling neural implicit representations for appearance with a parametric neural ODE for dynamics. It introduces a per-scene model that renders photorealistic frames while identifying initial conditions and physical parameters through differentiable rendering and end-to-end optimization of a known dynamical system, $\\dot{\mathbf{z}} = f(\mathbf{z}, t; \theta_{ode})$. Key contributions include exact parameter identification from a single video, interpretable dynamics, long-term predictions, and high-resolution, editable renders of scenes with modified physical parameters. The approach demonstrates strong results on synthetic and real data, enabling data-efficient physical parameter inference and potential extensions to 3D rendering and analysis.

Abstract

Neural networks have recently been used to analyze diverse physical systems and to identify the underlying dynamics. While existing methods achieve impressive results, they are limited by their strong demand for training data and their weak generalization abilities to out-of-distribution data. To overcome these limitations, in this work we propose to combine neural implicit representations for appearance modeling with neural ordinary differential equations (ODEs) for modelling physical phenomena to obtain a dynamic scene representation that can be identified directly from visual observations. Our proposed model combines several unique advantages: (i) Contrary to existing approaches that require large training datasets, we are able to identify physical parameters from only a single video. (ii) The use of neural implicit representations enables the processing of high-resolution videos and the synthesis of photo-realistic images. (iii) The embedded neural ODE has a known parametric form that allows for the identification of interpretable physical parameters, and (iv) long-term prediction in state space. (v) Furthermore, the photo-realistic rendering of novel scenes with modified physical parameters becomes possible.

Figures (17)

• Figure 1: Our method infers physical parameters directly from real-world videos, like the shown pendulum motion. Separated by the red line, the right half of each image shows the input frame, and the left half shows our reconstruction based on physical parameters that we estimate from the input. We show 6 out of 10 frames that were used for training. The proposed model can precisely recover the metric length of the pendulum from the monocular video (relative error to true length is less than 4.1%). Best viewed on screen with magnification. Please also consider the supplementary video.
• Figure 2: Overview of our approach. Dynamics Representation: The dynamics in the video are modelled by an ordinary differential equation (ODE), which is solved depending on unknown initial conditions $\mathbf{z}_0$ and unknown physical parameters $\theta_{\text{ode}}$. The solution curve $\mathbf{z}\left(t; \mathbf{z}_0, \theta_{\text{ode}}\right)$ is used to parametrize a time-dependent transformation $T(\mathbf{z}\left(t; \mathbf{z}_0, \theta_{\text{ode}}\right),\theta_{+})$ from the global coordinates $XY$ of the background to the local coordinates $xy$ of the moving object. The (unknown) parameters $\theta_{+}$ encode additional degrees of freedom of the transformation, for example the pivot point of a pendulum. Scene Representation: The functions $F(\cdot;\theta_{\text{bg}})$ and $G(\cdot;\theta_{\text{obj}})$ are neural networks that model the appearance of the background and of the object, using color $c$ and opacity $o$ (only for the foreground objects). Rendering: Rendering is done by blending the foreground and the background color based on the opacity of the foreground objects. Loss: We can estimate the unknown physical parameters for a given video based on a rendering loss which penalizes the discrepancy between the input video frames and the rendered video. All estimated parameters and network weights are shown in green text in the figure.
• Figure 3: Two masses spring system in which MNIST digits are connected by an (invisible) spring (DBLP:conf/iclr/JaquesBH20 sequence 6). The arrow indicates the start of the prediction of unseen frames. We compare our results to DBLP:conf/iclr/JaquesBH20, both trained on the full dataset (B: Full) and overfitted to sequence 6 (B: Overfit). For the spring constant and equilibrium distance ($k$, $l$) the different methods achieve the relative errors $(2.7 \%,~81.0 \%)$ (B: Overfit); $(3.7 \%,~1.8 \%)$ B: Full; and $(\mathbf{0.7} \%,~\mathbf{0.7} \%)$ (Ours). (Best viewed magnified on screen)
• Figure 4: Prediction when training on the first 10 frames of sequence 0 of the pendulum test data by DBLP:conf/nips/ZhongL20. Each image shows the prediction of the respective method in white, and the ground truth as green overlay. For both methods, the prediction of images seen during training (frames 1,7,10) works well. For unseen data (frames 11,12,16,20), our method leads to more reliable predictions, meaning that our physical parameter estimation is more accurate.
• Figure 5: Rendered frames for sequence 1 of the wallclock. The left image is part of the training set, "unseen frame 1" is between two training frames, "unseen frame 2" is a future frame after the interval seen during training. While our methods makes photorealistic predictions for both unseen frames, the time-dependent background ("Baseline-t") fails in both cases. Note the visible blending between the neighboring frames in the baseline (red arrows) and the fine detail on the pendulum recovered by our method (green arrow).