Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video

Abstract

We introduce latent intuitive physics, a transfer learning framework for physics simulation that can infer hidden properties of fluids from a single 3D video and simulate the observed fluid in novel scenes. Our key insight is to use latent features drawn from a learnable prior distribution conditioned on the underlying particle states to capture the invisible and complex physical properties. To achieve this, we train a parametrized prior learner given visual observations to approximate the visual posterior of inverse graphics, and both the particle states and the visual posterior are obtained from a learned neural renderer. The converged prior learner is embedded in our probabilistic physics engine, allowing us to perform novel simulations on unseen geometries, boundaries, and dynamics without knowledge of the true physical parameters. We validate our model in three ways: (i) novel scene simulation with the learned visual-world physics, (ii) future prediction of the observed fluid dynamics, and (iii) supervised particle simulation. Our model demonstrates strong performance in all three tasks.

Figures (17)

• Figure 1: Our approach captures unobservable physical properties from image observations using a parametrized latent space and adapts them for simulating novel scenes with different fluid geometries, boundary conditions, and dynamics. To achieve this, we introduce a variational method that connects the particle space, observation space, and latent space for intuitive physical inference.
• Figure 2: Graphical model of the pretraining--inference--transfer pipeline of latent intuitive physics. (a) Particle-space pretraining for probabilistic fluid simulation. (b) Visual posterior optimization from visual observations with a photometric loss. (c) Adaptation of the prior learner to the converged visual posteriors $\hat{z}$. (d) Novel scene simulation with the adapted prior learner. The training parts are highlighted in color. We present details of these training stages in Figure \ref{['fig:pipeline_supp']} in the appendix.
• Figure 3: Our model consists of four network components parametrized by $\theta,\psi,\xi,\phi$ respectively. We present the losses for pretraining the simulator and the renderer. For schematics of other training stages (i.e., visual posterior inference and prior adaptation), please refer to Figure \ref{['fig:pipeline_supp']} in the appendix.
• Figure 4: The first row shows the visual observation on the observed scene with physical parameters $\rho=2000, \nu=0.065$. Rows 2-6 show qualitative results of simulated particles on novel scenes (Left: unseen geometries, Right: unseen boundaries). More qualitative results are provided in the appendix.
• Figure 5: Qualitative results on generalization to unseen dynamics of heterogeneous fluids. We present simulation results on the observed scene (Left) and a novel scene (Right).