Node-RF: Learning Generalized Continuous Space-Time Scene Dynamics with Neural ODE-based NeRFs

Abstract

Predicting scene dynamics from visual observations is challenging. Existing methods capture dynamics only within observed boundaries failing to extrapolate far beyond the training sequence. Node-RF (Neural ODE-based NeRF) overcomes this limitation by integrating Neural Ordinary Differential Equations (NODEs) with dynamic Neural Radiance Fields (NeRFs), enabling a continuous-time, spatiotemporal representation that generalizes beyond observed trajectories at constant memory cost. From visual input, Node-RF learns an implicit scene state that evolves over time via an ODE solver, propagating feature embeddings via differential calculus. A NeRF-based renderer interprets calculated embeddings to synthesize arbitrary views for long-range extrapolation. Training on multiple motion sequences with shared dynamics allows for generalization to unseen conditions. Our experiments demonstrate that Node-RF can characterize abstract system behavior without explicit model to identify critical points for future predictions.

Figures (8)

• Figure 1: Overview of Node-RF. $z_{can}$, $z_{t_0}$, $z_{t_1}$ are learned during warmup iterations. The Multiple Sequence Training along with NeRF (Common Loop) illustrates the Generalized Multi-Sequence Learning task. The Single Sequence Training along with NeRF (Common Loop) illustrates the Continuous Single-Sequence Dynamics task.
• Figure 2: Extrapolation results on the Pendulum dataset hofherr2023neural.
• Figure 3: Long-term extrapolations, Bouncing Ballspumarola2021d scene.
• Figure 4: Pose error for a novel trajectory in Bifurcating Hill.
• Figure 5: Dynamic Flow Masks for the predictions of a novel sequence in the Oscillating Ball scene