Gaussian Splatting to Real World Flight Navigation Transfer with Liquid Networks

TL;DR

This work builds a simulator by integrating Gaussian Splatting with quadrotor flight dynamics, and trains robust navigation policies using Liquid neural networks, and obtains a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, crafty programming of expert demonstration training data, and the task understanding capabilities of Liquid networks.

Abstract

Simulators are powerful tools for autonomous robot learning as they offer scalable data generation, flexible design, and optimization of trajectories. However, transferring behavior learned from simulation data into the real world proves to be difficult, usually mitigated with compute-heavy domain randomization methods or further model fine-tuning. We present a method to improve generalization and robustness to distribution shifts in sim-to-real visual quadrotor navigation tasks. To this end, we first build a simulator by integrating Gaussian Splatting with quadrotor flight dynamics, and then, train robust navigation policies using Liquid neural networks. In this way, we obtain a full-stack imitation learning protocol that combines advances in 3D Gaussian splatting radiance field rendering, crafty programming of expert demonstration training data, and the task understanding capabilities of Liquid networks. Through a series of quantitative flight tests, we demonstrate the robust transfer of navigation skills learned in a single simulation scene directly to the real world. We further show the ability to maintain performance beyond the training environment under drastic distribution and physical environment changes. Our learned Liquid policies, trained on single target manoeuvres curated from a photorealistic simulated indoor flight only, generalize to multi-step hikes onboard a real hardware platform outdoors.

Figures (10)

• Figure 1: We create a photorealistic policy learning environment by combining Gaussian Splatting with quadrotor flight dynamics to generate high-fidelity training datasets. A model, $\pi$, with parameters $\theta$, is trained using input images $x_t$ to match its predicted output $\mu_t$ with ground truth labels $y_t = [v_t^x, v_t^y, v_t^z, \dot{\psi}_t]$, where $v_t$ denotes desired triaxial velocities and $\dot{\psi}_t$ is the yaw rate. Liquid neural networks learn the task through behavior cloning and achieve sim-to-real transfer.
• Figure 2: Gaussian Splatting dataset sample trajectories. The trajectories are designed in two phases, first an approach where the drone gets close to the target and centers it in the middle of the image, followed by a turn procedure in the direction corresponding to the color of the target.
• Figure 3: Diagram of the in-simulation inference protocol with physics engine dynamics and GS rendering. At time $t$, the GS image $x_t$ is rendered from the simulated position and attitude, and processed through the CNN of the policy network $\pi(x_t)$. Obtained features, along with the elapsed time $\delta t = t - t_{-}$, are input into the Liquid network, which outputs the velocity control command.
• Figure 4: Results of the infinite simulated hike experiment for 10 different initializations of the goal positions in the PyBullet environment.
• Figure 5: Distribution of 100 initial altitudes and yaw offsets for trajectory generation.