ReMoSPLAT: Reactive Mobile Manipulation Control on a Gaussian Splat

TL;DR

ReMoSPLAT presents a QP-based reactive mobile manipulation controller that uses Gaussian Splat representations to perform obstacle avoidance without full planning. It analyzes two distance-query strategies—sphere-to-ellipsoid geometry and depth rasterisation—and integrates distances as hard constraints and a distance-based cost in the QP, achieving performance close to a perfect ground-truth baseline in both synthetic and real-world-like scenarios. The results show depth rasterisation is more robust to low-opacity splats and noisy reconstructions, while maintaining real-time performance. This work advances reactive manipulation by leveraging GS to encode detailed geometry for safe, efficient motion in cluttered environments, with clear paths for incremental and semantics-aware future enhancements.

Abstract

Reactive control can gracefully coordinate the motion of the base and the arm of a mobile manipulator. However, incorporating an accurate representation of the environment to avoid obstacles without involving costly planning remains a challenge. In this work, we present ReMoSPLAT, a reactive controller based on a quadratic program formulation for mobile manipulation that leverages a Gaussian Splat representation for collision avoidance. By integrating additional constraints and costs into the optimisation formulation, a mobile manipulator platform can reach its intended end effector pose while avoiding obstacles, even in cluttered scenes. We investigate the trade-offs of two methods for efficiently calculating robot-obstacle distances, comparing a purely geometric approach with a rasterisation-based approach. Our experiments in simulation on both synthetic and real-world scans demonstrate the feasibility of our method, showing that the proposed approach achieves performance comparable to controllers that rely on perfect ground-truth information.

Figures (5)

• Figure 1: Illustration of our reactive control approach for a mobile manipulator tasked with reaching a 3D pose inside a bookshelf. The scene is represented as a 2D Gaussian Splat, visualised as ellipsoids in the Front view (top). The controller computes joint velocities to drive the robot toward the target pose, producing the yellow end-effector and blue base trajectories. Collision avoidance is enforced through hard and soft constraints, shown as red line segments derived from distance estimates against the Gaussian Splat.
• Figure 2: Depiction of the median depth rasterisation process. In blue, we see a camera ray $\overrightarrow{r}$; on the right plot, we visualise the transmittance $T$ of each intersected ellipsoid with respect to their distance along the z-axis of the camera. $z_m$ denotes the median depth, and $z_s$ denotes the distance to the first intersected ellipsoid surface.
• Figure 3: High-level diagram of our approach, showing the same robot state and how it models the constraints depending on the chosen distance method. (Left) Illustrates how the sphere-to-ellipsoid generates constraints based on the closest point on the ellipsoid surface of each sphere (\ref{['sec:euclidean']}). (Right) Illustrates how the depth rasterisation approach relies on rendering depth images from candidate orientations (\ref{['eq:rotations']}), resulting in one collision constraint per rendered view (\ref{['sec:distance_depth']}).
• Figure 4: Resulting 2DGS Ellipsoids on a sample of the bookshelf and table scenarios utilised for our first set of experiments.
• Figure 5: Resulting Gaussian Splats of the printer scene, including the resulting motions showing in red and black the trajectories of the end effector and the robot base, respectively, on (a) Clean reconstruction of the scene in which the ellipsoids align with the underlying geometry. (b) Noisy reconstruction obtained by adding 1,000 random points before training, in which we can see a set of floating ellipsoids.