DefVINS: Visual-Inertial Odometry for Deformable Scenes

TL;DR

DefVINS addresses visual–inertial odometry in deformable scenes by decoupling a rigid, IMU-anchored state from a non-rigid deformation graph and activating deformation degrees of freedom based on estimator conditioning. It combines IMU preintegration, gravity residuals, visual reprojection, and elastic–viscous–photometric priors on a deformation graph within a sliding-window optimization, augmented by an explicit observability/conditioning analysis. The approach yields improved robustness and accuracy over rigid VIO and purely deformable methods, demonstrated on synthetic and real deformable sequences across varying deformation levels. This work enables reliable metric-scale localization in non-rigid environments, with practical impact for robotics and AR in clothing, textiles, and similar scenes.

Abstract

Deformable scenes violate the rigidity assumptions underpinning classical visual-inertial odometry (VIO), often leading to over-fitting to local non-rigid motion or severe drift when deformation dominates visual parallax. We introduce DefVINS, a visual-inertial odometry framework that explicitly separates a rigid, IMU-anchored state from a non--rigid warp represented by an embedded deformation graph. The system is initialized using a standard VIO procedure that fixes gravity, velocity, and IMU biases, after which non-rigid degrees of freedom are activated progressively as the estimation becomes well conditioned. An observability analysis is included to characterize how inertial measurements constrain the rigid motion and render otherwise unobservable modes identifiable in the presence of deformation. This analysis motivates the use of IMU anchoring and informs a conditioning-based activation strategy that prevents ill-posed updates under poor excitation. Ablation studies demonstrate the benefits of combining inertial constraints with observability-aware deformation activation, resulting in improved robustness under non-rigid environments.

Figures (4)

• Figure 1: Structure of the full observability matrix $\mathcal{O}$. The matrix explicitly incorporates the accelerometer bias into the state. Each block row corresponds to the Jacobians of the different measurement and constraint terms, including IMU preintegration factors, visual residuals, non--rigid motion priors, and bias and gravity constraints. This structured formulation highlights the contribution of each sensing modality to the overall system observability.
• Figure 2: Illustrative observability analysis under synthetic conditions. Evolution of the conditioning score $\log_{10}(\rho_k)$ as a function of the number of stacked keyframe pairs $k$. Inertial sensing and non-rigid regularization significantly improve numerical conditioning, yielding well-observed directions with only a few frames.
• Figure 3: Qualitative comparison of DefVINS operating modes and baselines under R4 sequence. a) Rigid VI SLAM (ORB-SLAM3): a representative state-of-the-art rigid visual--inertial system, which may suffer from tracking loss and relocalization in deformable scenes. b) Non-rigid SLAM (NR-SLAM): a representative state-of-the-art for non-rigid environments, which also suffer from tracking loss and relocalization due to its focus on medical datasets. c) DefVINS-V (visual-only, non-rigid): absence of inertial constraints leads to accumulated drift, particularly during turning motions. b) DefVINS-VI (rigid): introducing inertial sensing stabilizes rotational estimates, but assuming scene rigidity limits performance under deformation. e) DefVINS (full): by jointly enabling inertial sensing and explicit non-rigid modeling, the proposed system achieves the highest global consistency and lowest trajectory error. Dashed line represent the ground-truth while colored line represent the camera trajectory.
• Figure 4: Deformation graph on sequence R4. Green and red edges denote the graph at times $t\!-\!1$ and $t$, respectively. Their differences indicate medium-to-high non-rigid deformation, with stronger effects in the lower-left region.