3D Freehand Ultrasound using Visual Inertial and Deep Inertial Odometry for Measuring Patellar Tracking

TL;DR

This paper tackles the challenge of observing PFJ dynamics without radiation or expensive infrastructure by proposing a handheld ultrasound-based 3D reconstruction workflow that relies on Visual-Inertial Odometry (VIO) or learning-based IMU-only tracking to localize the scanner. The authors implement and compare three tracking pipelines—external motion capture, VIO, and IMU-only—coupled with 2D ultrasound bone segmentation and ICP/TSDF-based surface fusion to produce 3D bone meshes. They demonstrate that VIO achieves reconstruction accuracy close to the external tracking baseline (mean errors around $1.25$ mm vs $1.21$ mm for ground truth), while the IMU-only method shows higher drift and mean error (~$1.85$ mm). The results indicate infrastructure-free VIO is a viable path toward practical, radiation-free PFJ assessment with potential clinical impact for monitoring patellar tracking in painful TKA cases.

Abstract

Patellofemoral joint (PFJ) issues affect one in four people, with 20% experiencing chronic knee pain despite treatment. Poor outcomes and pain after knee replacement surgery are often linked to patellar mal-tracking. Traditional imaging methods like CT and MRI face challenges, including cost and metal artefacts, and there's currently no ideal way to observe joint motion without issues such as soft tissue artefacts or radiation exposure. A new system to monitor joint motion could significantly improve understanding of PFJ dynamics, aiding in better patient care and outcomes. Combining 2D ultrasound with motion tracking for 3D reconstruction of the joint using semantic segmentation and position registration can be a solution. However, the need for expensive external infrastructure to estimate the trajectories of the scanner remains the main limitation to implementing 3D bone reconstruction from handheld ultrasound scanning clinically. We proposed the Visual-Inertial Odometry (VIO) and the deep learning-based inertial-only odometry methods as alternatives to motion capture for tracking a handheld ultrasound scanner. The 3D reconstruction generated by these methods has demonstrated potential for assessing the PFJ and for further measurements from free-hand ultrasound scans. The results show that the VIO method performs as well as the motion capture method, with average reconstruction errors of 1.25 mm and 1.21 mm, respectively. The VIO method is the first infrastructure-free method for 3D reconstruction of bone from wireless handheld ultrasound scanning with an accuracy comparable to methods that require external infrastructure.

Figures (8)

• Figure 1: Flow diagram of complete reconstruction pipeline. The estimation module computes the scanning trajectory using either IMU-only, VIO or motion capture external tracking. Separately, 2D ultrasound points are segmented for the presence of bone. These points are projected in 3D using the estimated trajectory and accumulated into separate point clouds for each sweep over the bone. After manually segmenting out the surface, the point clouds are aligned using ICP and integrated into the surface estimate. Finally, once all the sweeps have been integrated, the final surface is sampled and converted to a mesh using the Poisson method.
• Figure 2: Experimental setup for collecting ultrasound dataset. A Realsense stereo camera is mounted on the Clarius ultrasound scanner and a mount holds the scanner in place. The human operator picks up the scanner and passes it over the bone model several times before returning it to the mount.
• Figure 3: Two example factor graph frameworks for five states $j={0,1,2,3,4}$. On top is the VIO method which uses prior, visual and IMU factors. On the bottom is the learned inertial odometry method which only uses IMU and learned factors. The four types of factors are: prior factors constraining the start and end pose to be equivalent, IMU integration factor from Forster2017, visual landmark factors from Wisth2022 and our learned displacement factor from Buchanan2021CORL.
• Figure 4: Ultrasound images collected through the Clarius API are in a size of $640 \times480$. The origin of the ultrasound image frame $\mathtt{M}$ was defined as pixel (0,0). A fixed, known transformation from $\mathtt{M}$ into the ultrasound scanner's frame $\mathtt{U}$ was applied to all segmented pixels to project them into 3D points. Finally, the 3D points in $\mathtt{U}$ were transformed into the global coordinate system using the selected odometry source.
• Figure 5: Trajectories for the different algorithms while scanning Bone 1. Visual-inertial and learned inertial methods both use the loop closure constraint. In these plots, the trajectory is cropped to start when the scanner is in contact with the bone and cropped to end when scanning is finished. The first pose is aligned with ground truth in position and yaw, not roll or pitch.