RayEmb: Arbitrary Landmark Detection in X-Ray Images Using Ray Embedding Subspace

TL;DR

This work proposes a novel method to address intra-operative 2D-3D registration of X-ray images with pre-operatively acquired CT scans by detecting arbitrary landmark points in X-ray images, and eliminates the need for manually annotating fixed landmarks.

Abstract

Intra-operative 2D-3D registration of X-ray images with pre-operatively acquired CT scans is a crucial procedure in orthopedic surgeries. Anatomical landmarks pre-annotated in the CT volume can be detected in X-ray images to establish 2D-3D correspondences, which are then utilized for registration. However, registration often fails in certain view angles due to poor landmark visibility. We propose a novel method to address this issue by detecting arbitrary landmark points in X-ray images. Our approach represents 3D points as distinct subspaces, formed by feature vectors (referred to as ray embeddings) corresponding to intersecting rays. Establishing 2D-3D correspondences then becomes a task of finding ray embeddings that are close to a given subspace, essentially performing an intersection test. Unlike conventional methods for landmark estimation, our approach eliminates the need for manually annotating fixed landmarks. We trained our model using the synthetic images generated from CTPelvic1K CLINIC dataset, which contains 103 CT volumes, and evaluated it on the DeepFluoro dataset, comprising real X-ray images. Experimental results demonstrate the superiority of our method over conventional methods. The code is available at https://github.com/Pragyanstha/rayemb.

Figures (6)

• Figure 1: Given a query X-ray image and its corresponding CT volume, 3D landmark points within the volume are randomly sampled. For each sampled point, the feature vectors of its projected 2D points are collected to form a subspace spanned by these vectors. The feature vectors from the query X-ray image are then compared with their projections onto each subspace. This process generates a heatmap that exhibits strong activation near the 2D location corresponding to the projected 3D landmark.
• Figure 2: Comparison of landmark detection results between conventional fixed landmark estimation and our arbitrary landmark estimation method. The 3D landmarks are shown in magenta on the left, while the estimated 2D landmarks are displayed in cyan and the ground truth in magenta on the right. Our method can generate a large number of corresponding pairs of 3D landmarks and 2D projections, whereas the fixed landmark estimation approach is limited to the available 2D projections and does not provide 3D landmark information.
• Figure 3: Box plots comparing the mean Target Registration Errors (mTRE) for six specimens using three different registration methods: DiffDRR, Fixed Landmark, and our proposed method, RayEmb (trained on DeepFluoro) and RayEmb* (trained on CTPelvic1K CLINIC). A red dashed line at 10 mm indicates the critical threshold for acceptable performance limit.
• Figure 4: Two example cases (Top and Bottom) where RayEmb provides better initial pose estimates compared to other methods. The ground truth landmarks are shown in magenta, and the estimated landmarks are in cyan (left). A 3D visualization of the initial pose estimate is displayed in green, the optimized pose in blue, and the ground truth pose in red (right).
• Figure 5: Examples of the estimated heatmaps (left) and the similarity score versus projection error plot (right). In the heatmaps, the cyan point indicates the predicted point, while the magenta point represents the corresponding ground truth. In the plot, red points correspond to sampling points outside the field of view, and blue points correspond to visible sampling points.