Morphology-Aware Interactive Keypoint Estimation

TL;DR

This work addresses the bottleneck of manual anatomical keypoint annotation in medical X-ray images by introducing an interactive framework that enables clinicians to correct a small subset of predictions and automatically revise the remaining keypoints. It combines a morphology-aware loss and an interaction-guided gating mechanism within an HRNet-based heatmap regression network to propagate user corrections across the image and preserve inter-keypoint geometry. The key contributions are the interactive keypoint estimation network, the interaction-guided gating network, and the morphology-aware loss, validated on Cephalometric X-ray and AASCE datasets, showing improved efficiency and accuracy over baselines. The approach has practical implications for reducing annotation costs and supporting diagnostic workflows with human-in-the-loop guidance, though it assumes correct user edits and points to future work on handling noisy inputs and guidance strategies for revision order.

Abstract

Diagnosis based on medical images, such as X-ray images, often involves manual annotation of anatomical keypoints. However, this process involves significant human efforts and can thus be a bottleneck in the diagnostic process. To fully automate this procedure, deep-learning-based methods have been widely proposed and have achieved high performance in detecting keypoints in medical images. However, these methods still have clinical limitations: accuracy cannot be guaranteed for all cases, and it is necessary for doctors to double-check all predictions of models. In response, we propose a novel deep neural network that, given an X-ray image, automatically detects and refines the anatomical keypoints through a user-interactive system in which doctors can fix mispredicted keypoints with fewer clicks than needed during manual revision. Using our own collected data and the publicly available AASCE dataset, we demonstrate the effectiveness of the proposed method in reducing the annotation costs via extensive quantitative and qualitative results. A demo video of our approach is available on our project webpage.

Figures (7)

• Figure 1: Interactive keypoint estimation results on a cephalometric X-ray image. The goal is to estimate (a) 13 keypoints on the cervical vertebrae, each of which determines vertebrae morphology. Here, (b) the initial prediction misses one vertebra at the top, making the entire prediction wrong. A manual revision will be no better than annotating from scratch. However, in our method, if (c) a user corrects only one point, (d) the remaining points come up together. All keypoints appropriately reflect user-interaction information in (e) the final result.
• Figure 2: Overview of proposed interactive keypoint estimation model. It receives an image, user interaction (User inter.), and its previous prediction (Prev. pred.) and outputs a heatmap of keypoint locations that reflects interactive user feedback.
• Figure 3: Illustration of Morphology-aware loss.
• Figure 4: Comparison with manual revision on AASCE.
• Figure 5: Qualitative interactive keypoint estimation results on AASCE. To visualize the prediction error, we draw a line between the predicted keypoints and the corresponding groundtruth keypoints; the shorter, the better. Given user feedback, the area where the green lines are dominant is where errors are significantly reduced compared to initial predictions. Initial, initial prediction error; After, prediction error after one user modification; $\Delta$, Initial minus After.