NOISe: Nuclei-Aware Osteoclast Instance Segmentation for Mouse-to-Human Domain Transfer

TL;DR

The paper tackles the bottleneck of manual, annotation-heavy osteoclast analysis by introducing NOISe, a nuclei-aware, end-to-end instance segmentation framework built on YOLOv8l-seg. It provides the first public dataset of ~2×10^5 mouse osteoclast masks and ~4×10^4 human masks, plus a small nuclei dataset used to pretrain a detector that injects biological priors into segmentation. NOISe demonstrates strong cross-domain transfer from mouse to human, achieving a $mAP_{0.5}$ of 0.82 on mouse data and improved human segmentation compared to a mouse-only baseline, highlighting the value of nuclei-aware pretraining for domain generalization. The work offers public code, pretrained weights, and benchmarks to accelerate osteoporosis research and paves the way for broader domain adaptation in cellular image analysis.

Abstract

Osteoclast cell image analysis plays a key role in osteoporosis research, but it typically involves extensive manual image processing and hand annotations by a trained expert. In the last few years, a handful of machine learning approaches for osteoclast image analysis have been developed, but none have addressed the full instance segmentation task required to produce the same output as that of the human expert led process. Furthermore, none of the prior, fully automated algorithms have publicly available code, pretrained models, or annotated datasets, inhibiting reproduction and extension of their work. We present a new dataset with ~2*10^5 expert annotated mouse osteoclast masks, together with a deep learning instance segmentation method which works for both in vitro mouse osteoclast cells on plastic tissue culture plates and human osteoclast cells on bone chips. To our knowledge, this is the first work to automate the full osteoclast instance segmentation task. Our method achieves a performance of 0.82 mAP_0.5 (mean average precision at intersection-over-union threshold of 0.5) in cross validation for mouse osteoclasts. We present a novel nuclei-aware osteoclast instance segmentation training strategy (NOISe) based on the unique biology of osteoclasts, to improve the model's generalizability and boost the mAP_0.5 from 0.60 to 0.82 on human osteoclasts. We publish our annotated mouse osteoclast image dataset, instance segmentation models, and code at github.com/michaelwwan/noise to enable reproducibility and to provide a public tool to accelerate osteoporosis research.

Figures (7)

• Figure 1: Human osteoclast cell segmentations from our nuclei-aware osteoclast instance segmentation (NOISe) model, compared to a baseline YOLOv8 model. Since both models are trained only on mouse osteoclast data, this highlights the effectiveness of our nuclei-aware pretraining strategy for transfer learning to the human domain, where osteoclast microscope samples are harder to obtain and annotate.
• Figure 2: An illustration of an osteoclast cell in purple acting on bone structure in light brown. Osteoclasts are responsible for bone-resorption, and are characterized by having three or more nuclei and being positive for purple TRAP enzyme stain.
• Figure 3: An overview of NOISe, our nuclei-aware osteoclast instance segmentation training pipeline. A two-stage training process features a pretraining stage for multiclass detection weakly supervised by nuclei location information. The pretraining significantly boosts subsequent performance of the overall osteoclast instance segmentation model, especially in the data-scarce human domain.
• Figure 4: Detail from a mouse osteoclast microscope image, together with weak ground truth nuclei labels (note the uniform size and shape of the boxes), and YOLOv8 object detection predictions of the same. Our nuclei-aware training method exploits this weak ground truth information to improve the generalizability of osteoclast instance segmentation.
• Figure 5: Cell images and cropped patches from mouse experiments M1 through M5, and human experiments H1 and H2, illustrating the diversity in slide lighting, background, and appearance, and in osteoclast size, shape, and density in our data.