Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging AI Predicted and Expert Revised Annotations in Interactive Segmentation: Continual Tuning or Full Training?

Tiezheng Zhang, Xiaoxi Chen, Chongyu Qu, Alan Yuille, Zongwei Zhou

TL;DR

Continual Tuning is proposed to address the problems from two perspectives: network design and data reuse and demonstrates that Continual Tuning achieves a speed 16× greater than repeatedly training AI from scratch without compromising the performance.

Abstract

Interactive segmentation, an integration of AI algorithms and human expertise, premises to improve the accuracy and efficiency of curating large-scale, detailed-annotated datasets in healthcare. Human experts revise the annotations predicted by AI, and in turn, AI improves its predictions by learning from these revised annotations. This interactive process continues to enhance the quality of annotations until no major revision is needed from experts. The key challenge is how to leverage AI predicted and expert revised annotations to iteratively improve the AI. Two problems arise: (1) The risk of catastrophic forgetting--the AI tends to forget the previously learned classes if it is only retrained using the expert revised classes. (2) Computational inefficiency when retraining the AI using both AI predicted and expert revised annotations; moreover, given the dominant AI predicted annotations in the dataset, the contribution of newly revised annotations--often account for a very small fraction--to the AI training remains marginal. This paper proposes Continual Tuning to address the problems from two perspectives: network design and data reuse. Firstly, we design a shared network for all classes followed by class-specific networks dedicated to individual classes. To mitigate forgetting, we freeze the shared network for previously learned classes and only update the class-specific network for revised classes. Secondly, we reuse a small fraction of data with previous annotations to avoid over-computing. The selection of such data relies on the importance estimate of each data. The importance score is computed by combining the uncertainty and consistency of AI predictions. Our experiments demonstrate that Continual Tuning achieves a speed 16x greater than repeatedly training AI from scratch without compromising the performance.

Leveraging AI Predicted and Expert Revised Annotations in Interactive Segmentation: Continual Tuning or Full Training?

TL;DR

Continual Tuning is proposed to address the problems from two perspectives: network design and data reuse and demonstrates that Continual Tuning achieves a speed 16× greater than repeatedly training AI from scratch without compromising the performance.

Abstract

Interactive segmentation, an integration of AI algorithms and human expertise, premises to improve the accuracy and efficiency of curating large-scale, detailed-annotated datasets in healthcare. Human experts revise the annotations predicted by AI, and in turn, AI improves its predictions by learning from these revised annotations. This interactive process continues to enhance the quality of annotations until no major revision is needed from experts. The key challenge is how to leverage AI predicted and expert revised annotations to iteratively improve the AI. Two problems arise: (1) The risk of catastrophic forgetting--the AI tends to forget the previously learned classes if it is only retrained using the expert revised classes. (2) Computational inefficiency when retraining the AI using both AI predicted and expert revised annotations; moreover, given the dominant AI predicted annotations in the dataset, the contribution of newly revised annotations--often account for a very small fraction--to the AI training remains marginal. This paper proposes Continual Tuning to address the problems from two perspectives: network design and data reuse. Firstly, we design a shared network for all classes followed by class-specific networks dedicated to individual classes. To mitigate forgetting, we freeze the shared network for previously learned classes and only update the class-specific network for revised classes. Secondly, we reuse a small fraction of data with previous annotations to avoid over-computing. The selection of such data relies on the importance estimate of each data. The importance score is computed by combining the uncertainty and consistency of AI predictions. Our experiments demonstrate that Continual Tuning achieves a speed 16x greater than repeatedly training AI from scratch without compromising the performance.
Paper Structure (7 sections, 1 equation, 2 figures, 2 tables)

This paper contains 7 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiments & Results & Discussion
  4. Continual Tuning models pre-trained on one dataset
  5. Continual Tuning models pre-trained on 14 datasets
  6. Continual Tuning: Impact on Model Scales
  7. Conclusion

Figures (2)

  • Figure 1: (a) Catastrophic gorgetting in Swin UNETR tang2022self and U-Net ronneberger2015u backbones. The old classes will be forgotten at the first few epochs when continual training AI models on data of new classes. (b) Comparison of Continual Tuning and Full Training. Two lines illustrate the mean DSC score using Continual Tuning method, while the asterisks show the final DSC score when applying Full Training. (c) Shared Networks with Class-Specific Extensions. The figure shows the networks we use, and we take the stomach as an example of the new class. (d) Results of Continual Tuning on Two Rounds. The blue region represents first-round results of Continual Tuning, and the red region, the second-round results.
  • Figure 2: Examples of Hybrid Data. In the upper row, the revised annotations for gall bladder, postcava (IVC), and stomach & aorta are presented from left to right. The lower row displays the corresponding hybrid annotations with old classes (liver, pancreas, left kidney, right kidney, and spleen).