Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery

Yuyang Du, Kexin Chen, Yue Zhan, Chang Han Low, Tao You, Mobarakol Islam, Ziyu Guo, Yueming Jin, Guangyong Chen, Pheng-Ann Heng

TL;DR

A new multi-teacher CL framework that leverages a multimodal large language model (LLM) as the additional teacher and an adaptive weight assignment methodology is developed that balances the generalization ability of the LLM and the domain expertise of the old CL model.

Abstract

Visual question answering (VQA) is crucial for promoting surgical education. In practice, the needs of trainees are constantly evolving, such as learning more surgical types, adapting to different robots, and learning new surgical instruments and techniques for various surgeries. However, patient data privacy often restricts the availability of old data when updating the model, necessitating an exemplar-free continual learning (CL) setup. Prior CL studies overlooked two vital problems in the surgical domain: 1) large domain shifts from diverse surgical operations collected from multiple sources, and 2) severe data imbalance arising from the uneven presence of surgical instruments or activities. This paper proposes addressing these problems with a multimodal large language model (LLM) and an adaptive weight assignment methodology. We first develop a new multi-teacher CL framework that leverages a multimodal LLM as the additional teacher. The strong generalization ability of the LLM can bridge the knowledge gap when domain shifts and data imbalances occur. We then put forth a novel data processing method that transforms complex LLM embeddings into logits compatible with our CL framework. We further design an adaptive weight assignment approach that balances the generalization ability of the LLM and the domain expertise of the old CL model. Finally, to comprehensively test the effectiveness of our proposed method, we have also constructed two new surgical VQA datasets that are largely different from existing ones and could be valuable resources for future research. Extensive experimental results on the tested datasets demonstrate the superiority of our method to other advanced CL schemes.

LLM-Assisted Multi-Teacher Continual Learning for Visual Question Answering in Robotic Surgery

TL;DR

A new multi-teacher CL framework that leverages a multimodal large language model (LLM) as the additional teacher and an adaptive weight assignment methodology is developed that balances the generalization ability of the LLM and the domain expertise of the old CL model.

Abstract

Visual question answering (VQA) is crucial for promoting surgical education. In practice, the needs of trainees are constantly evolving, such as learning more surgical types, adapting to different robots, and learning new surgical instruments and techniques for various surgeries. However, patient data privacy often restricts the availability of old data when updating the model, necessitating an exemplar-free continual learning (CL) setup. Prior CL studies overlooked two vital problems in the surgical domain: 1) large domain shifts from diverse surgical operations collected from multiple sources, and 2) severe data imbalance arising from the uneven presence of surgical instruments or activities. This paper proposes addressing these problems with a multimodal large language model (LLM) and an adaptive weight assignment methodology. We first develop a new multi-teacher CL framework that leverages a multimodal LLM as the additional teacher. The strong generalization ability of the LLM can bridge the knowledge gap when domain shifts and data imbalances occur. We then put forth a novel data processing method that transforms complex LLM embeddings into logits compatible with our CL framework. We further design an adaptive weight assignment approach that balances the generalization ability of the LLM and the domain expertise of the old CL model. Finally, to comprehensively test the effectiveness of our proposed method, we have also constructed two new surgical VQA datasets that are largely different from existing ones and could be valuable resources for future research. Extensive experimental results on the tested datasets demonstrate the superiority of our method to other advanced CL schemes.
Paper Structure (11 sections, 12 equations, 4 figures, 2 tables)

This paper contains 11 sections, 12 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Preliminaries
  4. Multi-teacher CL Framework with LLM
  5. Adaptive Weight Assignment
  6. Experiments and Analysis
  7. Dataset
  8. Baselines and Implementation Details
  9. Experimental Results
  10. Why LLM Works: Discussion and Case Study
  11. Conclusion and Future Work

Figures (4)

  • Figure 1: The proposed LLM-assisted multi-teacher CL framework. The model is used to process bimodal input (text and image) and provide predictions for the VQA task. The proposed weight adaption scheme (highlighted in the light orange zone) is designed to trade off the general knowledge of the LLM and the medical expertise of the previous CL model. The frozen LLM is highlighted in light red, while the conventional teacher and student models are highlighted in light blue. Ground truth lies in the light green zone.
  • Figure 2: The workflow of logits transformation after embeddings.
  • Figure 3: Logits of CL teacher, LLM teacher, and the student for the case in Fig. \ref{['Fig_6']}.
  • Figure 4: Two EndoVis18 images showing the cutting operation on a kidney. If a conventional CL approach were applied, the VQA model at $t=2$ would misclassify the second operation in (b) as "idle".