Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing the Traditional Chinese Medicine Capabilities of Large Language Model through Reinforcement Learning from AI Feedback

Song Yu, Xiaofei Xu, Fangfei Xu, Li Li

TL;DR

This work proposes a framework to improve the performance of large language models for TCM tasks using only a small amount of data, and uses medical case data for supervised fine-tuning of the large model, making it initially capable of performing TCM tasks.

Abstract

Although large language models perform well in understanding and responding to user intent, their performance in specialized domains such as Traditional Chinese Medicine (TCM) remains limited due to lack of expertise. In addition, high-quality data related to TCM is scarce and difficult to obtain, making large language models ineffective in handling TCM tasks. In this work, we propose a framework to improve the performance of large language models for TCM tasks using only a small amount of data. First, we use medical case data for supervised fine-tuning of the large model, making it initially capable of performing TCM tasks. Subsequently, we further optimize the model's performance using reinforcement learning from AI feedback (RLAIF) to align it with the preference data. The ablation study also demonstrated the performance gain is attributed to both supervised fine-tuning and the direct policy optimization. The experimental results show that the model trained with a small amount of data achieves a significant performance improvement on a representative TCM task.

Enhancing the Traditional Chinese Medicine Capabilities of Large Language Model through Reinforcement Learning from AI Feedback

TL;DR

This work proposes a framework to improve the performance of large language models for TCM tasks using only a small amount of data, and uses medical case data for supervised fine-tuning of the large model, making it initially capable of performing TCM tasks.

Abstract

Although large language models perform well in understanding and responding to user intent, their performance in specialized domains such as Traditional Chinese Medicine (TCM) remains limited due to lack of expertise. In addition, high-quality data related to TCM is scarce and difficult to obtain, making large language models ineffective in handling TCM tasks. In this work, we propose a framework to improve the performance of large language models for TCM tasks using only a small amount of data. First, we use medical case data for supervised fine-tuning of the large model, making it initially capable of performing TCM tasks. Subsequently, we further optimize the model's performance using reinforcement learning from AI feedback (RLAIF) to align it with the preference data. The ablation study also demonstrated the performance gain is attributed to both supervised fine-tuning and the direct policy optimization. The experimental results show that the model trained with a small amount of data achieves a significant performance improvement on a representative TCM task.

Paper Structure

This paper contains 21 sections, 3 equations, 2 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large language models
  4. Large Language Models in Medical
  5. Method
  6. Data Construction
  7. Supervised fine-tuning
  8. Reinforcement Learning from AI Feedback
  9. AI Feedback for TCM
  10. Reinforcement Learning
  11. Experiments and Evaluation
  12. Training Details
  13. Baseline
  14. Evaluation Metrics
  15. Results
  16. ...and 6 more sections

Figures (2)

  • Figure 1: The structure of our proposed framework includes data collection, supervised fine-tuning, automatic labeling, and direct preference optimization. After supervised fine-tuning, the model will generate multiple outputs, which are labeled using automatic labeling to obtain preference data. The model is further optimized using dpo and preference data.
  • Figure 2: Increasing the log probability of a preferred sample versus decreasing the log probability of a non-preferred sample response, the model's policy tends to select the preferred sample.