Integrating Physician Diagnostic Logic into Large Language Models: Preference Learning from Process Feedback

TL;DR

This work addresses the challenge of maintaining logical, physician-consistent dialogue in medical LLMs across multiple rounds. It introduces Preference Learning from Process Feedback (PLPF), a three-phase framework combining flowchart-based rule modeling, Rule Evaluation Model (REM) scoring, and preference alignment via Direct Preference Optimization to train models that follow diagnostic processes. Through the CSPT dataset and retrieval-augmented patient simulators, PLPF achieves a notable 17.6% improvement in diagnostic accuracy over baselines and demonstrates robustness in both multi-round and single-round settings. The framework offers a principled path to safer, more reliable medical dialogue by encoding clinical reasoning into the training signal and may inform future medical AI deployments where structured clinical workflows matter.

Abstract

The use of large language models in medical dialogue generation has garnered significant attention, with a focus on improving response quality and fluency. While previous studies have made progress in optimizing model performance for single-round medical Q&A tasks, there is a need to enhance the model's capability for multi-round conversations to avoid logical inconsistencies. To address this, we propose an approach called preference learning from process feedback~(PLPF), which integrates the doctor's diagnostic logic into LLMs. PLPF involves rule modeling, preference data generation, and preference alignment to train the model to adhere to the diagnostic process. Experimental results using Standardized Patient Testing show that PLPF enhances the diagnostic accuracy of the baseline model in medical conversations by 17.6%, outperforming traditional reinforcement learning from human feedback. Additionally, PLPF demonstrates effectiveness in both multi-round and single-round dialogue tasks, showcasing its potential for improving medical dialogue generation.

Figures (15)

• Figure 1: Medical diagnosis flowchart (left) and its corresponding rules (right). In the flowchart, we use blue boxes for activities, orange diamonds for judgment conditions, and gray boxes for additional constraints. We use the letters A-F to indicate the correspondence between the rules and the elements in the flowchart.
• Figure 2: Overview of the training process. The training process is divided into three steps, with key activities indicated using orange rounded rectangular boxes. To distinguish the different stages of the data, we labeled them with different colors and provided data descriptions in the upper right corner of the image.
• Figure 3: Ranking of how well each model follows the rules. Each axis of the radar graph corresponds to a rule in Fig. \ref{['fig:flow']}, and we use the letters A-F to denote the mapping between rules and axes.
• Figure 4: The average output length of LLMs over different datasets, where Goden represents the average length of the standard answer.
• Figure 5: Patient Simulator Architecture.