Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Prompt Optimization to Multi-Dimensional Credibility Evaluation: Enhancing Trustworthiness of Chinese LLM-Generated Liver MRI Reports

Qiuli Wang, Jie Chen, Yongxu Liu, Xingpeng Zhang, Xiaoming Li, Wei Chen

TL;DR

The paper addresses the credibility of Chinese LLM-generated liver MRI reports by introducing a model-agnostic Multi-Dimensional Credibility Assessment (MDCA) framework that evaluates Semantic Coherence ($SC$), Diagnostic Correctness ($DC$), and Clinical Prioritization Alignment ($CPA$) with the overall score $MDCA = 0.2×SC + 0.4×DC + 0.4×CPA$. It then combines a comprehensive prompt-design protocol (two-component prompts and 11 configurations) with evaluation across multiple Chinese LLMs on the SiliconFlow platform to identify practical prompts that enhance trustworthiness. The study finds that instruction-rich prompts plus a moderate number of examples (approximately 10–15) yield the best balance of fluency, accuracy, and clinically relevant prioritization, with Kimi-K2-Instruct-0905 and DeepSeek-V3 providing the strongest overall credibility. The MDCA framework proves model-agnostic and correlates with radiologist judgments, highlighting its potential for automated quality control and trainee education in radiology workflows, while also noting the need for multicenter validation and future multimodal extensions.

Abstract

Large language models (LLMs) have demonstrated promising performance in generating diagnostic conclusions from imaging findings, thereby supporting radiology reporting, trainee education, and quality control. However, systematic guidance on how to optimize prompt design across different clinical contexts remains underexplored. Moreover, a comprehensive and standardized framework for assessing the trustworthiness of LLM-generated radiology reports is yet to be established. This study aims to enhance the trustworthiness of LLM-generated liver MRI reports by introducing a Multi-Dimensional Credibility Assessment (MDCA) framework and providing guidance on institution-specific prompt optimization. The proposed framework is applied to evaluate and compare the performance of several advanced LLMs, including Kimi-K2-Instruct-0905, Qwen3-235B-A22B-Instruct-2507, DeepSeek-V3, and ByteDance-Seed-OSS-36B-Instruct, using the SiliconFlow platform.

From Prompt Optimization to Multi-Dimensional Credibility Evaluation: Enhancing Trustworthiness of Chinese LLM-Generated Liver MRI Reports

TL;DR

The paper addresses the credibility of Chinese LLM-generated liver MRI reports by introducing a model-agnostic Multi-Dimensional Credibility Assessment (MDCA) framework that evaluates Semantic Coherence (), Diagnostic Correctness (), and Clinical Prioritization Alignment () with the overall score . It then combines a comprehensive prompt-design protocol (two-component prompts and 11 configurations) with evaluation across multiple Chinese LLMs on the SiliconFlow platform to identify practical prompts that enhance trustworthiness. The study finds that instruction-rich prompts plus a moderate number of examples (approximately 10–15) yield the best balance of fluency, accuracy, and clinically relevant prioritization, with Kimi-K2-Instruct-0905 and DeepSeek-V3 providing the strongest overall credibility. The MDCA framework proves model-agnostic and correlates with radiologist judgments, highlighting its potential for automated quality control and trainee education in radiology workflows, while also noting the need for multicenter validation and future multimodal extensions.

Abstract

Large language models (LLMs) have demonstrated promising performance in generating diagnostic conclusions from imaging findings, thereby supporting radiology reporting, trainee education, and quality control. However, systematic guidance on how to optimize prompt design across different clinical contexts remains underexplored. Moreover, a comprehensive and standardized framework for assessing the trustworthiness of LLM-generated radiology reports is yet to be established. This study aims to enhance the trustworthiness of LLM-generated liver MRI reports by introducing a Multi-Dimensional Credibility Assessment (MDCA) framework and providing guidance on institution-specific prompt optimization. The proposed framework is applied to evaluate and compare the performance of several advanced LLMs, including Kimi-K2-Instruct-0905, Qwen3-235B-A22B-Instruct-2507, DeepSeek-V3, and ByteDance-Seed-OSS-36B-Instruct, using the SiliconFlow platform.
Paper Structure (17 sections, 4 equations, 6 figures, 4 tables)

This paper contains 17 sections, 4 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Materials and Methods
  3. Collection of Clinical Reports
  4. Multi-Dimensional Credibility Assessment Framework
  5. Prompt Design and Implementation
  6. Preparation of Large Language Models
  7. Results
  8. Effect of Prompt Composition on Model Performance
  9. Effect of Example Number on Model Performance
  10. Comparison of Chinese LLMs and Validation within the DeepSeek Family
  11. Consistency Check between MDCA Framework and Radiologist Judgments
  12. Discussion
  13. Prompt Design Principles and Mechanistic Insights
  14. Model Performance and Selection Considerations
  15. Practical Applications and Clinical Implications
  16. ...and 2 more sections

Figures (6)

  • Figure 1: The collection process of clinical reports. This process followed a well-designed exclusion criteria to ensure the quality and relevance of the reports used for LLM evaluation.
  • Figure 2: Experimental results for Prompt Designs with KIMI-K2 and DeepSeek-V3. The figure illustrates the performance of both models across various prompt configurations, highlighting the impact of different prompt components on Semantic Coherence (SC), Diagnostic Correctness (DC), Top-1 matching, Clinical Prioritization Alignment (CPA), and overall MDCA scores.
  • Figure 3: Experimental Results with Different Example Numbers. The figure illustrates how varying the number of example reports in the prompts affects the performance of KIMI-K2 and DeepSeek-V3 across multiple evaluation metrics, including Semantic Coherence (SC), Diagnostic Correctness (DC), Top-1 matching, Clinical Prioritization Alignment (CPA), and overall MDCA scores.
  • Figure 4: Experimental Results with Four LLMs. The figure illustrates the performance of KIMI-K2, DeepSeek-V3, ByteDance-Seed, and Qwen3-235B across various prompt configurations.
  • Figure 5: Experimental Results with DeepSeek Family. The figure illustrates the performance of DeepSeek-V3, DeepSeek-V3.1, and DeepSeek-R1 across various prompt configurations.
  • ...and 1 more figures