Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach

TL;DR

This work addresses secure, fresh data management in IoMT by integrating multi-modal LLMs with a hybrid retrieval-augmented framework. It combines cross-chain data training, hybrid multi-modal RAG with MIS filtering, AoI-based data quality metrics, and contract theory to incentivize high-quality, timely data sharing, solved via a generative diffusion model-based contract design. The approach yields substantial gains in MLLM output quality and data-sharing efficiency, and demonstrates improved security via PBFT consensus as network scale increases. The results highlight a practical pathway for robust, privacy-preserving IoMT data management with incentive-compatible data sharing and diffusion-enabled contract optimization.

Abstract

Secure data management and effective data sharing have become paramount in the rapidly evolving healthcare landscape, especially with the growing integration of the Internet of Medical Things (IoMT). The rise of generative artificial intelligence has further elevated Multi-modal Large Language Models (MLLMs) as essential tools for managing and optimizing healthcare data in IoMT. MLLMs can support multi-modal inputs and generate diverse types of content by leveraging large-scale training on vast amounts of multi-modal data. However, critical challenges persist in developing medical MLLMs, including security and freshness issues of healthcare data, affecting the output quality of MLLMs. To this end, in this paper, we propose a hybrid Retrieval-Augmented Generation (RAG)-empowered medical MLLM framework for healthcare data management. This framework leverages a hierarchical cross-chain architecture to facilitate secure data training. Moreover, it enhances the output quality of MLLMs through hybrid RAG, which employs multi-modal metrics to filter various unimodal RAG results and incorporates these retrieval results as additional inputs to MLLMs. Additionally, we employ age of information to indirectly evaluate the data freshness impact of MLLMs and utilize contract theory to incentivize healthcare data holders to share their fresh data, mitigating information asymmetry during data sharing. Finally, we utilize a generative diffusion model-based deep reinforcement learning algorithm to identify the optimal contract for efficient data sharing. Numerical results demonstrate the effectiveness of the proposed schemes, which achieve secure and efficient healthcare data management.

Figures (7)

• Figure 1: The overview of the hybrid RAG-empowered medical MLLM framework for healthcare data management in IoMT. Part A shows the cross-chain interaction for secure healthcare data sharing. Part B depicts the processes of multi-modal input optimization based on a hybrid multi-modal RAG module. Part C presents the framework of MLLM inference based on the multi-modal healthcare data.
• Figure 2: A real case study of hybrid RAG-empowered medical MLLMs. In the proposed hybrid RAG-empowered medical MLLM, the RAG initially retrieves healthcare data using unimodal methods. Next, we re-rank the information using metrics such as structural similarity index measurewang2020deep, normalized cross-correlation, and bert scorebert-score. The detailed information is then combined with the task query and input into the MLLM to generate results.
• Figure 3: Performance comparison between the proposed framework under different healthcare data cases. Note that the initial two users provide conditional healthcare data cases, while the subsequent three users provide normal healthcare data cases.
• Figure 4: Reward comparison of our scheme with other schemes, i.e., contract-based incentive mechanism with complete information, greedy, and random.
• Figure 5: Performance comparison between the GDM and DRL-PPO in optimal contract design.

Theorems & Definitions (2)

• Definition 1
• Definition 2