Knowledge-based learning in Text-RAG and Image-RAG
Alexander Shim, Khalil Saieh, Samuel Clarke
TL;DR
This paper addresses hallucination and reliability in multi-modal chest X-ray interpretation by evaluating retrieval-augmented generation (RAG) strategies. It compares image-based RAG and text-based RAG using an EVA-ViT image encoder with GPT-2 mini and LLaMA-2 LLMs on the NIH Chest X-ray dataset, incorporating preprocessing techniques to mitigate class imbalance. The study finds image-based RAG offers stronger stability and calibration, while text-based RAG provides contextual grounding with variable hallucination behavior; GPT-2 mini generally performs better than LLaMA-3 under the tested conditions. Overall, the work highlights the trade-offs between grounding sources and model choices, suggesting that balanced data and hardware scalability are key to realizing reliable, clinically useful multi-modal radiology AI.
Abstract
This research analyzed and compared the multi-modal approach in the Vision Transformer(EVA-ViT) based image encoder with the LlaMA or ChatGPT LLM to reduce the hallucination problem and detect diseases in chest x-ray images. In this research, we utilized the NIH Chest X-ray image to train the model and compared it in image-based RAG, text-based RAG, and baseline. [3] [5] In a result, the text-based RAG[2] e!ectively reduces the hallucination problem by using external knowledge information, and the image-based RAG improved the prediction con"dence and calibration by using the KNN methods. [4] Moreover, the GPT LLM showed better performance, a low hallucination rate, and better Expected Calibration Error(ECE) than Llama Llama-based model. This research shows the challenge of data imbalance, a complex multi-stage structure, but suggests a large experience environment and a balanced example of use.