MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering

TL;DR

This work addresses multimodal physics question answering for Indian high school level problems by integrating image captioning and reinforcement learning from human feedback into LLaVA based models. The MM-PhyQA dataset serves as the domain foundation, with a 8000 sample preference dataset created via multi-model responses ranked by Gemini Pro to train a Reward Model and apply PPO updates. Image captions generated by Infi-MM enrich diagrams and reduce hallucinations, while RLHF aligns model reasoning with human preferences. Empirical results show that combining image captioning with full multimodal input yields the highest accuracy, and RLHF further enhances performance, highlighting the approach as a promising step toward effective educational multimodal AI assistants.

Abstract

Recent advancements in LLMs have shown their significant potential in tasks like text summarization and generation. Yet, they often encounter difficulty while solving complex physics problems that require arithmetic calculation and a good understanding of concepts. Moreover, many physics problems include images that contain important details required to understand the problem's context. We propose an LMM-based chatbot to answer multimodal physics MCQs. For domain adaptation, we utilize the MM-PhyQA dataset comprising Indian high school-level multimodal physics problems. To improve the LMM's performance, we experiment with two techniques, RLHF (Reinforcement Learning from Human Feedback) and Image Captioning. In image captioning, we add a detailed explanation of the diagram in each image, minimizing hallucinations and image processing errors. We further explore the integration of Reinforcement Learning from Human Feedback (RLHF) methodology inspired by the ranking approach in RLHF to enhance the human-like problem-solving abilities of the models. The RLHF approach incorporates human feedback into the learning process of LLMs, improving the model's problem-solving skills, truthfulness, and reasoning capabilities, minimizing the hallucinations in the answers, and improving the quality instead of using vanilla-supervised fine-tuned models. We employ the LLaVA open-source model to answer multimodal physics MCQs and compare the performance with and without using RLHF.

Figures (1)

• Figure 1: Preference dataset Pipeline