Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant

Yifan Xiang, Zhenxi Zhang, Bin Li, Yixuan Weng, Shoujun Zhou, Yangfan He, Keqin Li

TL;DR

ReGraP-LLaVA tackles relational reasoning over personalized knowledge in multimodal settings by coupling knowledge graphs with chain-of-thought data. It introduces ReGraP, a dataset of images, KGs, and CoT QA pairs across 120 personalized knowledge sets, and ReGraP-LLaVA, a training framework that uses both soft graph prompts via GNN+MLP embeddings and hard graph prompts via reasoning tokens. The authors establish the ReGraP Benchmark to evaluate relational reasoning and knowledge connectivity, and demonstrate state-of-the-art performance on both close- and open-ended tasks compared to multiple baselines. The work advances personalized MLLMs by enabling learning of relational knowledge and its reasoning over personalized concepts, with code and data released for reproducibility.

Abstract

Recent advances in personalized MLLMs enable effective capture of user-specific concepts, supporting both recognition of personalized concepts and contextual captioning. However, humans typically explore and reason over relations among objects and individuals, transcending surface-level information to achieve more personalized and contextual understanding. To this end, existing methods may face three main limitations: Their training data lacks multi-object sets in which relations among objects are learnable. Building on the limited training data, their models overlook the relations between different personalized concepts and fail to reason over them. Their experiments mainly focus on a single personalized concept, where evaluations are limited to recognition and captioning tasks. To address the limitations, we present a new dataset named ReGraP, consisting of 120 sets of personalized knowledge. Each set includes images, KGs, and CoT QA pairs derived from the KGs, enabling more structured and sophisticated reasoning pathways. We propose ReGraP-LLaVA, an MLLM trained with the corresponding KGs and CoT QA pairs, where soft and hard graph prompting methods are designed to align KGs within the model's semantic space. We establish the ReGraP Benchmark, which contains diverse task types: multiple-choice, fill-in-the-blank, True/False, and descriptive questions in both open- and closed-ended settings. The proposed benchmark is designed to evaluate the relational reasoning and knowledge-connection capability of personalized MLLMs. We conduct experiments on the proposed ReGraP-LLaVA and other competitive MLLMs. Results show that the proposed model not only learns personalized knowledge but also performs relational reasoning in responses, achieving the SoTA performance compared with the competitive methods. All the codes and datasets are released at: https://github.com/xyfyyds/ReGraP.

ReGraP-LLaVA: Reasoning enabled Graph-based Personalized Large Language and Vision Assistant

TL;DR

ReGraP-LLaVA tackles relational reasoning over personalized knowledge in multimodal settings by coupling knowledge graphs with chain-of-thought data. It introduces ReGraP, a dataset of images, KGs, and CoT QA pairs across 120 personalized knowledge sets, and ReGraP-LLaVA, a training framework that uses both soft graph prompts via GNN+MLP embeddings and hard graph prompts via reasoning tokens. The authors establish the ReGraP Benchmark to evaluate relational reasoning and knowledge connectivity, and demonstrate state-of-the-art performance on both close- and open-ended tasks compared to multiple baselines. The work advances personalized MLLMs by enabling learning of relational knowledge and its reasoning over personalized concepts, with code and data released for reproducibility.

Abstract

Recent advances in personalized MLLMs enable effective capture of user-specific concepts, supporting both recognition of personalized concepts and contextual captioning. However, humans typically explore and reason over relations among objects and individuals, transcending surface-level information to achieve more personalized and contextual understanding. To this end, existing methods may face three main limitations: Their training data lacks multi-object sets in which relations among objects are learnable. Building on the limited training data, their models overlook the relations between different personalized concepts and fail to reason over them. Their experiments mainly focus on a single personalized concept, where evaluations are limited to recognition and captioning tasks. To address the limitations, we present a new dataset named ReGraP, consisting of 120 sets of personalized knowledge. Each set includes images, KGs, and CoT QA pairs derived from the KGs, enabling more structured and sophisticated reasoning pathways. We propose ReGraP-LLaVA, an MLLM trained with the corresponding KGs and CoT QA pairs, where soft and hard graph prompting methods are designed to align KGs within the model's semantic space. We establish the ReGraP Benchmark, which contains diverse task types: multiple-choice, fill-in-the-blank, True/False, and descriptive questions in both open- and closed-ended settings. The proposed benchmark is designed to evaluate the relational reasoning and knowledge-connection capability of personalized MLLMs. We conduct experiments on the proposed ReGraP-LLaVA and other competitive MLLMs. Results show that the proposed model not only learns personalized knowledge but also performs relational reasoning in responses, achieving the SoTA performance compared with the competitive methods. All the codes and datasets are released at: https://github.com/xyfyyds/ReGraP.
Paper Structure (23 sections, 6 equations, 6 figures, 37 tables)

This paper contains 23 sections, 6 equations, 6 figures, 37 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. ReGraP Dataset: Data Generation Pipeline
  4. ReGraP-LLaVA: Training Framework
  5. Soft-Prompting LLM with Graph.
  6. Hard-Prompting LLM with Graph.
  7. Experimental Setup
  8. Results
  9. Close-ended QA
  10. Open-ended QA
  11. Ablation Study
  12. Case Study
  13. Conclusion
  14. Additional Ablation Studies
  15. Human Evaluation
  16. ...and 8 more sections

Figures (6)

  • Figure 1: The comparison between ReGraP-LLaVA and other personalized MLLMs.
  • Figure 2: The data generation pipeline. We first construct knowledge graph that represents the personalized knowledge, and then derive CoT QA pairs from the knowledge graph.
  • Figure 3: The framework of ReGraP-LLaVA. The left side shows the framework to soft prompt graphs, and the right side shows the framework to hard prompt graphs.
  • Figure 4: The ablation study of the number of personalized objects.
  • Figure 5: The ablation study of the length of answers in CoT QA pairs.
  • ...and 1 more figures