Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images

Yichi Zhang, Zhuo Chen, Lingbing Guo, Lei Liang, Wen Zhang, Huajun Chen

TL;DR

This work addresses the challenge of structured and abstractive reasoning over images that convey multi-modal relational knowledge (MMRK). It introduces a STAR Data Engine that synthesizes MMRK-labeled images from multi-modal knowledge graphs and generates multi-task instruction data with detailed chain-of-thought reasoning, forming the STAR-64K dataset. A two-stage capability enhancement training framework—comprising supervised fine-tuning and preference alignment—substantially improves STAR performance across multiple open-source MLLMs, with smaller models sometimes surpassing larger proprietary systems in STAR tasks. Through extensive experiments, the authors analyze transferability, data scalability, and design choices, demonstrating that targeted data synthesis and training can unlock robust STAR capabilities while preserving commonsense knowledge.

Abstract

Understanding and reasoning with abstractive information from the visual modality presents significant challenges for current multi-modal large language models (MLLMs). Among the various forms of abstractive information, Multi-Modal Relational Knowledge (MMRK), which represents abstract relational structures between multi-modal entities using node-edge formats, remains largely under-explored. In particular, STructured and Abstractive Reasoning (STAR) on such data has received little attention from the research community. To bridge the dual gaps in large-scale high-quality data and capability enhancement methodologies, this paper makes the following key contributions: (i). An automatic STAR data engine capable of synthesizing images with MMRK to build multi-modal instruction data with reliable chain-of-thought thinking for various STAR tasks and (ii). A comprehsive two-stage capability enhancement training framework, accompanied by a suite of evaluation protocols tailored to different STAR tasks. Based upon these contributions, we introduce STAR-64K, a dataset comprising 64K high-quality multi-modal instruction samples, and conduct experiments across 5 open-source MLLMs. Experimental results show that our two-stage enhancement framework enables smaller 3B/7B models to significantly outperform GPT-4o in STAR. Additionally, we provide in-depth analysis regarding the effectiveness of various designs, data transferability, and scalability.

Structured and Abstractive Reasoning on Multi-modal Relational Knowledge Images

TL;DR

This work addresses the challenge of structured and abstractive reasoning over images that convey multi-modal relational knowledge (MMRK). It introduces a STAR Data Engine that synthesizes MMRK-labeled images from multi-modal knowledge graphs and generates multi-task instruction data with detailed chain-of-thought reasoning, forming the STAR-64K dataset. A two-stage capability enhancement training framework—comprising supervised fine-tuning and preference alignment—substantially improves STAR performance across multiple open-source MLLMs, with smaller models sometimes surpassing larger proprietary systems in STAR tasks. Through extensive experiments, the authors analyze transferability, data scalability, and design choices, demonstrating that targeted data synthesis and training can unlock robust STAR capabilities while preserving commonsense knowledge.

Abstract

Understanding and reasoning with abstractive information from the visual modality presents significant challenges for current multi-modal large language models (MLLMs). Among the various forms of abstractive information, Multi-Modal Relational Knowledge (MMRK), which represents abstract relational structures between multi-modal entities using node-edge formats, remains largely under-explored. In particular, STructured and Abstractive Reasoning (STAR) on such data has received little attention from the research community. To bridge the dual gaps in large-scale high-quality data and capability enhancement methodologies, this paper makes the following key contributions: (i). An automatic STAR data engine capable of synthesizing images with MMRK to build multi-modal instruction data with reliable chain-of-thought thinking for various STAR tasks and (ii). A comprehsive two-stage capability enhancement training framework, accompanied by a suite of evaluation protocols tailored to different STAR tasks. Based upon these contributions, we introduce STAR-64K, a dataset comprising 64K high-quality multi-modal instruction samples, and conduct experiments across 5 open-source MLLMs. Experimental results show that our two-stage enhancement framework enables smaller 3B/7B models to significantly outperform GPT-4o in STAR. Additionally, we provide in-depth analysis regarding the effectiveness of various designs, data transferability, and scalability.
Paper Structure (40 sections, 2 equations, 13 figures, 3 tables)

This paper contains 40 sections, 2 equations, 13 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multi-modal Large Language Model (MLLM) Enhancement and Evaluation
  4. The STAR Data Engine
  5. Data Engine Overview
  6. Detailed Synthesis Pipeline
  7. Step 1. Data Source.
  8. Step 2. Subgraph Sampling.
  9. Step 3. Task-specific Processing.
  10. Step 4. MMRK Image Data Synthesis.
  11. Step 5. Instruction Data Synthesis.
  12. Information of the Synthetic Data.
  13. Training and Evaluation Protocol
  14. Two-stage Training for Capability Enhancement
  15. Stage 1: Supervised Fine-tuning for General Enhancement.
  16. ...and 25 more sections

Figures (13)

  • Figure 1: Different kinds of images contain abstractive information with complex semantics.
  • Figure 2: The overview of our data engine, the training pipeline, the seed tasks, and the CoT prompts.
  • Figure 3: The task-wise transferability experiments. The y-axis is the basic task.
  • Figure 4: The scalability experiments Qwen2.5-VL-7B for 8 STAR tasks.
  • Figure 5: Ablation study on the effectiveness of CoT prompts in the instruction data.
  • ...and 8 more figures