Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DynamicGTR: Leveraging Graph Topology Representation Preferences to Boost VLM Capabilities on Graph QAs

Yanbin Wei, Jiangyue Yan, Chun Kang, Yang Chen, Hua Liu, James Kwok, Yu Zhang

TL;DR

The DynamicGTR framework is proposed, which dynamically selects the optimal GTR for each query during inference, thereby enhancing the zero-shot graph QA capabilities of VLMs with a customizable accuracy and brevity trade-off.

Abstract

Vision-Language Models (VLMs) have emerged as versatile solutions for zero-shot question answering (QA) across various domains. However, enabling VLMs to effectively comprehend structured graphs and perform accurate, efficient QA remains challenging. Existing approaches typically rely on one single graph topology representation (GTR), such as fixed-style visual images or unified text descriptions. This ``one-size-fits-all'' strategy often neglects model-specific and task-specific preferences, resulting in inaccurate or over-lengthy responses to graph-related queries. To address this, we propose the $\mbox{DynamicGTR}$ framework, which dynamically selects the optimal GTR for each query during inference, thereby enhancing the zero-shot graph QA capabilities of VLMs with a customizable accuracy and brevity trade-off. Extensive experiments show that DynamicGTR not only improves VLM-based graph algorithm QA performance but also successfully transfers the experience trained from synthetic graph algorithm tasks to real-world applications like link prediction and node classification, without any additional training. Additionally, DynamicGTR demonstrates strong transferability across tasks, domains, and models, suggesting its potential as a flexible solution for broad graph scenarios.

DynamicGTR: Leveraging Graph Topology Representation Preferences to Boost VLM Capabilities on Graph QAs

TL;DR

The DynamicGTR framework is proposed, which dynamically selects the optimal GTR for each query during inference, thereby enhancing the zero-shot graph QA capabilities of VLMs with a customizable accuracy and brevity trade-off.

Abstract

Vision-Language Models (VLMs) have emerged as versatile solutions for zero-shot question answering (QA) across various domains. However, enabling VLMs to effectively comprehend structured graphs and perform accurate, efficient QA remains challenging. Existing approaches typically rely on one single graph topology representation (GTR), such as fixed-style visual images or unified text descriptions. This ``one-size-fits-all'' strategy often neglects model-specific and task-specific preferences, resulting in inaccurate or over-lengthy responses to graph-related queries. To address this, we propose the framework, which dynamically selects the optimal GTR for each query during inference, thereby enhancing the zero-shot graph QA capabilities of VLMs with a customizable accuracy and brevity trade-off. Extensive experiments show that DynamicGTR not only improves VLM-based graph algorithm QA performance but also successfully transfers the experience trained from synthetic graph algorithm tasks to real-world applications like link prediction and node classification, without any additional training. Additionally, DynamicGTR demonstrates strong transferability across tasks, domains, and models, suggesting its potential as a flexible solution for broad graph scenarios.
Paper Structure (27 sections, 3 equations, 4 figures, 19 tables)

This paper contains 27 sections, 3 equations, 4 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. VLM-based Graph QA.
  4. Zero-shot GTR Pool
  5. Analyzing GTRs in Graph QA
  6. Constructing the ${R}_{ZS}$
  7. DynamicGTR
  8. Problem Formulation
  9. Framework Overview
  10. Graph Response Efficiency (GRE)
  11. GTR Preference Dataset
  12. GTR Router
  13. Experiment
  14. Experimental Setup
  15. Tasks and Datasets.
  16. ...and 12 more sections

Figures (4)

  • Figure 1: Illustrating the impacts of diverse GTRs on the same question: suboptimal GTRs may lead to incorrect answers or unnecessarily lengthy responses compared to a preferred GTR.
  • Figure 2: An Illustration of eight candidate GTRs in $\mathcal{R}_{ZS}$.
  • Figure 3: Overview of the DynamicGTR framework, where the GTR Router guides the VLM Reasoner to use the most appropriate GTR based on the question.
  • Figure D: An Illustration of eight candidate GTRs in $\mathcal{R}_{ZS}$.