Visual Graph Question Answering with ASP and LLMs for Language Parsing
Jakob Johannes Bauer, Thomas Eiter, Nelson Higuera Ruiz, Johannes Oetsch
TL;DR
This work tackles Visual Graph Question Answering (VGQA) by proposing NSGRAPH, a modular neuro-symbolic system that integrates optical graph recognition, OCR, LLM-based semantic parsing, and Answer-Set Programming for reasoning over graph images. It introduces CLEGR^V, a VGQA dataset derived from CLEGR with transit-like graphs across small, medium, and large sizes, and demonstrates a first baseline accuracy of $73\%$ on this dataset. The paper further investigates semantic parsing with multiple Large Language Models to extract ASP predicates, comparing models such as GPT-4 and Zephyr, and finds GPT-4 to perform best while smaller models show promise for cost-efficient setups. Overall, NSGRAPH showcases the viability and interpretability of modular neuro-symbolic VQA systems that leverage pretrained components without task-specific training, and highlights avenues for improving the visual parsing modules and extending to real-world graph imagery.
Abstract
Visual Question Answering (VQA) is a challenging problem that requires to process multimodal input. Answer-Set Programming (ASP) has shown great potential in this regard to add interpretability and explainability to modular VQA architectures. In this work, we address the problem of how to integrate ASP with modules for vision and natural language processing to solve a new and demanding VQA variant that is concerned with images of graphs (not graphs in symbolic form). Images containing graph-based structures are an ubiquitous and popular form of visualisation. Here, we deal with the particular problem of graphs inspired by transit networks, and we introduce a novel dataset that amends an existing one by adding images of graphs that resemble metro lines. Our modular neuro-symbolic approach combines optical graph recognition for graph parsing, a pretrained optical character recognition neural network for parsing labels, Large Language Models (LLMs) for language processing, and ASP for reasoning. This method serves as a first baseline and achieves an overall average accuracy of 73% on the dataset. Our evaluation provides further evidence of the potential of modular neuro-symbolic systems, in particular with pretrained models that do not involve any further training and logic programming for reasoning, to solve complex VQA tasks.