The Contribution of Knowledge in Visiolinguistic Learning: A Survey on Tasks and Challenges
Maria Lymperaiou, Giorgos Stamou
TL;DR
Visiolinguistic models struggle to generalize beyond visual cues due to limited world knowledge in training data. The paper surveys how external knowledge sources such as knowledge graphs and large language models, along with hybrid architectures, can fill these gaps. It provides a taxonomy of explicit vs implicit knowledge, outlines reasoning capabilities in KGs and LLMs, and reviews knowledge enhancing VL tasks including VQA, VCR, IC, sequential generation, and multi task learning, concluding with KG versus LLM considerations. The discussion identifies knowledge demanding datasets, explainability, and retrieval challenges as key directions to guide future research and practical deployment.
Abstract
Recent advancements in visiolinguistic (VL) learning have allowed the development of multiple models and techniques that offer several impressive implementations, able to currently resolve a variety of tasks that require the collaboration of vision and language. Current datasets used for VL pre-training only contain a limited amount of visual and linguistic knowledge, thus significantly limiting the generalization capabilities of many VL models. External knowledge sources such as knowledge graphs (KGs) and Large Language Models (LLMs) are able to cover such generalization gaps by filling in missing knowledge, resulting in the emergence of hybrid architectures. In the current survey, we analyze tasks that have benefited from such hybrid approaches. Moreover, we categorize existing knowledge sources and types, proceeding to discussion regarding the KG vs LLM dilemma and its potential impact to future hybrid approaches.