Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Explainability in Multimodal Large Language Models Using Ontological Context

Jihen Amara, Birgitta König-Ries, Sheeba Samuel

TL;DR

This work proposes a new framework that combines ontology with MLLMs to classify images of plant diseases and offers a new direction for synergizing ontologies and MLLMs.

Abstract

Recently, there has been a growing interest in Multimodal Large Language Models (MLLMs) due to their remarkable potential in various tasks integrating different modalities, such as image and text, as well as applications such as image captioning and visual question answering. However, such models still face challenges in accurately captioning and interpreting specific visual concepts and classes, particularly in domain-specific applications. We argue that integrating domain knowledge in the form of an ontology can significantly address these issues. In this work, as a proof of concept, we propose a new framework that combines ontology with MLLMs to classify images of plant diseases. Our method uses concepts about plant diseases from an existing disease ontology to query MLLMs and extract relevant visual concepts from images. Then, we use the reasoning capabilities of the ontology to classify the disease according to the identified concepts. Ensuring that the model accurately uses the concepts describing the disease is crucial in domain-specific applications. By employing an ontology, we can assist in verifying this alignment. Additionally, using the ontology's inference capabilities increases transparency, explainability, and trust in the decision-making process while serving as a judge by checking if the annotations of the concepts by MLLMs are aligned with those in the ontology and displaying the rationales behind their errors. Our framework offers a new direction for synergizing ontologies and MLLMs, supported by an empirical study using different well-known MLLMs.

Enhancing Explainability in Multimodal Large Language Models Using Ontological Context

TL;DR

This work proposes a new framework that combines ontology with MLLMs to classify images of plant diseases and offers a new direction for synergizing ontologies and MLLMs.

Abstract

Recently, there has been a growing interest in Multimodal Large Language Models (MLLMs) due to their remarkable potential in various tasks integrating different modalities, such as image and text, as well as applications such as image captioning and visual question answering. However, such models still face challenges in accurately captioning and interpreting specific visual concepts and classes, particularly in domain-specific applications. We argue that integrating domain knowledge in the form of an ontology can significantly address these issues. In this work, as a proof of concept, we propose a new framework that combines ontology with MLLMs to classify images of plant diseases. Our method uses concepts about plant diseases from an existing disease ontology to query MLLMs and extract relevant visual concepts from images. Then, we use the reasoning capabilities of the ontology to classify the disease according to the identified concepts. Ensuring that the model accurately uses the concepts describing the disease is crucial in domain-specific applications. By employing an ontology, we can assist in verifying this alignment. Additionally, using the ontology's inference capabilities increases transparency, explainability, and trust in the decision-making process while serving as a judge by checking if the annotations of the concepts by MLLMs are aligned with those in the ontology and displaying the rationales behind their errors. Our framework offers a new direction for synergizing ontologies and MLLMs, supported by an empirical study using different well-known MLLMs.
Paper Structure (18 sections, 1 equation, 8 figures, 1 table, 1 algorithm)

This paper contains 18 sections, 1 equation, 8 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methods
  4. Incorporating Domain Knowledge through Ontology Usage
  5. Query Construction and Reasoning
  6. Prompt design using extracted concepts from the ontology
  7. Experiments and results
  8. Experimental settings
  9. Evaluated MLLMs
  10. Evaluation metrics
  11. Results and Discussion
  12. Experiment 1: Ontology as a judge of MLLM performance
  13. Multimodal large model evaluation per general concept identification:
  14. Multimodal large model evaluation per concept identification for each class:
  15. Distribution of concepts abnormalities for each model :
  16. ...and 3 more sections

Figures (8)

  • Figure 1: The framework of the proposed method.
  • Figure 2: Description logic queries for classifying the disease based on the concepts identified by the Multimodal Large Language model (MLLM).
  • Figure 3: Sample images from the rice leaf disease dataset hosain2022rice
  • Figure 4: Comparative performance of different MLLMs per concept (i.e, symptom, color and shape) identification.
  • Figure 5: Comparative performance of different MLLMs per concept per class.
  • ...and 3 more figures