Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Vision Language Models with Logic Reasoning for Situational Awareness

Pavana Pradeep, Krishna Kant, Suya Yu

TL;DR

The paper tackles vision-language model–based situational awareness for infrequent but critical events by integrating VLMs with traditional computer vision through explicit logic reasoning. It introduces a consistency-driven fine-tuning framework that uses main, auxiliary, and proxy task structures, grounded in SMT-based reasoning, to guide directed FT and enable inference-time justifications. Across TU_DAT, Taekwondo, and Kinetics datasets, the directed FT approach consistently yields higher classification accuracy and CIF scores than undirected FT, while incurring manageable overhead; the method scales across image-, video-, and non-transformer VLMs. Practically, this framework improves reliability in SA tasks and provides a principled mechanism to justify or question model outputs in real time, with potential for sub-second inference using newer hardware. The work also highlights tradeoffs such as catastrophic forgetting and the additional resources required for running auxiliary reasoning components.

Abstract

Vision-Language Models (VLMs) offer the ability to generate high-level, interpretable descriptions of complex activities from images and videos, making them valuable for situational awareness (SA) applications. In such settings, the focus is on identifying infrequent but significant events with high reliability and accuracy, while also extracting fine-grained details and assessing recognition quality. In this paper, we propose an approach that integrates VLMs with traditional computer vision methods through explicit logic reasoning to enhance SA in three key ways: (a) extracting fine-grained event details, (b) employing an intelligent fine-tuning (FT) strategy that achieves substantially higher accuracy than uninformed selection, and (c) generating justifications for VLM outputs during inference. We demonstrate that our intelligent FT mechanism improves the accuracy and provides a valuable means, during inferencing, to either confirm the validity of the VLM output or indicate why it may be questionable.

Enhancing Vision Language Models with Logic Reasoning for Situational Awareness

TL;DR

The paper tackles vision-language model–based situational awareness for infrequent but critical events by integrating VLMs with traditional computer vision through explicit logic reasoning. It introduces a consistency-driven fine-tuning framework that uses main, auxiliary, and proxy task structures, grounded in SMT-based reasoning, to guide directed FT and enable inference-time justifications. Across TU_DAT, Taekwondo, and Kinetics datasets, the directed FT approach consistently yields higher classification accuracy and CIF scores than undirected FT, while incurring manageable overhead; the method scales across image-, video-, and non-transformer VLMs. Practically, this framework improves reliability in SA tasks and provides a principled mechanism to justify or question model outputs in real time, with potential for sub-second inference using newer hardware. The work also highlights tradeoffs such as catastrophic forgetting and the additional resources required for running auxiliary reasoning components.

Abstract

Vision-Language Models (VLMs) offer the ability to generate high-level, interpretable descriptions of complex activities from images and videos, making them valuable for situational awareness (SA) applications. In such settings, the focus is on identifying infrequent but significant events with high reliability and accuracy, while also extracting fine-grained details and assessing recognition quality. In this paper, we propose an approach that integrates VLMs with traditional computer vision methods through explicit logic reasoning to enhance SA in three key ways: (a) extracting fine-grained event details, (b) employing an intelligent fine-tuning (FT) strategy that achieves substantially higher accuracy than uninformed selection, and (c) generating justifications for VLM outputs during inference. We demonstrate that our intelligent FT mechanism improves the accuracy and provides a valuable means, during inferencing, to either confirm the validity of the VLM output or indicate why it may be questionable.
Paper Structure (24 sections, 9 figures, 13 tables)

This paper contains 24 sections, 9 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Proposed Framework
  3. Motivation for VLM/TCV Integration
  4. Tasks and Activities
  5. Consistency Driven Fine-tuning
  6. Related Work
  7. Justifying Inferences
  8. Implementing Directed Fine Tuning
  9. Identifying Proxy and Auxiliary Activities
  10. Using Explicit Logic Reasoning
  11. Fine Tuning Algorithm
  12. Details of Experimental Setup
  13. VLMs Used For Evaluation
  14. Description of Datasets Used
  15. Classes Used to Fine-Tune VLMs
  16. ...and 9 more sections

Figures (9)

  • Figure 1: Task and Activity Relationship
  • Figure 2: Application Directed Fine-tuning of VLMs
  • Figure 3: Finding key objects in the scene
  • Figure 4: Green belt movements in Taekwondo
  • Figure 5: Few activities from Kinetics dataset
  • ...and 4 more figures