Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chain-of-Anomaly Thoughts with Large Vision-Language Models

Pedro Domingos, João Pereira, Vasco Lopes, João Neves, David Semedo

TL;DR

This work tackles the normality bias of large vision-language models in surveillance anomaly detection by introducing Chain-of-Anomaly Thoughts (CoAT), a multi-agent reasoning framework with a final anomaly-focused classification layer. CoAT orchestrates a Witness LVLM, a Detective LLM, and a Supervisor LLM to build a State-of-Thoughts graph and perform structured, layered exploration, culminating in an anomaly-biased decision. The approach yields substantial gains on challenging low-resolution data (AD F1 +11.8 p.p.) and high-resolution anomaly classification (AC +3.78 p.p.), while analyses reveal the importance of task-specific inductive biases and the potential downside of accumulating normality bias in joint reasoning. The results highlight the practical value of domain-aligned reasoning in enhancing surveillance robustness and crime recognition in LVLM-based systems.

Abstract

Automated video surveillance with Large Vision-Language Models is limited by their inherent bias towards normality, often failing to detect crimes. While Chain-of-Thought reasoning strategies show significant potential for improving performance in language tasks, the lack of inductive anomaly biases in their reasoning further steers the models towards normal interpretations. To address this, we propose Chain-of-Anomaly-Thoughts (CoAT), a multi-agent reasoning framework that introduces inductive criminal bias in the reasoning process through a final, anomaly-focused classification layer. Our method significantly improves Anomaly Detection, boosting F1-score by 11.8 p.p. on challenging low-resolution footage and Anomaly Classification by 3.78 p.p. in high-resolution videos.

Chain-of-Anomaly Thoughts with Large Vision-Language Models

TL;DR

This work tackles the normality bias of large vision-language models in surveillance anomaly detection by introducing Chain-of-Anomaly Thoughts (CoAT), a multi-agent reasoning framework with a final anomaly-focused classification layer. CoAT orchestrates a Witness LVLM, a Detective LLM, and a Supervisor LLM to build a State-of-Thoughts graph and perform structured, layered exploration, culminating in an anomaly-biased decision. The approach yields substantial gains on challenging low-resolution data (AD F1 +11.8 p.p.) and high-resolution anomaly classification (AC +3.78 p.p.), while analyses reveal the importance of task-specific inductive biases and the potential downside of accumulating normality bias in joint reasoning. The results highlight the practical value of domain-aligned reasoning in enhancing surveillance robustness and crime recognition in LVLM-based systems.

Abstract

Automated video surveillance with Large Vision-Language Models is limited by their inherent bias towards normality, often failing to detect crimes. While Chain-of-Thought reasoning strategies show significant potential for improving performance in language tasks, the lack of inductive anomaly biases in their reasoning further steers the models towards normal interpretations. To address this, we propose Chain-of-Anomaly-Thoughts (CoAT), a multi-agent reasoning framework that introduces inductive criminal bias in the reasoning process through a final, anomaly-focused classification layer. Our method significantly improves Anomaly Detection, boosting F1-score by 11.8 p.p. on challenging low-resolution footage and Anomaly Classification by 3.78 p.p. in high-resolution videos.
Paper Structure (9 sections, 3 figures, 1 table)

This paper contains 9 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Chain-of-Anomaly Thoughts (CoAT)
  4. State-of-Thoughts
  5. Anomaly Classification Layer.
  6. Results
  7. Experimental Setup
  8. Discussion
  9. Conclusion

Figures (3)

  • Figure 1: CoAT’s advantage over other approaches. Traditional methods struggle with generalization, while LVLMs and reasoning-based approaches fail due to increased normality bias. CoAT addresses these issues by employing a criminal-biased layer within its reasoning.
  • Figure 2: Overview of CoAT, a multi-agent reasoning framework for anomaly detection and classification.
  • Figure 3: Impact of higher video resolution on the AC task, displayed through the difference (high-resolution - low-resolution) of row-normalized confusion matrices. Left: baseline difference confusion matrix. Right: proposed solution (L4) difference confusion matrix.