MonitorVLM:A Vision Language Framework for Safety Violation Detection in Mining Operations

TL;DR

Mining safety monitoring is hindered by manual inspections; this work introduces MonitorVLM, a multimodal system that maps mining-violations in video to exact regulatory clauses. It combines a mining-specific VQA dataset fine-tuned with LoRA on Qwen2.5-VL-Instruct, a dynamic Top-5 clause-filter, and a behavior magnifier to enhance fine-grained action recognition, achieving substantial gains over unfine-tuned baselines. The approach delivers high-precision and recall (e.g., up to 93.05% precision, 89.57% recall, 91.28% F1) with improved inference efficiency, and is deployable via a lightweight web interface for timestamped violation reporting. The methodology is generalizable to other high-risk domains and sets the stage for broader automated safety surveillance across industries.

Abstract

Industrial accidents, particularly in high-risk domains such as surface and underground mining, are frequently caused by unsafe worker behaviors. Traditional manual inspection remains labor-intensive, error-prone, and insufficient for large-scale, dynamic environments, highlighting the urgent need for intelligent and automated safety monitoring. In this paper, we present MonitorVLM, a novel vision--language framework designed to detect safety violations directly from surveillance video streams. MonitorVLM introduces three key innovations: (1) a domain-specific violation dataset comprising 9,000 vision--question--answer (VQA) samples across 40 high-frequency mining regulations, enriched with augmentation and auxiliary detection cues; (2) a clause filter (CF) module that dynamically selects the Top-$K$ most relevant clauses, reducing inference latency by 13.56\% while maintaining accuracy; and (3) a behavior magnifier (BM) module that enhances worker regions to improve fine-grained action recognition, yielding additional gains of 3.45% in precision and 8.62% in recall. Experimental results demonstrate that MonitorVLM significantly outperforms baseline vision--language models, achieving improvements of 22.01% in precision, 34.22\% in recall, and 28.37% in F1 score over the 72B unfine-tuned baseline. A lightweight web-based interface further integrates MonitorVLM into practical workflows, enabling automatic violation reporting with video timestamping. This study highlights the potential of multimodal large models to enhance occupational safety monitoring in mining and beyond.

Figures (11)

• Figure 1: Illustration of industrial safety inspection tasks. The left panels show representative violation video frames, while the right panel summarizes excerpts from the 40 high-frequency regulations adopted in this study.
• Figure 2: Overall architecture of the proposed MonitorVLM framework. The system integrates a CF for clause filter, a BM for fine-grained worker action recognition, and a LoRA-fine-tuned VLM for comprehensive violation detection.
• Figure 3: Structure of the constructed violation dataset. Each training instance consists of an image triplet (three temporally contiguous frames) and an instruction triplet (system prompt, user prompt, and assistant response), supporting systematic violation reasoning.
• Figure 4: Examples of dataset enrichment strategies: (a) original image, (b) image augmentation (horizontal flipping, low-light synthesis, mask occlusion), and (c) auxiliary bounding-box annotations obtained via open-vocabulary detection.
• Figure 5: Illustration of the dynamic CF mechanism. The module predicts relevance scores between input frames and regulatory clauses, selecting the Top-$K$ most probable clauses for violation analysis.