AIVD: Adaptive Edge-Cloud Collaboration for Accurate and Efficient Industrial Visual Detection

TL;DR

AIVD tackles the challenge of precise localization and robust semantic reasoning in industrial inspection by integrating lightweight edge detectors with cloud-based multimodal LLMs and a heterogeneously-aware dynamic scheduler. The approach combines a visual-semantic synergistic fine-tuning pipeline, including context-aware cropping, HSV augmentations, semantic prompts, and Low-Rank Adaptation with $W^* = W + \lambda BA$, to align local cues with semantic mappings, while a resource-aware scheduler assigns tasks across edge and cloud nodes using a weighted score $S_i = w^T R_i$. Key contributions include the efficient fine-tuning strategy with semantic augmentation, the design of a heterogeneous dynamic scheduling algorithm, and comprehensive experiments showing reduced resource consumption and improved classification and semantic generation under varying network conditions. The results demonstrate significant throughput and latency benefits in multi-edge deployments, enabling real-time, high-precision industrial defect detection with robust semantic reasoning.

Abstract

Multimodal large language models (MLLMs) demonstrate exceptional capabilities in semantic understanding and visual reasoning, yet they still face challenges in precise object localization and resource-constrained edge-cloud deployment. To address this, this paper proposes the AIVD framework, which achieves unified precise localization and high-quality semantic generation through the collaboration between lightweight edge detectors and cloud-based MLLMs. To enhance the cloud MLLM's robustness against edge cropped-box noise and scenario variations, we design an efficient fine-tuning strategy with visual-semantic collaborative augmentation, significantly improving classification accuracy and semantic consistency. Furthermore, to maintain high throughput and low latency across heterogeneous edge devices and dynamic network conditions, we propose a heterogeneous resource-aware dynamic scheduling algorithm. Experimental results demonstrate that AIVD substantially reduces resource consumption while improving MLLM classification performance and semantic generation quality. The proposed scheduling strategy also achieves higher throughput and lower latency across diverse scenarios.

Figures (6)

• Figure 1: We propose AIVD, a novel framework for high-precision industrial visual detection. (a) Detected by MLLM-only: Struggle to meet precise localization demands in industrial scenarios. (b) Our approach outperforms existing methods. (c) Detected by AIVD: Achieves efficient object localization while generating precise defect cause descriptions.
• Figure 2: Overall framework of AIVD.
• Figure 3: Example of AVID semantic analysis.
• Figure 4: Comparison of edge-side throughput under different numbers of nodes.
• Figure 5: Comparison of edge-side latency under different numbers of nodes.