VIoTGPT: Learning to Schedule Vision Tools in LLMs towards Intelligent Video Internet of Things

TL;DR

VIoTGPT addresses the challenge of intelligently scheduling domain-specific vision tools over large-scale VIoT video data by treating an LLM as the agent that queries a knowledge video base and orchestrates a toolbox of 11 vision algorithms. It introduces VIoT-Tool, a semi-automatically labeled dataset used to instruction-tune the LLM with ReAct-style prompts, enabling multi-step reasoning and inter-tool coordination. Experiments show that instruction-tuned, small-to-medium LLMs (e.g., Llama/Vicuna with IT) substantially improve tool selection and multi-tool workflows, though inter-tool coordination remains harder for weaker baselines and ChatGPT excels mainly on isolated tasks. The work advances intelligent, human-centered VIoT by enabling efficient, scalable querying of knowledge videos and scheduling of perception models, with VIoT-Tool serving as a public benchmark for future research.

Abstract

Video Internet of Things (VIoT) has shown full potential in collecting an unprecedented volume of video data. How to schedule the domain-specific perceiving models and analyze the collected videos uniformly, efficiently, and especially intelligently to accomplish complicated tasks is challenging. To address the challenge, we build VIoTGPT, the framework based on LLMs to correctly interact with humans, query knowledge videos, and invoke vision models to analyze multimedia data collaboratively. To support VIoTGPT and related future works, we meticulously crafted the VIoT-Tool dataset, including the training dataset and the benchmark involving 11 representative vision models across three categories based on semi-automatic annotations. To guide LLM to act as the intelligent agent towards intelligent VIoT, we resort to the ReAct instruction tuning method based on VIoT-Tool to learn the tool capability. Quantitative and qualitative experiments and analyses demonstrate the effectiveness of VIoTGPT. We believe VIoTGPT contributes to improving human-centered experiences in VIoT applications. The project website is https://github.com/zhongyy/VIoTGPT.

Figures (7)

• Figure 1: Illustration of VIoTGPT, which mainly consists of three fundamental modules, the videos that contain real-world observations, the human-centric algorithms acting as the tool set, and LLM as the intelligent agent.
• Figure 2: Pre-defined instructions and the response $A_{i,t}$ of the LLM $\theta$ at each step. In the intermediate steps, the LLM decides to use the tools ("Thought"), selected tool names ("Action"), and the input of tools ("Action Input"). At the end of each step, the context will be updated by combing previous actions and observations to conversation history $C_{i,t} = (C_{i,t-1}, A_{i,t}, o_{i,t})$. At the final step, the LLM decides not to use tools and returns the final feedback $fi$.
• Figure 3: Illustration of VIoTGPT's capabilities and applications. "FaceRecognition", "PersonReidentification", "GaitRecognition", "LicensePlateRecognition", "VehicleReidentification", "CrowdCounting" and "FireSmokeDetection" represent responses with single tool. While "AnomalyDetection" and "ActionAnalysis" are two event-related pipelines that require scheduling interrelated tools.
• Figure S1: Statistics of training and testing dataset of VIoT-Tool. A total of 200K training instructions (2.79 billion tokens) and 1,841 test samples are concluded in the VIoT-Tool dataset, related to 11 tools across three categories.
• Figure S2: Illustration of semi-annotated instructions, with initial annotations for practicability, ChatGPT for semantic generalization, and expert review for correctness.