TrafficLens: Multi-Camera Traffic Video Analysis Using LLMs

TL;DR

TrafficLens tackles the high cost of converting multi-camera traffic video into text for downstream LLM queries. It employs a two-stage, overlap-aware ingestion: a base camera is processed first with a high token limit to produce rich base descriptions, followed by other cameras using lower token limits to add undetected details, with an IoU-based similarity detector to skip redundant clips. The method couples this accelerated video-to-text pipeline with a retrieval-augmented generation framework to answer user queries, and ablation shows a practical similarity threshold balances efficiency and information gain. Experimental results on StreetAware demonstrate substantial ingestion-time reductions while preserving answer quality, indicating practical impact for real-time traffic analytics and incident investigation.

Abstract

Traffic cameras are essential in urban areas, playing a crucial role in intelligent transportation systems. Multiple cameras at intersections enhance law enforcement capabilities, traffic management, and pedestrian safety. However, efficiently managing and analyzing multi-camera feeds poses challenges due to the vast amount of data. Analyzing such huge video data requires advanced analytical tools. While Large Language Models (LLMs) like ChatGPT, equipped with retrieval-augmented generation (RAG) systems, excel in text-based tasks, integrating them into traffic video analysis demands converting video data into text using a Vision-Language Model (VLM), which is time-consuming and delays the timely utilization of traffic videos for generating insights and investigating incidents. To address these challenges, we propose TrafficLens, a tailored algorithm for multi-camera traffic intersections. TrafficLens employs a sequential approach, utilizing overlapping coverage areas of cameras. It iteratively applies VLMs with varying token limits, using previous outputs as prompts for subsequent cameras, enabling rapid generation of detailed textual descriptions while reducing processing time. Additionally, TrafficLens intelligently bypasses redundant VLM invocations through an object-level similarity detector. Experimental results with real-world datasets demonstrate that TrafficLens reduces video-to-text conversion time by up to $4\times$ while maintaining information accuracy.

Figures (7)

• Figure 1: An overview of the RAG-based traffic video analysis system. It operates in two phases: a) the multi-camera videos is initially converted into a text document, the combined texts from all documents are then chunked into smaller parts and stored in a vector database; b) queries are answered using a Large Language Model (LLM), leveraging query and contextual information retrieved from the vector database through semantic search.
• Figure 2:
• Figure 3: A sample image from the StreetAware piadyk2023streetaware dataset.
• Figure 4: Latency of VLM output generation increases with the increase in number of output tokens.
• Figure 5: Accelerated video-to-text conversion workflow of TrafficLens.