MITracker: Multi-View Integration for Visual Object Tracking

TL;DR

MITracker tackles occlusion and target loss in multi-view visual object tracking by forming a BEV-guided $3$D feature volume from per-view 2D features and applying a geometry-aware attention module for cross-view refinement. The authors introduce the MVTrack dataset, a 234K-frame, 27-object, 9-attribute benchmark collected from calibrated 3–4 camera rigs, enabling class-agnostic MVOT research. Empirically, MITracker achieves state-of-the-art results on MVTrack and GMTD, with notable improvements in multi-view $P_{Norm}$ and strong single-view transfer, while demonstrating robust recovery after target disappearance. These contributions advance practical multi-view tracking by providing a large-scale dataset and a robust BEV-guided fusion framework suitable for long sequences and varied viewpoints.

Abstract

Multi-view object tracking (MVOT) offers promising solutions to challenges such as occlusion and target loss, which are common in traditional single-view tracking. However, progress has been limited by the lack of comprehensive multi-view datasets and effective cross-view integration methods. To overcome these limitations, we compiled a Multi-View object Tracking (MVTrack) dataset of 234K high-quality annotated frames featuring 27 distinct objects across various scenes. In conjunction with this dataset, we introduce a novel MVOT method, Multi-View Integration Tracker (MITracker), to efficiently integrate multi-view object features and provide stable tracking outcomes. MITracker can track any object in video frames of arbitrary length from arbitrary viewpoints. The key advancements of our method over traditional single-view approaches come from two aspects: (1) MITracker transforms 2D image features into a 3D feature volume and compresses it into a bird's eye view (BEV) plane, facilitating inter-view information fusion; (2) we propose an attention mechanism that leverages geometric information from fused 3D feature volume to refine the tracking results at each view. MITracker outperforms existing methods on the MVTrack and GMTD datasets, achieving state-of-the-art performance. The code and the new dataset will be available at https://mii-laboratory.github.io/MITracker/.

Figures (12)

• Figure 1: Overview of MITracker's multi-view integration mechanism. Given $K$ camera views, our method projects features from views with visible targets into a 3D feature volume space, which is then used to refine tracking in views where the target is occluded.
• Figure 2: Example sequences, annotations, and their corresponding tracking attributes in the MVTrack dataset.
• Figure 3: The framework of MITracker. (a) The view-specific feature extraction module employs a ViT that utilizes temporal tokens to process each view independently, outputting unrefined results that can be further improved by multi-view information. The multi-view integration module contains (b) 3D feature volume construction that aggregates features into 3D space with BEV guidance and (c) spatial-enhanced attention that refines tracking results by 3D spatial information.
• Figure 4: General experiments on the MVTrack dataset evaluate tracking robustness. MITracker provides multi-view results, while other methods yield single-view results. In (a), methods are ranked by AUC and noted in the legend. For (b), the numbers in the legend represent the method's recovery rate within 10 frames after the target disappears.
• Figure 5: Qualitative comparison results. Comparison of our tracker with two SOTA methods on MVTrack dataset (top) and GMTD (bottom). Each frame is cropped for better visualization. IoU curves of each method’s prediction and ground truth are shown above, where IoU reflects tracking quality. MITracker demonstrates superior re-tracking performance upon target reappearance.