EffOWT: Transfer Visual Language Models to Open-World Tracking Efficiently and Effectively

TL;DR

EffOWT addresses the inefficiency of full fine-tuning and the suboptimality of zero-shot transfer when adapting Visual Language Models to Open-World Tracking. It introduces a compact, independent side network that runs in parallel with a frozen VLM backbone, augmented by a Hybrid Side Network that blends CNN locality with Transformer processing and a multi-scale feature fusion module; Sparse Interactions on the MLP further reduces computation. Empirically, EffOWT delivers a 5.5% absolute gain on the OWTA metric for unknown categories, while updating only 1.3% of parameters and achieving a 36.4% memory saving compared to full fine-tuning, outperforming state-of-the-art Open-World Tracking methods. The approach provides a practical, scalable baseline for efficient transfer of VLMs to OWT, with strong generalization to unseen categories and robust performance on known classes.

Abstract

Open-World Tracking (OWT) aims to track every object of any category, which requires the model to have strong generalization capabilities. Trackers can improve their generalization ability by leveraging Visual Language Models (VLMs). However, challenges arise with the fine-tuning strategies when VLMs are transferred to OWT: full fine-tuning results in excessive parameter and memory costs, while the zero-shot strategy leads to sub-optimal performance. To solve the problem, EffOWT is proposed for efficiently transferring VLMs to OWT. Specifically, we build a small and independent learnable side network outside the VLM backbone. By freezing the backbone and only executing backpropagation on the side network, the model's efficiency requirements can be met. In addition, EffOWT enhances the side network by proposing a hybrid structure of Transformer and CNN to improve the model's performance in the OWT field. Finally, we implement sparse interactions on the MLP, thus reducing parameter updates and memory costs significantly. Thanks to the proposed methods, EffOWT achieves an absolute gain of 5.5% on the tracking metric OWTA for unknown categories, while only updating 1.3% of the parameters compared to full fine-tuning, with a 36.4% memory saving. Other metrics also demonstrate obvious improvement.

Figures (5)

• Figure 1: A visual comparison of different fine-tuning strategies. (a) The full fine-tuning does not perform any freezing operation. (b) Zero-shot freezes the backbone and only fine-tunes the head. (c) Our method is to build a small and independent network next to the backbone, using the intermediate features from the backbone as input. During the fine-tuning phase, the backbone is frozen. Only the parameters on the side network and head can be updated.
• Figure 2: Overview of EffOWT. During the fine-tuning phase, the backbone is frozen and only the parameters on the side network and head will be updated. This operation greatly reduces the parameters and memory costs required for fine-tuning. In addition, side layers are aggregated into side blocks, and CNN is introduced to form the Hybrid Side Network to avoid the model from overfitting to known classes. Finally, the multi-scale feature fusion module is proposed to provide richer appearance information to the head.
• Figure 3: An illustration of Hybrid Side Network for OWT. (a) shows the structure of the hybrid side network and the feature processing flow. Among them, the purple block represents the CNN module. In (b), the network structure of the proposed multi-scale feature fusion module is described.
• Figure 4: A visual comparison of vanilla MLP and SIM in interactive calculations. In a vanilla MLP, each token interacts with all other tokens. In SIM, a single token only interacts with tokens on its horizontal, vertical, and two diagonal lines.
• Figure 5: Visual comparison between SOTA method SimOWT and our EffOWT. The visual comparison is conducted under that all these models provide top 10 predictions. Specifically, we use the green box to denote the prediction box which overlaps well with a known object. The boxes colored blue are used to represent the potential and unknown objects. And the other boxes are colored red to indicate some meaningless or inaccurate predictions.