ACTRESS: Active Retraining for Semi-supervised Visual Grounding

TL;DR

ACTRESS tackles semi-supervised visual grounding in Transformer-based models by introducing a quantized detection head to expose pseudo-label quality and an active sampling strategy that optimizes Faithfulness, Robustness, and Confidence. A selective retraining scheme periodically reinitializes parts of the network while leveraging high-quality pseudo labels, enabling effective use of unlabeled data and avoiding local minima. The approach is demonstrated to yield state-of-the-art results on RefCOCO, RefCOCO+, and RefCOCOg-umd across TransVG and VLTVG under limited supervision, with notable improvements over RefTeacher and related methods. Overall, ACTRESS provides a practical, compatible SSVG framework that extends modern VG architectures to semi-supervised learning while reducing training time and preserving grounding quality.

Abstract

Semi-Supervised Visual Grounding (SSVG) is a new challenge for its sparse labeled data with the need for multimodel understanding. A previous study, RefTeacher, makes the first attempt to tackle this task by adopting the teacher-student framework to provide pseudo confidence supervision and attention-based supervision. However, this approach is incompatible with current state-of-the-art visual grounding models, which follow the Transformer-based pipeline. These pipelines directly regress results without region proposals or foreground binary classification, rendering them unsuitable for fitting in RefTeacher due to the absence of confidence scores. Furthermore, the geometric difference in teacher and student inputs, stemming from different data augmentations, induces natural misalignment in attention-based constraints. To establish a compatible SSVG framework, our paper proposes the ACTive REtraining approach for Semi-Supervised Visual Grounding, abbreviated as ACTRESS. Initially, the model is enhanced by incorporating an additional quantized detection head to expose its detection confidence. Building upon this, ACTRESS consists of an active sampling strategy and a selective retraining strategy. The active sampling strategy iteratively selects high-quality pseudo labels by evaluating three crucial aspects: Faithfulness, Robustness, and Confidence, optimizing the utilization of unlabeled data. The selective retraining strategy retrains the model with periodic re-initialization of specific parameters, facilitating the model's escape from local minima. Extensive experiments demonstrates our superior performance on widely-used benchmark datasets.

Figures (4)

• Figure 2: The overall framework of ACTRESS. Take 1% labeled data as an example. Upper Left: We first train the model with 1% labeled data, which is called the Burn-in stage. The model is equipped with an additional quantized detection head, which is supervised by a quantized ground truth bounding box. Upper Right: In the Active Retraining Stage, the model performs inference on 99% unlabeled data and evaluates three metrics: Faithfulness, Robustness, and Confidence. Following that, we sample N% new pseudo label with a high score of these three metrics. The sampled data together with 1% labeled data go through geometrical transformation w/ or w/o quantization to supervise the two detection heads. Then the model selectively retrains the backbone and the detection heads, whose parameters are re-initialized at the beginning of the stage, and keeps training the fusion module. We periodically perform this stage for K times. Bottom: Illustration of our three metrics.
• Figure 3: Predictions from two detection heads. The blue box is the ground truth and the red box is the prediction. We compare the prediction quality of the model trained after the Burn-in stage and after multiple rounds of the Active Retraining stage (Act). It is evident that the quality improves consistently with an increase in the number of rounds of the Active Retraining stage. Texts on the right side are the input text.
• Figure 4: Visualization of correct cases and false cases of ACTRESS. Red boxes are ground truth boxes, green boxes are detection outputs, and blue boxes are quantized detection outputs.
• Figure : ACTRESS