Enhancing Object Detection with Privileged Information: A Model-Agnostic Teacher-Student Approach

TL;DR

This work tackles improving object detection by leveraging privileged information available only during training. It introduces a model-agnostic teacher–student framework where a teacher receives RGB inputs plus privileged signals (e.g., bounding box masks) and guides a RGB-only student through latent-layer alignment, optimized by a combined loss that balances standard detection supervision with teacher guidance. Empirically, the approach yields consistent accuracy gains across five detectors and multiple datasets (including UAV litter datasets and Pascal VOC 2012) without increasing inference cost, with the best gains for medium/large objects and when the teacher balance $\alpha$ is set to intermediate values. The findings demonstrate that LUPI is a practical strategy for enhancing detection performance in both high- and low-resource settings, while maintaining deployment efficiency and offering interpretability benefits via attention analyses.

Abstract

This paper investigates the integration of the Learning Using Privileged Information (LUPI) paradigm in object detection to exploit fine-grained, descriptive information available during training but not at inference. We introduce a general, model-agnostic methodology for injecting privileged information-such as bounding box masks, saliency maps, and depth cues-into deep learning-based object detectors through a teacher-student architecture. Experiments are conducted across five state-of-the-art object detection models and multiple public benchmarks, including UAV-based litter detection datasets and Pascal VOC 2012, to assess the impact on accuracy, generalization, and computational efficiency. Our results demonstrate that LUPI-trained students consistently outperform their baseline counterparts, achieving significant boosts in detection accuracy with no increase in inference complexity or model size. Performance improvements are especially marked for medium and large objects, while ablation studies reveal that intermediate weighting of teacher guidance optimally balances learning from privileged and standard inputs. The findings affirm that the LUPI framework provides an effective and practical strategy for advancing object detection systems in both resource-constrained and real-world settings.

Figures (7)

• Figure 1: Visual comparison of baseline object detection predictions and those of a LUPI-trained student model, showing improved accuracy while keeping the same architecture. The figure also illustrates the LUPI training pipeline, including privileged information, teacher models, and knowledge distillation, with these boosts arising solely from the bolstered learning process.
• Figure 2: Detailed architecture of the training setup. The teacher network receives both RGB images and privileged input channels, producing richer intermediate representations. The student network only processes RGB images, but is trained with additional supervision through knowledge distillation from the teacher. A baseline RGB-only model is included for comparison. The student demonstrates refined predictions relative to the baseline.
• Figure 3: Investigation of different forms of privileged information using the RetinaNet model on the SODA 1-metre dataset. The comparison includes saliency, depth, fusion, and bounding box mask representations. The bounding box mask yielded the highest improvement in detection accuracy.
• Figure 4: Comparative analysis of baseline and best LUPI-trained student models across all datasets for the five architectures, shown as a multi-radar graph. The figure highlights notable improvements in strict mAP and F1 score, with the largest boosts observed in within-dataset evaluations, while other datasets show smaller yet meaningful improvements using identical architectures.
• Figure 5: Ablation study results across all datasets and experiments using COCO metrics. Baseline model corresponds to $\alpha = 0$; other lines represent student models. Red downward arrows indicate top performance in the strict map@50--95 metric. Best results are generally observed for $\alpha = 0.25$ and 0.5, with $\alpha = 0.75$ also performing well, while $\alpha = 1$ shows lower average performance.