Modality Translation for Object Detection Adaptation Without Forgetting Prior Knowledge

TL;DR

The paper tackles the challenge of adapting RGB-trained object detectors to a new modality (IR) without forgetting prior knowledge. It introduces ModTr, a lightweight input translator that jointly optimizes for the detection task while keeping the detector weights fixed, using a residual translation and Hadamard fusion to produce RGB-like inputs from IR data. Across LLVIP and FLIR datasets, ModTr demonstrates competitive or superior detection performance compared with traditional image-translation methods and standard fine-tuning, while preserving the original detector’s zero-shot capabilities and enabling memory-efficient, server-based multi-modality deployment. The work also analyzes backbones, fusion strategies, and knowledge-preservation scenarios, and extends to Canny-edge and future generative translator directions, offering a practical pathway for flexible, modality-agnostic object detection systems.

Abstract

A common practice in deep learning involves training large neural networks on massive datasets to achieve high accuracy across various domains and tasks. While this approach works well in many application areas, it often fails drastically when processing data from a new modality with a significant distribution shift from the data used to pre-train the model. This paper focuses on adapting a large object detection model trained on RGB images to new data extracted from IR images with a substantial modality shift. We propose Modality Translator (ModTr) as an alternative to the common approach of fine-tuning a large model to the new modality. ModTr adapts the IR input image with a small transformation network trained to directly minimize the detection loss. The original RGB model can then work on the translated inputs without any further changes or fine-tuning to its parameters. Experimental results on translating from IR to RGB images on two well-known datasets show that our simple approach provides detectors that perform comparably or better than standard fine-tuning, without forgetting the knowledge of the original model. This opens the door to a more flexible and efficient service-based detection pipeline, where a unique and unaltered server, such as an RGB detector, runs constantly while being queried by different modalities, such as IR with the corresponding translations model. Our code is available at: https://github.com/heitorrapela/ModTr.

Figures (12)

• Figure 1: Bounding box predictions over different adaptations of the RGB detector (Faster R-CNN) for IR images on two benchmarks: LLVIP and FLIR. Yellow and red boxes show the ground truth and predicted detections, respectively. In a) we see the RGB data. In b) FastCUT is an unsupervised image translation approach that takes as input infrared images (IR) and produces pseudo-RGB images. It does not focus on detection and requires both modalities for training. In c) we have fine-tuning, which is the standard approach to adapting the detector to the new modality. It requires only IR data but forgets the original knowledge of the original RGB detector. Finally, in d) is the ModTr, which focuses the translation on detection, requires only IR data and does not forget the original knowledge so that it can be reused for other tasks. Bounding box predictions for other detectors are provided in the supplementary material.
• Figure 1: The simplest approach is to use a different detector adapted to each modality. This can lead to a high level of accuracy but requires storing models in memory multiple times. In purple is the RGB detector, in green is one IR detector for one dataset, and in yellow is another detector for another IR dataset.
• Figure 2: Different approaches to deal with multiple modalities and/or domains. (a) The simplest approach is to use a different detector adapted to each modality. This can lead to a high level of accuracy but requires storing several models in memory. (b) Our proposed solution uses a single pre-trained model normally trained on the more abundant data (RGB) and then adapts the input through our ModTr model. (c) A single detector is jointly trained on all modalities. This allows using of a single model but requires access to all modalities jointly, which is often impossible, especially when dealing with large pre-trained models.
• Figure 2: Our proposed solution is based on using a single pre-trained detector model normally trained on the more abundant data (RGB) and then adapting the input through our ModTr block.
• Figure 3: Illustration of a sequence of $8$ images of LLVIP and FLIR dataset for Faster R-CNN. For each dataset, the first row is the RGB modality, followed by the IR modality and different representations created by ModTr. For visualizations of other detectors and variants of ModTr, please refer to the supplementary materials.