Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Few-Shot LoRA Adaptation of a Flow-Matching Foundation Model for Cross-Spectral Object Detection

Maxim Clouser, Kia Khezeli, John Kalantari

TL;DR

This work investigates adapting a single flow-matching foundation model to cross-spectral translation (RGB→IR, RGB→SAR) using only 100 co-measured pairs via LoRA adapters, enabling pixel-aligned synthetic data for downstream detection. The methodology leverages FLUX.1 Kontext as a base and inserts small LoRA modules into the transformer to imprint cross-spectral mappings with minimal training. A key finding is that LPIPS computed on a tiny validation set strongly predicts downstream detector performance, enabling efficient LoRA selection, and that synthetic data—when combined with limited real data—consistently boosts IR and SAR detection. The results demonstrate practical gains in KAIST IR pedestrian detection and M4-SAR infrastructure detection, highlighting the potential of few-shot, foundation-model-based cross-spectral augmentation for safety-critical vision tasks. Overall, this approach offers a data-efficient pathway to extend vision foundation models to non-visible modalities with tangible improvements in real-world detection tasks.

Abstract

Foundation models for vision are predominantly trained on RGB data, while many safety-critical applications rely on non-visible modalities such as infrared (IR) and synthetic aperture radar (SAR). We study whether a single flow-matching foundation model pre-trained primarily on RGB images can be repurposed as a cross-spectral translator using only a few co-measured examples, and whether the resulting synthetic data can enhance downstream detection. Starting from FLUX.1 Kontext, we insert low-rank adaptation (LoRA) modules and fine-tune them on just 100 paired images per domain for two settings: RGB to IR on the KAIST dataset and RGB to SAR on the M4-SAR dataset. The adapted model translates RGB images into pixel-aligned IR/SAR, enabling us to reuse existing bounding boxes and train object detection models purely in the target modality. Across a grid of LoRA hyperparameters, we find that LPIPS computed on only 50 held-out pairs is a strong proxy for downstream performance: lower LPIPS consistently predicts higher mAP for YOLOv11n on both IR and SAR, and for DETR on KAIST IR test data. Using the best LPIPS-selected LoRA adapter, synthetic IR from external RGB datasets (LLVIP, FLIR ADAS) improves KAIST IR pedestrian detection, and synthetic SAR significantly boosts infrastructure detection on M4-SAR when combined with limited real SAR. Our results suggest that few-shot LoRA adaptation of flow-matching foundation models is a promising path toward foundation-style support for non-visible modalities.

Few-Shot LoRA Adaptation of a Flow-Matching Foundation Model for Cross-Spectral Object Detection

TL;DR

This work investigates adapting a single flow-matching foundation model to cross-spectral translation (RGB→IR, RGB→SAR) using only 100 co-measured pairs via LoRA adapters, enabling pixel-aligned synthetic data for downstream detection. The methodology leverages FLUX.1 Kontext as a base and inserts small LoRA modules into the transformer to imprint cross-spectral mappings with minimal training. A key finding is that LPIPS computed on a tiny validation set strongly predicts downstream detector performance, enabling efficient LoRA selection, and that synthetic data—when combined with limited real data—consistently boosts IR and SAR detection. The results demonstrate practical gains in KAIST IR pedestrian detection and M4-SAR infrastructure detection, highlighting the potential of few-shot, foundation-model-based cross-spectral augmentation for safety-critical vision tasks. Overall, this approach offers a data-efficient pathway to extend vision foundation models to non-visible modalities with tangible improvements in real-world detection tasks.

Abstract

Foundation models for vision are predominantly trained on RGB data, while many safety-critical applications rely on non-visible modalities such as infrared (IR) and synthetic aperture radar (SAR). We study whether a single flow-matching foundation model pre-trained primarily on RGB images can be repurposed as a cross-spectral translator using only a few co-measured examples, and whether the resulting synthetic data can enhance downstream detection. Starting from FLUX.1 Kontext, we insert low-rank adaptation (LoRA) modules and fine-tune them on just 100 paired images per domain for two settings: RGB to IR on the KAIST dataset and RGB to SAR on the M4-SAR dataset. The adapted model translates RGB images into pixel-aligned IR/SAR, enabling us to reuse existing bounding boxes and train object detection models purely in the target modality. Across a grid of LoRA hyperparameters, we find that LPIPS computed on only 50 held-out pairs is a strong proxy for downstream performance: lower LPIPS consistently predicts higher mAP for YOLOv11n on both IR and SAR, and for DETR on KAIST IR test data. Using the best LPIPS-selected LoRA adapter, synthetic IR from external RGB datasets (LLVIP, FLIR ADAS) improves KAIST IR pedestrian detection, and synthetic SAR significantly boosts infrastructure detection on M4-SAR when combined with limited real SAR. Our results suggest that few-shot LoRA adaptation of flow-matching foundation models is a promising path toward foundation-style support for non-visible modalities.
Paper Structure (28 sections, 2 equations, 3 figures, 2 tables)

This paper contains 28 sections, 2 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Cross-Spectral Image Translation
  4. LoRA and Parameter-Efficient Fine-Tuning of Generative Models
  5. Flow-Matching and Score-Based Generative Models in Vision
  6. Synthetic IR/SAR Data for Object Detection
  7. Method: LoRA-Adapted Flow Matching for Cross-Spectral Translation
  8. Problem Setting
  9. Base Model: FLUX.1 Kontext
  10. LoRA-Adapted Flow Matching
  11. Model Selection via LPIPS
  12. Synthetic Dataset Construction for Detection
  13. Experimental Setup
  14. Datasets and Splits
  15. KAIST Multispectral Pedestrian
  16. ...and 13 more sections

Figures (3)

  • Figure 1: Overview of our pipeline for LoRA-adapted flow-matching cross-spectral translation and detection. (A) A small Sensor Sample split of paired RGB and IR/SAR images is used to train multiple LoRA configurations on top of a frozen FLUX.1 Kontext base model. (B) A separate Sensor Val split is translated and scored against real images with LPIPS to select the best LoRA. (C) The selected LoRA $\lambda^\ast$ is applied to the RGB Train split to generate a sensor-aligned synthetic target-domain set, reusing the original RGB bounding box annotations. (D) The sensor-aligned synthetic set, optionally combined with real target-domain images, is used to train an object detection model in the target modality. All panels show KAIST RGB$\rightarrow$IR; the same pipeline is applied for RGB$\rightarrow$SAR.
  • Figure 2: LPIPS on Sensor Val versus YOLOv11n / DETR mAP@0.50 on the real Test sets. From left to right: (i) KAIST with YOLOv11n, (ii) KAIST with DETR, and (iii) M4-SAR with YOLOv11n. Each point corresponds to a LoRA configuration: color encodes the number of LoRA training steps, marker shape encodes learning rate, and filled versus unfilled markers encode LoRA rank. Points show the mean over 5 runs per configuration and error bars indicate $\pm 1$ standard deviation. Solid lines show least-squares linear fits. Panel titles report Pearson (r) and Spearman ($\rho$) correlation coefficients with associated p-values, all indicating strong negative correlations between LPIPS and downstream detection performance.
  • Figure 3: Examples of cross-modal image translation on the KAIST (RGB--IR) and M4-SAR (RGB--SAR) datasets. For each dataset and scene, columns show the input RGB image, the corresponding real IR/SAR image, and synthetic IR/SAR images generated by the best- and worst-performing LoRA configurations, ordered from left to right. Best and worst are ranked by downstream YOLOv11n mAP@0.50.