RLE: A Unified Perspective of Data Augmentation for Cross-Spectral Re-identification

TL;DR

A Random Linear Enhancement (RLE) strategy is proposed which is designed to provide diverse image transformations that satisfy the original linear correlations under constrained conditions, whereas Radical Random Linear Enhancement seeks to generate local linear transformations directly without relying on external information.

Abstract

This paper makes a step towards modeling the modality discrepancy in the cross-spectral re-identification task. Based on the Lambertain model, we observe that the non-linear modality discrepancy mainly comes from diverse linear transformations acting on the surface of different materials. From this view, we unify all data augmentation strategies for cross-spectral re-identification by mimicking such local linear transformations and categorizing them into moderate transformation and radical transformation. By extending the observation, we propose a Random Linear Enhancement (RLE) strategy which includes Moderate Random Linear Enhancement (MRLE) and Radical Random Linear Enhancement (RRLE) to push the boundaries of both types of transformation. Moderate Random Linear Enhancement is designed to provide diverse image transformations that satisfy the original linear correlations under constrained conditions, whereas Radical Random Linear Enhancement seeks to generate local linear transformations directly without relying on external information. The experimental results not only demonstrate the superiority and effectiveness of RLE but also confirm its great potential as a general-purpose data augmentation for cross-spectral re-identification. The code is available at \textcolor{magenta}{\url{https://github.com/stone96123/RLE}}.

Figures (6)

• Figure 1: Illustration of the cross-spectral transformation. G refers to the green channel of the visible image. Under the same illumination, the cross-spectral transformation could be described as a linear transformation in a material-similar surface. Still, in the whole image level, the transformation is nonlinear due to the diversity of materials. Since the Re-ID image pairs are not well aligned, we select the cross-spectral image pairs from brown2011multi.
• Figure 2: Example images from the VIS-NIR scene dataset brown2011multi. After we divide the visible image into the red, green, and blue channels and form chromaticity band ratios from these three spectra and the NIR image, it is clear that the ratio for pixels from the surface with high material-similarity is nearly constant.
• Figure 3: A example about how modality discrepancy occurs. Feature space visualization of 100 randomly selected images with (dot) and without (fork) the local linear transformation on the original image. (a)$\sim$(b): The same linear factor takes effect on the whole image bringing limited modality discrepancy. (c): Variable linear factors take effect on different parts showing a huge modality discrepancy. The ’cross’ and ’dot’ marks indicate the samples from the original one and the generated one respectively.
• Figure 4: The motivation of RLE. Herein, we construct an ideal person with only two different surfaces and ignore the background. (a): As demonstrated above, to obtain a spectral-invariant feature representation, the network should be robust to such a transformation that takes effect upon definite surfaces by definite linear factors. (b): An ideal data augmentation strategy that takes effect upon definite surfaces by random linear factors. However, this method needs a hard-achieved extra material-aware network for segmentation. (c): The idea of RRLE. By taking effect upon random surfaces by random linear factors, the RRLE encourages the network to be robust to a linear transformation anywhere in the image. Under this condition, the cross-spectral transformation can be considered as an easy state of RRLE space.
• Figure 5: Visualization results of RLE. Since the MRLE can not take effect on the infrared images, we use '$\sim$' instead. Meanwhile, Both MRLE and RRLE are used with a certain probability. Therefore, all of the augmentation images above are potential results.