Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test-time Training for Hyperspectral Image Super-resolution

Ke Li, Luc Van Gool, Dengxin Dai

TL;DR

A novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated so that the model can be further trained with them to improve performance.

Abstract

The progress on Hyperspectral image (HSI) super-resolution (SR) is still lagging behind the research of RGB image SR. HSIs usually have a high number of spectral bands, so accurately modeling spectral band interaction for HSI SR is hard. Also, training data for HSI SR is hard to obtain so the dataset is usually rather small. In this work, we propose a new test-time training method to tackle this problem. Specifically, a novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated so that the model can be further trained with them to improve performance. In order to better support our test-time training method, we also propose a new network architecture to learn HSI SR without modeling spectral band interaction and propose a new data augmentation method Spectral Mixup to increase the diversity of the training data at test time. We also collect a new HSI dataset with a diverse set of images of interesting objects ranging from food to vegetation, to materials, and to general scenes. Extensive experiments on multiple datasets show that our method can improve the performance of pre-trained models significantly after test-time training and outperform competing methods significantly for HSI SR.

Test-time Training for Hyperspectral Image Super-resolution

TL;DR

A novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated so that the model can be further trained with them to improve performance.

Abstract

The progress on Hyperspectral image (HSI) super-resolution (SR) is still lagging behind the research of RGB image SR. HSIs usually have a high number of spectral bands, so accurately modeling spectral band interaction for HSI SR is hard. Also, training data for HSI SR is hard to obtain so the dataset is usually rather small. In this work, we propose a new test-time training method to tackle this problem. Specifically, a novel self-training framework is developed, where more accurate pseudo-labels and more accurate LR-HR relationships are generated so that the model can be further trained with them to improve performance. In order to better support our test-time training method, we also propose a new network architecture to learn HSI SR without modeling spectral band interaction and propose a new data augmentation method Spectral Mixup to increase the diversity of the training data at test time. We also collect a new HSI dataset with a diverse set of images of interesting objects ranging from food to vegetation, to materials, and to general scenes. Extensive experiments on multiple datasets show that our method can improve the performance of pre-trained models significantly after test-time training and outperform competing methods significantly for HSI SR.
Paper Structure (12 sections, 8 equations, 4 figures, 5 tables, 1 algorithm)

This paper contains 12 sections, 8 equations, 4 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Test-time Training
  4. Test-time Training for HSI SR
  5. Self-training with Pseudo Samples
  6. Source HSI SR Model
  7. Spectral Mixup
  8. Full Algorithm
  9. Experiments
  10. Experimental Setup
  11. Experimental Results
  12. Conclusion

Figures (4)

  • Figure 1: Exemplar samples from our SpectralWorld Dataset.
  • Figure 2: Visual Results on the Spectral World dataset for scaling factor $\times 4$. Spectral bands $80$, $45$, and $20$ are used as the R, G, and B channels of a color image for this visualization. We compare the baseline method LIIF implicit and its test-time trained version by our method.
  • Figure 3: Visual Results on the Spectral World dataset for scaling factor $\times 4$. Spectral bands $80$, $45$, and $20$ are used as the R, G, and B channels of a color image for this visualization. We compare the baseline method HSIwRGB Li_2022_WACV and its test-time trained version by our method.
  • Figure 4: Visual Results on the Spectral World dataset for scaling factor $\times 4$. Spectral bands $80$, $45$, and $20$ are used as the R, G, and B channels of a color image for this visualization. We compare the baseline method SwinIR swinIR and its test-time trained version by our method.