On Efficient Transformer-Based Image Pre-training for Low-Level Vision

TL;DR

The paper tackles how pre-training impacts low-level vision tasks and introduces an efficient encoder-decoder-based transformer (EDT) to explore this space. It couples a window-based transformer with CK A-driven diagnostics to reveal how internal representations evolve across super-resolution, denoising, and deraining, showing task-specific benefits and the superiority of multi-related-task pre-training for data efficiency. The authors demonstrate state-of-the-art performance on several low-level tasks, with SR and deraining benefiting substantially from pre-training while denoising shows limited gains, and they provide comprehensive analyses across data scale, model size, and architecture comparisons. Overall, the work offers practical guidelines for pre-training in low-level vision and delivers pretrained models that balance accuracy and efficiency.

Abstract

Pre-training has marked numerous state of the arts in high-level computer vision, while few attempts have ever been made to investigate how pre-training acts in image processing systems. In this paper, we tailor transformer-based pre-training regimes that boost various low-level tasks. To comprehensively diagnose the influence of pre-training, we design a whole set of principled evaluation tools that uncover its effects on internal representations. The observations demonstrate that pre-training plays strikingly different roles in low-level tasks. For example, pre-training introduces more local information to higher layers in super-resolution (SR), yielding significant performance gains, while pre-training hardly affects internal feature representations in denoising, resulting in limited gains. Further, we explore different methods of pre-training, revealing that multi-related-task pre-training is more effective and data-efficient than other alternatives. Finally, we extend our study to varying data scales and model sizes, as well as comparisons between transformers and CNNs-based architectures. Based on the study, we successfully develop state-of-the-art models for multiple low-level tasks. Code is released at https://github.com/fenglinglwb/EDT.

Figures (21)

• Figure 1:
• Figure 2: Sub-figures (a)-(c) show CKA similarities between all pairs of layers in $\times 2$ SR, light streak deraining and level-15 denoising EDT-B models with single-task pre-training, and the corresponding similarities between with and without pre-training are shown in (e)-(g). Sub-figure (d) shows the cross-model comparison between SR and denoising models and (h) shows the ratios of layer similarity larger than 0.6, where "$s$" means the similarity between the current layer in SR and any layer in denoising.
• Figure 3: Sub-figures (a)-(d) show CKA similarities of $\times 2$ SR models, without pre-training as well as with pre-training on a single task ($\times 2$), unrelated tasks ($\times 2$, $\times 3$ SR, g15 denoising) and highly related tasks ($\times 2$, $\times 3$, $\times 4$ SR). Sub-figures (e)-(h) show the corresponding attention head mean distances of transformer blocks. We do not plot shifted local windows in (e)-(h) so that the last blue dotted line ("---") has no matching point. The red boxes indicate the same attention modules.
• Figure 4: PSNR improvements of single-task, multi-unrelated-task and multi-related-task pre-training for EDT-B in $\times 2$ SR.
• Figure 5: PSNR improvements of different data scales during single-task pre-training for EDT-B in $\times 2$ SR.