Federated Learning for Blind Image Super-Resolution

TL;DR

The paper tackles privacy-preserving blind image super-resolution by integrating Federated Learning to learn real-world degradations directly from user data without centralizing it. It introduces an FL-based framework with cross-degradation evaluation, assigning each client a single degradation type and modeling degradations with a high-order formulation; non-IID distributions across clients are simulated using a Dirichlet process, and a centralized one-client upper bound serves as a reference. Experiments with SRResNet and RRDB show that more participating clients generally improve robustness to complex degradations, while noise and JPEG artifacts dominate performance variations; degradation diversity still challenges single-degradation training. The work delivers new benchmarks and guidelines for future FL-based blind SR research, while noting practical constraints such as VRAM demands and the need for memory-efficient simulation techniques to scale to broader setups.

Abstract

Traditional blind image SR methods need to model real-world degradations precisely. Consequently, current research struggles with this dilemma by assuming idealized degradations, which leads to limited applicability to actual user data. Moreover, the ideal scenario - training models on data from the targeted user base - presents significant privacy concerns. To address both challenges, we propose to fuse image SR with federated learning, allowing real-world degradations to be directly learned from users without invading their privacy. Furthermore, it enables optimization across many devices without data centralization. As this fusion is underexplored, we introduce new benchmarks specifically designed to evaluate new SR methods in this federated setting. By doing so, we employ known degradation modeling techniques from SR research. However, rather than aiming to mirror real degradations, our benchmarks use these degradation models to simulate the variety of degradations found across clients within a distributed user base. This distinction is crucial as it circumvents the need to precisely model real-world degradations, which limits contemporary blind image SR research. Our proposed benchmarks investigate blind image SR under new aspects, namely differently distributed degradation types among users and varying user numbers. We believe new methods tested within these benchmarks will perform more similarly in an application, as the simulated scenario addresses the variety while federated learning enables the training on actual degradations.

Figures (13)

• Figure 1: Proposed setup of FL for blind image SR. Each client contributes by training on their local data and individual degradation type to the collaborative training of a global model. The federated server collects the training results, updates the global model, and synchronizes the global model with the clients. During testing, the trained global model is evaluated on variations of the test dataset with all possible combinations of degradations.
• Figure 2: The PSNR (db) results of SRResNet and RRDB with $\times 4$ scaling and varying number of clients (4, 8, 12, 16) on different test datasets (Set14, BSD100, Urban100, Manga109). We tested the SR model on eight degradation variations listed in the legend on the right side (clean, blur, noise, JPEG, and combinations). The degradation legend shows triangles, circles, and squares for single, double, and triple degradations, respectively.
• Figure 3: Experimental results of different degradation distributions on BSD100. On the left side is the degradation distribution during training. The resulting performance on the respective degraded test set is on the right side (relative to uniform distribution). We identified five clusters that exhibit similar performance variations (see left side: Few Clean or Few Blur; Few Noise; Few JPEG; Few JPEG but Many Noise, Few Clean but Many Blur).
• Figure 4: Visual results on Manga109 with blur and noise degradations in the LR image.
• Figure 5: Experimental results of different degradation distributions on Set5. On the left side is the degradation distribution during training. The resulting performance on the respective degraded test set is on the right side (relative to uniform distribution).