Space-scale Exploration of the Poor Reliability of Deep Learning Models: the Case of the Remote Sensing of Rooftop Photovoltaic Systems

TL;DR

This work tackles the reliability gap in mapping rooftop PV installations from aerial imagery caused by distribution shifts across acquisition conditions, geography, and resolution. It combines a benchmark dataset (BDAPPV) with space-scale analysis, using a by-design Scattering transform and a post-hoc WCAM explainability method to diagnose how CNNs rely on features across image scales. The study reveals that acquisition conditions predominantly drive performance loss, with CNNs depending on coarse-scale discriminative features and noisy fine-scale content; a targeted data augmentation strategy (Blurring + Wavelet Perturbation) improves robustness without sacrificing precision. The findings offer practical guidance for training data selection and augmentation to better support public authorities and TSOs in generating reliable rooftop PV registries, facilitating more accurate grid integration planning. The approach provides a framework for evaluating and improving robustness under distribution shifts, with potential extension to other remote-sensing mapping tasks and foundation-model-based pipelines.

Abstract

Photovoltaic (PV) energy grows rapidly and is crucial for the decarbonization of electric systems. However, centralized registries recording the technical characteristifs of rooftop PV systems are often missing, making it difficult to accurately monitor this growth. The lack of monitoring could threaten the integration of PV energy into the grid. To avoid this situation, the remote sensing of rooftop PV systems using deep learning emerged as a promising solution. However, existing techniques are not reliable enough to be used by public authorities or transmission system operators (TSOs) to construct up-to-date statistics on the rooftop PV fleet. The lack of reliability comes from the fact that deep learning models are sensitive to distribution shifts. This work proposes a comprehensive evaluation of the effects of distribution shifts on the classification accuracy of deep learning models trained to detect rooftop PV panels on overhead imagery. We construct a benchmark to isolate the sources of distribution shift and introduce a novel methodology that leverages explainable artificial intelligence (XAI) and decomposition of the input image and model's decision in terms of scales to understand how distribution shifts affect deep learning models. Finally, based on our analysis, we introduce a data augmentation technique meant to improve the robustness of deep learning classifiers to varying acquisition conditions. We show that our proposed approach outperforms competing methods. We discuss some practical recommendations for mapping PV systems using overhead imagery and deep learning models.

Figures (13)

• Figure 1: Examples of images of the same PV panels but with different providers and acquisition dates (Up Google, down: IGN)
• Figure 2: Test images on which a model trained on Google images (downsampled to 20 cm/px of GSD, "Google baseline") is evaluated. "Google 10 cm/pixel" corresponds to the source Google image, before downsampling and evaluates the effect of varying ground sampling distances. "Google OOD" corresponds to Google images taken outside of France. "IGN" corresponds to images depicting the same installations as Google baseline but with a different provider
• Figure 3: Decomposition of a PV panel into scales
• Figure 4: Image and associated two-level dyadic wavelet transform with indications to interpret the wavelet transform of the image. "Horizontal," "diagonal," and "vertical" indicate the direction of the detail coefficients. The direction is the same at all levels
• Figure 5: A scattering propagator $U_J$ applied to $x$ computes each $U[\lambda_1] x = |x \star \psi_{\lambda_1}|$ and outputs $S_J [\emptyset] x = x \star \phi_{2^J}$ (black arrow). Applying $U_J$ to each $U[\lambda_1] x$ computes all $U[\lambda_1,\lambda_2]x$ and outputs $S_J[\lambda_1] = U[\lambda_1] \star \phi_{2^J}$ (black arrows). Applying $U_J$ iteratively to each $U[p]x$ outputs $S_J[p]x = U[p]x \star \phi_{2^J}$ (black arrows) and computes the next path layer. Figure borrowed from bruna_invariant_2013. Note: On the image, the input $x$ corresponds to $f$ and $\lambda=2^j r$ is a frequency variable corresponding the the $j^\textrm{th}$ scale with $r$ rotations