CausalSR: Structural Causal Model-Driven Super-Resolution with Counterfactual Inference

TL;DR

CausalSR reframes image super-resolution as a causal inference problem, modeling degradation as a structured process with latent factors $\mathbf{Z}$, semantic $\mathbf{S}$, and context $\mathbf{C}$ to enable robust, semantically guided restoration. It introduces a semantic-contextual translation module, counterfactual learning, and an intervention mechanism built on a probabilistic graph prior, with a multi-objective objective and convergence guarantees. Empirically, CausalSR achieves significant PSNR gains (e.g., $0.86$–$1.21$ dB at $\times4$) over state-of-the-art on diverse benchmarks and excels under complex degradations, while providing interpretable insights into degradation dynamics. The work demonstrates that incorporating structural causal reasoning and semantic guidance into low-level vision tasks enhances generalization and restoration quality, albeit with challenges in scalability and optimization stability.

Abstract

Physical and optical factors interacting with sensor characteristics create complex image degradation patterns. Despite advances in deep learning-based super-resolution, existing methods overlook the causal nature of degradation by adopting simplistic black-box mappings. This paper formulates super-resolution using structural causal models to reason about image degradation processes. We establish a mathematical foundation that unifies principles from causal inference, deriving necessary conditions for identifying latent degradation mechanisms and corresponding propagation. We propose a novel counterfactual learning strategy that leverages semantic guidance to reason about hypothetical degradation scenarios, leading to theoretically-grounded representations that capture invariant features across different degradation conditions. The framework incorporates an adaptive intervention mechanism with provable bounds on treatment effects, allowing precise manipulation of degradation factors while maintaining semantic consistency. Through extensive empirical validation, we demonstrate that our approach achieves significant improvements over state-of-the-art methods, particularly in challenging scenarios with compound degradations. On standard benchmarks, our method consistently outperforms existing approaches by significant margins (0.86-1.21dB PSNR), while providing interpretable insights into the restoration process. The theoretical framework and empirical results demonstrate the fundamental importance of causal reasoning in understanding image restoration systems.

Figures (7)

• Figure 1: Proposed causal graph structure for image super-resolution. The graph illustrates the relationships between different domains: from high-resolution image ($X$) to semantic features ($S$) extracted by CLIP, then to contextual features ($C$) through semantic translation, which influence degradation factors ($Z$) that ultimately determine the low-resolution observation ($Y$).
• Figure 2: Overview of the proposed CausalSR framework. The network architecture consists of three main components: (a) Encoder Network extracts hierarchical features from the input LR image through a sequence of convolutional and residual blocks. (b) Latent Space models degradation factors via a probabilistic causal graph that captures structured dependencies. The latent variables are learned through a variational inference scheme with structure priors. (c) Decoder Network reconstructs the SR output through upsampling and refinement modules.
• Figure 3: Visual comparison of different methods for 4$\times$ super-resolution under complex degradation (blur + noise). From left to right: (a) LR input with complex degradation (blur kernel width = 2.0, noise level = 15), (b) EDSR, (c) RCAN, (d) SwinIR, (e) EDT, (f) CausalSR (ours), and (g) HR ground truth.
• Figure 4: Visual results on RealSR dataset with multiple upscaling factors. Note the superior text restoration quality in $\times$3 results, where CausalSR better preserves character legibility and edge sharpness.
• Figure 5: Visual comparison of restoration results under multiple degradation scenarios on Urban100 dataset ($\times$4). From left to right: HR reference image and degraded versions: original LR, LR with noise ($\sigma$=15), LR with Gaussian blur ($\sigma$=2.0), and LR with JPEG compression ($Q$=30). CausalSR demonstrates superior restoration of fine architectural textures and structural details across all degradation types.