DualTSR: Unified Dual-Diffusion Transformer for Scene Text Image Super-Resolution

Abstract

Scene Text Image Super-Resolution (STISR) aims to restore high-resolution details in low-resolution text images, which is crucial for both human readability and machine recognition. Existing methods, however, often depend on external Optical Character Recognition (OCR) models for textual priors or rely on complex multi-component architectures that are difficult to train and reproduce. In this paper, we introduce DualTSR, a unified end-to-end framework that addresses both issues. DualTSR employs a single multimodal transformer backbone trained with a dual diffusion objective. It simultaneously models the continuous distribution of high-resolution images via Conditional Flow Matching and the discrete distribution of textual content via discrete diffusion. This shared design enables visual and textual information to interact at every layer, allowing the model to infer text priors internally instead of relying on an external OCR module. Compared with prior multi-branch diffusion systems, DualTSR offers a simpler end-to-end formulation with fewer hand-crafted components. Experiments on synthetic Chinese benchmarks and a curated real-world evaluation protocol show that DualTSR achieves strong perceptual quality and text fidelity.

Figures (13)

• Figure 1: A comparison of text super-resolution architectures. Our proposed method unifies image generation and text recognition within a single model. (a) Baseline: Directly models the conditional distribution of high-resolution images from low-resolution inputs, without a text-based prior. (b) DiffTSR zhang2024diffusion: Independently models the image and text priors, subsequently fusing their representations using a Multi-modality Module (MoM). (c) Proposed Method: A unified multimodal transformer jointly optimizes for image generation and text recognition through two distinct but interconnected diffusion processes.
• Figure 2: Illustration of the joint attention mechanism. Following the MM-DiT block design in SD3 esser2024scaling, we incorporate a joint attention module that enables the image latent refinement and text reconstruction processes to operate in parallel.
• Figure 3: Qualitative comparison for the synthetic dataset CTR-TSR-Test with different methods on $\times$4 scale. The comparison methods include ESRGAN wang2018ESRGAN, MSRResNet 9022144, SwinIR liang2021swinir, SRFormer zhou2023srformer, MARCONet li2023learning, MARCONet++ li2025enhanced, DiffTSR zhang2024diffusion and our method.
• Figure 4: Qualitative comparison for RealCE with different methods on $\times$4 scale. The comparison methods include ESRGAN wang2018ESRGAN, MSRResNet 9022144, SwinIR liang2021swinir, SRFormer zhou2023srformer, MARCONet li2023learning, MARCONet++ li2025enhanced, DiffTSR zhang2024diffusion and our method.
• Figure 5: Trade-off between perceptual quality and text fidelity across sampling steps. FID improves with more sampling steps, reaching its best value around $\sim\!40$, while NED worsens as text strokes become over-smoothed. We select $4$ steps as a balanced operating point.