Perceive, Understand and Restore: Real-World Image Super-Resolution with Autoregressive Multimodal Generative Models

TL;DR

This work addresses Real-ISR in challenging real-world scenes by leveraging autoregressive multimodal large language models. It introduces PURE, which Perceives degradation, Understands content via semantic descriptions, and Restores HQ images using an autoregressive token-based pipeline built on Lumina-mGPT with an enhanced vision tokenizer. A key novelty is the entropy-based dynamic Top-k sampling that adapts per-token uncertainty to better preserve structure. Experiments show improved perceptual realism and robustness in complex scenes, albeit with higher computational cost. The work highlights the potential of end-to-end autoregressive multimodal models for robust Real-ISR.

Abstract

By leveraging the generative priors from pre-trained text-to-image diffusion models, significant progress has been made in real-world image super-resolution (Real-ISR). However, these methods tend to generate inaccurate and unnatural reconstructions in complex and/or heavily degraded scenes, primarily due to their limited perception and understanding capability of the input low-quality image. To address these limitations, we propose, for the first time to our knowledge, to adapt the pre-trained autoregressive multimodal model such as Lumina-mGPT into a robust Real-ISR model, namely PURE, which Perceives and Understands the input low-quality image, then REstores its high-quality counterpart. Specifically, we implement instruction tuning on Lumina-mGPT to perceive the image degradation level and the relationships between previously generated image tokens and the next token, understand the image content by generating image semantic descriptions, and consequently restore the image by generating high-quality image tokens autoregressively with the collected information. In addition, we reveal that the image token entropy reflects the image structure and present a entropy-based Top-k sampling strategy to optimize the local structure of the image during inference. Experimental results demonstrate that PURE preserves image content while generating realistic details, especially in complex scenes with multiple objects, showcasing the potential of autoregressive multimodal generative models for robust Real-ISR. The model and code will be available at https://github.com/nonwhy/PURE.

Figures (9)

• Figure 1: Examples of of (a) inaccurate and (b) unnatural restoration of current SD-based Real-ISR methods, such as SeeSRseesr and OSEDiff osediff, in complex and heavily degraded scenes.
• Figure 2: Overview of PURE. We utilize VQGAN's encoder as the vision tokenizer to generate discrete tokens of the given LQ image. The instruction is passed to the text tokenizer to generate text tokens, which are then concatenated with image tokens. Subsequently, the backbone transformer performs autoregressive prediction of the image degradation, semantic understanding and HQ image tokens. Finally, the output image tokens are passed to the vision de-tokenizer and decoded to the restored image.
• Figure 3: Visualization of entropy distribution in image token generation. The entropy heatmap shows token-wise uncertainty, where darker regions indicate higher entropy values and lighter regions represent lower entropy values.
• Figure 4: Qualitative comparisons of different Real-ISR methods. Please zoom in for a better view.
• Figure 5: User study results on PURE and five state-of-the-art diffusion-based Real-ISR methods (StableSR, PASD, DiffBIR, SeeSR, OSEDiff) with 50 real-world images.