Ship in Sight: Diffusion Models for Ship-Image Super Resolution

TL;DR

This work addresses the challenge of enhancing low-resolution ship images for maritime surveillance by leveraging a diffusion-based latent super-resolution framework. The authors introduce StableShip-SR, a class- and time-aware latent diffusion model trained with text conditioning from CLIP and guided by a frozen classifier, built atop a pre-trained Stable Diffusion backbone. A large ShipSpotting dataset (20 classes, ~507k high-resolution samples after filtering) supports training and downstream tasks like ship classification and detection. Empirical results show StableShip-SR achieving superior realism and downstream task performance (lower FID and better detection/classification) compared with state-of-the-art SR methods, while ablations highlight the value of conditioning components. The work provides practical impact for maritime monitoring and offers resources and code to advance research in ship-focused image restoration and analysis.

Abstract

In recent years, remarkable advancements have been achieved in the field of image generation, primarily driven by the escalating demand for high-quality outcomes across various image generation subtasks, such as inpainting, denoising, and super resolution. A major effort is devoted to exploring the application of super-resolution techniques to enhance the quality of low-resolution images. In this context, our method explores in depth the problem of ship image super resolution, which is crucial for coastal and port surveillance. We investigate the opportunity given by the growing interest in text-to-image diffusion models, taking advantage of the prior knowledge that such foundation models have already learned. In particular, we present a diffusion-model-based architecture that leverages text conditioning during training while being class-aware, to best preserve the crucial details of the ships during the generation of the super-resoluted image. Since the specificity of this task and the scarcity availability of off-the-shelf data, we also introduce a large labeled ship dataset scraped from online ship images, mostly from ShipSpotting\footnote{\url{www.shipspotting.com}} website. Our method achieves more robust results than other deep learning models previously employed for super resolution, as proven by the multiple experiments performed. Moreover, we investigate how this model can benefit downstream tasks, such as classification and object detection, thus emphasizing practical implementation in a real-world scenario. Experimental results show flexibility, reliability, and impressive performance of the proposed framework over state-of-the-art methods for different tasks. The code is available at: https://github.com/LuigiSigillo/ShipinSight .

Figures (5)

• Figure 1: Comparative sample images showcasing the super-resolution capabilities of different models. The visual assessments highlight variations in image quality, clarity, and detail enhancement achieved by each respective model in the upscale process. Ours is in the last column.
• Figure 2: Overview of Stable Ship-Image Super-Resolution framework. We pre-trained the classifier and trained only the class- and time- aware encoder. The other parts of the framework are frozen, which speeds up the overall training process.
• Figure 3: Comparison between LR image and SR images reconstructed by other state-of-the-art methods and our proposed model. The red bounding box contains a zoomed-in part of the image in order to have a better understanding of the image resolution quality of our method compared to the others.
• Figure 4: Comparison of SSIM (a) and PSNR (b), which measure the quality of the reconstructed images. We confront those results against the classification accuracy reached with the generated images on the ShipsSpotting dataset. This comparison shows how the classical reconstruction metrics are not a measure of how well a model performs in a super-resolution scenario since the accuracy and the FID reached are lower concerning other models that have lower SSIM and PSNR.
• Figure 5: Comparison for precision scores on the Seaships dataset. We employ the models pre-trained on our dataset, thus performing zero-shot super resolution, improving the detection of a finetuned YOLOv7Wang_2023_CVPR on Seaships.