Hierarchical Dual-Change Collaborative Learning for UAV Scene Change Captioning

Abstract

This paper proposes a novel task for UAV scene understanding - UAV Scene Change Captioning (UAV-SCC) - which aims to generate natural language descriptions of semantic changes in dynamic aerial imagery captured from a movable viewpoint. Unlike traditional change captioning that mainly describes differences between image pairs captured from a fixed camera viewpoint over time, UAV scene change captioning focuses on image-pair differences resulting from both temporal and spatial scene variations dynamically captured by a moving camera. The key challenge lies in understanding viewpoint-induced scene changes from UAV image pairs that share only partially overlapping scene content due to viewpoint shifts caused by camera rotation, while effectively exploiting the relative orientation between the two images. To this end, we propose a Hierarchical Dual-Change Collaborative Learning (HDC-CL) method for UAV scene change captioning. In particular, a novel transformer, \emph{i.e.} Dynamic Adaptive Layout Transformer (DALT) is designed to adaptively model diverse spatial layouts of the image pair, where the interrelated features derived from the overlapping and non-overlapping regions are learned within the flexible and unified encoding layer. Furthermore, we propose a Hierarchical Cross-modal Orientation Consistency Calibration (HCM-OCC) method to enhance the model's sensitivity to viewpoint shift directions, enabling more accurate change captioning. To facilitate in-depth research on this task, we construct a new benchmark dataset, named UAV-SCC dataset, for UAV scene change captioning. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance on this task. The dataset and code will be publicly released upon acceptance of this paper.

Figures (10)

• Figure 1: Comparison of three image captioning tasks. (a) Image Captioning generates semantic descriptions for a single image captured from a fixed camera viewpoint. (b) Change Captioning produces textual descriptions of differences between two images captured from the same viewpoint. (c) UAV Scene Change Captioning generates natural language descriptions of scene variations between image pairs captured from moving camera viewpoints. The gray boxes show example captions generated for each task.
• Figure 2: Overview of the HDC-CL framework. It consists of three steps: (1) image alignment for aligning the before–after image pair to correct spatial shifts and ensure consistent correspondence between regions, (2) scene change distillation, where GE, CE, and DE denote the global, common, and different encoders, and (3) caption generation based on distilled features.
• Figure 3: An illustration of the Dynamic Adaptive Layout Transformer (DALT) pipeline. For clarity, this example uses $N = 9$ patches per image.
• Figure 4: Qualitative comparison of generated captions on the UAV-SCCSimple dataset. Text in different colors highlights the described objects or regions.
• Figure 5: Performance comparison between GPT‑4o and our HDC-CL model on the UAV-SCCSimple dataset.