Imagine How To Change: Explicit Procedure Modeling for Change Captioning

TL;DR

This work introduces ProCap, a novel framework that reformulates change modeling from static image comparison to dynamic procedure modeling, and introduces learnable procedure queries to prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text.

Abstract

Change captioning generates descriptions that explicitly describe the differences between two visually similar images. Existing methods operate on static image pairs, thus ignoring the rich temporal dynamics of the change procedure, which is the key to understand not only what has changed but also how it occurs. We introduce ProCap, a novel framework that reformulates change modeling from static image comparison to dynamic procedure modeling. ProCap features a two-stage design: The first stage trains a procedure encoder to learn the change procedure from a sparse set of keyframes. These keyframes are obtained by automatically generating intermediate frames to make the implicit procedural dynamics explicit and then sampling them to mitigate redundancy. Then the encoder learns to capture the latent dynamics of these keyframes via a caption-conditioned, masked reconstruction task. The second stage integrates this trained encoder within an encoder-decoder model for captioning. Instead of relying on explicit frames from the previous stage -- a process incurring computational overhead and sensitivity to visual noise -- we introduce learnable procedure queries to prompt the encoder for inferring the latent procedure representation, which the decoder then translates into text. The entire model is then trained end-to-end with a captioning loss, ensuring the encoder's output is both temporally coherent and captioning-aligned. Experiments on three datasets demonstrate the effectiveness of ProCap. Code and pre-trained models are available at https://github.com/BlueberryOreo/ProCap

Figures (14)

• Figure 1: Comparison between static image pair modeling and our proposed dynamic procedure modeling. Dynamic procedures offer temporal cues: the yellow cylinder, initially partly obscured by the green cube, changes its location.
• Figure 2: Our two-stage ProCap framework. In the first stage, Explicit Procedure Modeling, a procedure encoder learns change dynamics from keyframes sampled from the generated explicit procedure frames. In the second stage, Implicit Procedure Captioning, learnable procedure queries, instead of explicit frames, prompt the encoder to infer an implicit representation for captioning.
• Figure 3: Four masking schemes in the proposed multi-granularity strategy. We mask visual patch embeddings for reconstruction during training; the masks are visualized at the patch level for clarity.
• Figure 4: Comparison of CIDEr scores across four sampling strategies with respect to the number of pseudo-frames $l$ on CLEVR-Change dataset (left) and Spot-the-Diff dataset (right). Each strategy is set to sample two key frames from the pseudo-frames.
• Figure 5: Uncontrollable predicted intermediate frames examples of diffusion-based FI models. Samples (a) and (b) show an unexpected object prediction, while samples (c) and (d) show an unexpected motion generation.