DECap: Towards Generalized Explicit Caption Editing via Diffusion Mechanism

TL;DR

DECap tackles the limited generalization of explicit caption editing models by reframing editing as a discrete diffusion process. It introduces an edit-based noising stage that corrupts ground-truth captions with word-level edits and a denoising stage that simultaneously predicts per-word edit operations and content words, enabling fast, non-autoregressive inference. Across in-domain and out-of-domain Ref-Caps, DECap demonstrates strong generalization, and it also competes with diffusion-based captioning models while offering improved speed and potential word-level controllability. This work presents a unified, diffusion-based framework for both caption editing and generation with explainable edit operations.

Abstract

Explicit Caption Editing (ECE) -- refining reference image captions through a sequence of explicit edit operations (e.g., KEEP, DETELE) -- has raised significant attention due to its explainable and human-like nature. After training with carefully designed reference and ground-truth caption pairs, state-of-the-art ECE models exhibit limited generalization ability beyond the original training data distribution, i.e., they are tailored to refine content details only in in-domain samples but fail to correct errors in out-of-domain samples. To this end, we propose a new Diffusion-based Explicit Caption editing method: DECap. Specifically, we reformulate the ECE task as a denoising process under the diffusion mechanism, and introduce innovative edit-based noising and denoising processes. Thanks to this design, the noising process can help to eliminate the need for meticulous paired data selection by directly introducing word-level noises for training, learning diverse distribution over input reference caption. The denoising process involves the explicit predictions of edit operations and corresponding content words, refining reference captions through iterative step-wise editing. To further efficiently implement our diffusion process and improve the inference speed, DECap discards the prevalent multi-stage design and directly generates edit operations and content words simultaneously. Extensive ablations have demonstrated the strong generalization ability of DECap in various scenarios. More interestingly, it even shows great potential in improving the quality and controllability of caption generation.

Figures (12)

• Figure 1: (a) An image example and its corresponding ground-truth caption (GT-Cap). (b) Data distribution of the COCO-EE dataset wang2022explicit. The distribution of the training set and test set are very similar, where most of the editing instances have ratios ranging from 0.4 to 0.6. (c) Editing results of state-of-the-art ECE model TIger and our DECap. The in-domain Ref-Cap sample is from the COCO-EE test set, and out-of-domain Ref-Cap samples are constructed by replacing the GT-Cap with other words, e.g., predicted by BERT (or sentences generated by pretrained captioning models).
• Figure 2: Edit-based noising process for DECap. Blue represents the REPLACE operation, red represents the DELETE operation, purple represents the INSERT operation, white and grey represent the KEEP operation for original word and random word respectively.
• Figure 3: The edit-based denoising step and architecture of DECap. DECap will predict a sequence of edit operations and content words to transform the caption. Specifically, contents words are used only when the predicted corresponding edit operation is INSERT or REPLACE, while the rest of the predicted words are abandoned, i.e., the shaded words.
• Figure 4: Performance on two kinds of out-of-domain GT-based reference captions constructed from COCO-EE test set. All models were trained on the COCO-EE training set. "Ref-Caps" denotes the initial quality of given reference captions, and "TIger-N" denotes the TIger trained with unpaired data.
• Figure 5: Controllability of DECap. The grey lines represent random words from the vocabulary, and other colored words represent the manually placed control words.