Improved Image Captioning via Policy Gradient optimization of SPIDEr

TL;DR

The paper identifies the mismatch between traditional maximum-likelihood training and human judgments in image captioning and proposes a robust policy-gradient framework to directly optimize a new SPIDEr metric (SPICE + CIDEr). By employing Monte Carlo rollouts for value estimation and a learned baseline to reduce variance, the method achieves faster convergence and greater stability than MIXER, enabling optimization of both BCMR metrics and SPIDEr. Empirically, PG-BCMR attains state-of-the-art BCMR results on COCO, while SPIDEr optimization yields captions that humans rate as significantly better than those produced by MLE or BCMR-focused training. This approach advances captioning quality by balancing semantic fidelity with syntactic fluency and demonstrates practical benefits for applications requiring high-quality image descriptions.

Abstract

Current image captioning methods are usually trained via (penalized) maximum likelihood estimation. However, the log-likelihood score of a caption does not correlate well with human assessments of quality. Standard syntactic evaluation metrics, such as BLEU, METEOR and ROUGE, are also not well correlated. The newer SPICE and CIDEr metrics are better correlated, but have traditionally been hard to optimize for. In this paper, we show how to use a policy gradient (PG) method to directly optimize a linear combination of SPICE and CIDEr (a combination we call SPIDEr): the SPICE score ensures our captions are semantically faithful to the image, while CIDEr score ensures our captions are syntactically fluent. The PG method we propose improves on the prior MIXER approach, by using Monte Carlo rollouts instead of mixing MLE training with PG. We show empirically that our algorithm leads to easier optimization and improved results compared to MIXER. Finally, we show that using our PG method we can optimize any of the metrics, including the proposed SPIDEr metric which results in image captions that are strongly preferred by human raters compared to captions generated by the same model but trained to optimize MLE or the COCO metrics.

Figures (4)

• Figure 1: The value of each action is estimated as the average rewards received by its $K$ rollout sequences (i.e. $K=3$). Solid arrows indicate the sequence of actions being evaluated. The tokens in green and yellow are respectively BOS (beginning of sequence) and EOS (end of sequence) tokens. Sequences in blue are rollout sequences sampled from partial sequences. Note that rollout sequences do not always have the same length, as they are separately sampled from a stochastic policy.
• Figure 2: Model architecture of Show and Tell image captioning system Vinyals2015. The tokens in green and yellow are respectively BOS (beginning of sequence) and EOS (end of sequence) tokens. At testing time, output from previous time step $g_{t-1}$ is used as input in lieu of $y_{t-1}$.
• Figure 3: Results of human evaluation on 492 images randomly sampled from the COCO test set. We report the difference in percentage of "not bad" captions for each method compared to baseline 38% of MLE model.
• Figure 4: Performance of PG-BCMR (blue-triangle), MIXER-BCMR-A (red-dashed) and MIXER-BCMR (green-solid) on the validation set during the first 1 million gradient steps.