MICap: A Unified Model for Identity-aware Movie Descriptions

TL;DR

MICap introduces a unified single-stage model for identity-aware movie description that jointly handles caption generation with local identities and the fill-in-the-blanks task. It employs a multimodal memory and a shared auto-regressive Transformer decoder to seamlessly switch between predicting captions with identities and filling in identity labels, enabling cross-video identity consistency. A new identity-aware metric, iSPICE, evaluates captions by focusing on identity semantics within the scene graph, addressing shortcomings of traditional metrics. On the LSMDc dataset, MICap achieves state-of-the-art gains in FITB accuracy and measurable improvements in captioning metrics, demonstrating the benefit of joint training and cross-video identity reasoning. The work also provides a comprehensive analysis of attention patterns and proposes a scalable approach to identity-aware captioning that does not rely on external knowledge bases or ground-truth identity mappings beyond the videoset.

Abstract

Characters are an important aspect of any storyline and identifying and including them in descriptions is necessary for story understanding. While previous work has largely ignored identity and generated captions with someone (anonymized names), recent work formulates id-aware captioning as a fill-in-the-blanks (FITB) task, where, given a caption with blanks, the goal is to predict person id labels. However, to predict captions with ids, a two-stage approach is required: first predict captions with someone, then fill in identities. In this work, we present a new single stage approach that can seamlessly switch between id-aware caption generation or FITB when given a caption with blanks. Our model, Movie-Identity Captioner (MICap), uses a shared auto-regressive decoder that benefits from training with FITB and full-caption generation objectives, while the encoder can benefit from or disregard captions with blanks as input. Another challenge with id-aware captioning is the lack of a metric to capture subtle differences between person ids. To this end, we introduce iSPICE, a caption evaluation metric that focuses on identity tuples created through intermediate scene graphs. We evaluate MICap on Large-Scale Movie Description Challenge (LSMDC), where we show a 4.2% improvement in FITB accuracy, and a 1-2% bump in classic captioning metrics.

Figures (10)

• Figure 1: Identity-aware captioning. Left: To understand the story in a set of videos, captions refer to characters by a unique local identifier (e.g. P1, P2, $\ldots$). The Fill-in-the-blanks (FITB) task provides these captions with blanks (removing names) and asks a model to fill local person ids. Middle: End-to-end captioning for a videoset is achieved in two stages fillin. First, captions are generated with someone tags, and then the FITB module is applied to fill-in names. Right: We propose a single-stage encoder-decoder id-aware captioning approach that can switch between generating the caption with ids or filling in the ids in a caption, jointly learning from both tasks.
• Figure 2: Identity-aware captioning. Left: illustrates the Transformer Encoder used to capture multimodal inputs such as text (blanks), action, semantic, and face. These tokens are used as memory for the Transformer Decoders. Right: the same Transformer Decoder can be used for both tasks of full caption generation and fill-in-the-blanks (FITB). The model is trained end-to-end with losses applied to tokens indicated in purple. Text tokens are not presented to the decoder for full caption generation. Joint training improves knowledge sharing resulting in performance improvements.
• Figure 3: We show a qualitative example of our joint training approach. The dataset is highly challenging, with shot changes and dark scenes that are typical in movies. Yet our model is able to perform reasonably well in this example. While the predicted captions (Pred) are different from the ground-truth (GT), they capture the overall meaning. MICap predicts diverse ids correctly in this case and does not overfit to only predicting P1, or P1 and P2. In fact, in the last clip, as P3 turns (indicated in GT), we see P4 sitting at the table (indicated in Pred), which is a correct caption! The last clip also highlights challenges of evaluating captions correctly.
• Figure 4: Cross-attention scores density plots for the id-aware captioning task. We group decoder output tokens into two types: person id label tokens (PT), and caption tokens that represent other words (CT). Attention scores are grouped across the three input visual features capturing actions (I3D, left), faces (Arcface, middle), and semantic content (CLIP, right). Please refer to \ref{['subsec:supp_attn_idwarecap']} for a discussion.
• Figure 5: We show 5 examples of our model's attention scores on the FITB task. For each example (row), we show the captionset (with blanks) and the attention scores grouped in various ways. The left column shows the attention score for each blank across all face detections in the video. The middle column shows attention scores for face detections grouped by clusters in each video. C1/V1 indicates faces appearing in cluster 1 and video 1, while C1/V2 indicates faces of the same cluster 1 appearing in video 2. The right column shows attention scores of each blank for face clusters (across videos). For each row in the attention scores, we indicate the ground-truth (GT) and predicted (Pr) person id label and the video index (V1 .. V5) in which this blank appeared. See \ref{['subsec:supp_attnfitb']} for a discussion.