Sequential Compositional Generalization in Multimodal Models

TL;DR

The paper introduces CompAct, a multimodal dataset derived from EK-100 to probe sequential compositional generalization in egocentric kitchen videos. It defines two tasks—next utterance prediction and atom classification—and evaluates a broad set of unimodal and multimodal baselines, including pretrained models, with cross-modal attention and prompting-based evaluation. Across experiments, multimodal inputs consistently outperform text-only baselines, with visual features (notably ImageBind-based representations) delivering the strongest gains, though generalization to novel compositions remains challenging, especially in out-of-domain settings. The work highlights the potential and limits of current multimodal foundation models for systematic compositional generalization and calls for further research into robust fusion strategies and open-domain data.

Abstract

The rise of large-scale multimodal models has paved the pathway for groundbreaking advances in generative modeling and reasoning, unlocking transformative applications in a variety of complex tasks. However, a pressing question that remains is their genuine capability for stronger forms of generalization, which has been largely underexplored in the multimodal setting. Our study aims to address this by examining sequential compositional generalization using \textsc{CompAct} (\underline{Comp}ositional \underline{Act}ivities)\footnote{Project Page: \url{http://cyberiada.github.io/CompAct}}, a carefully constructed, perceptually grounded dataset set within a rich backdrop of egocentric kitchen activity videos. Each instance in our dataset is represented with a combination of raw video footage, naturally occurring sound, and crowd-sourced step-by-step descriptions. More importantly, our setup ensures that the individual concepts are consistently distributed across training and evaluation sets, while their compositions are novel in the evaluation set. We conduct a comprehensive assessment of several unimodal and multimodal models. Our findings reveal that bi-modal and tri-modal models exhibit a clear edge over their text-only counterparts. This highlights the importance of multimodality while charting a trajectory for future research in this domain.

Figures (11)

• Figure 1: Overview of the compositional generalization setup in our CompAct dataset. During training, the model has seen the verbs wash, close, put down, throw, open, pour, cut and pick up with the objects garbage bin, fridge, sesame oil, onion, and celery. It has never seen the composition of cut and celery, and thus needs to generalize to this novel composition at test time.
• Figure 2: Overview of the AVL baseline which integrates image, object-level, audio, and textual features utilizing two crossmodal attention blocks incorporated within an encoder-decoder Transformer to predict the next utterance.
• Figure 3: Next Utterance Prediction qualitative results. Models consider different combinations of input modality, as described in Section \ref{['sec:fup_models']}. In the predictions, blue refers to correct, orange incorrect and purple semantically close.
• Figure 4: Plot on the top demonstrates the distribution of atoms while the plot on the bottom shows the distribution of compounds for the train/validation/test splits in compositional split setup.
• Figure 5: Curating dataset instances for compositional generalization. Targets such as put_downknife and rinsepan have never been observed by the learner during the training phase.