Consistency of Compositional Generalization across Multiple Levels

TL;DR

This work tackles the problem of achieving consistent compositional generalization across multiple levels of novel compositions. It introduces a meta-learning based framework that enforces progressive learning from simple to complex compositions via multilevel optimization, using K meta-weight-nets to selectively emphasize samples by complexity. A new dataset, GQA-CCG, is proposed to quantify cross-level consistency in VQA, along with a Consistency Score for evaluation. Empirical results show improved cross-level consistency and level-specific accuracy across VQA and TVG tasks, while preserving IID generalization, with ablations validating the importance of progressive learning and the meta-weighting strategy.

Abstract

Compositional generalization is the capability of a model to understand novel compositions composed of seen concepts. There are multiple levels of novel compositions including phrase-phrase level, phrase-word level, and word-word level. Existing methods achieve promising compositional generalization, but the consistency of compositional generalization across multiple levels of novel compositions remains unexplored. The consistency refers to that a model should generalize to a phrase-phrase level novel composition, and phrase-word/word-word level novel compositions that can be derived from it simultaneously. In this paper, we propose a meta-learning based framework, for achieving consistent compositional generalization across multiple levels. The basic idea is to progressively learn compositions from simple to complex for consistency. Specifically, we divide the original training set into multiple validation sets based on compositional complexity, and introduce multiple meta-weight-nets to generate sample weights for samples in different validation sets. To fit the validation sets in order of increasing compositional complexity, we optimize the parameters of each meta-weight-net independently and sequentially in a multilevel optimization manner. We build a GQA-CCG dataset to quantitatively evaluate the consistency. Experimental results on visual question answering and temporal video grounding, demonstrate the effectiveness of the proposed framework. We release GQA-CCG at https://github.com/NeverMoreLCH/CCG.

Figures (5)

• Figure 1: Illustration of multiple levels of novel compositions, including word-word level, phrase-word level and phrase-phrase level.
• Figure 2: Overview of the proposed framework.
• Figure 3: Validation set construction in the context of VQA, where the words in red font denote the longest phrase in the question.
• Figure 4: Overview of the pipeline for constructing GQA-CCG. (a) Preparations for constructing GQA-CCG. (b) Illustration of the sample generation process. The underlined words/phrases in red font denote the components of novel compositions. The words in black font in prompts of (b) denote the prompt for generating $[Q_{pw}, A_{pw}]$. The words in green font in prompts of (b) denote the info that needs to be replaced when generating $[Q_{ww}, A_{ww}]$.
• Figure 5: Qualitative comparisons between MMN+MLO (Ours) and MMN chen2021meta.