Re-evaluating the Need for Multimodal Signals in Unsupervised Grammar Induction

TL;DR

The paper questions whether multimodal inputs are necessary for unsupervised grammar induction and introduces LC-PCFG, a strong text-only baseline that injects pre-trained LLM sentence embeddings into a Compound PCFG framework. Across image and video benchmarks, LC-PCFG matches or surpasses state-of-the-art multimodal methods while using substantially fewer parameters and less training time, suggesting that large textual data with LLM representations can subsume multimodal benefits. Additional experiments show that re-adding visual signals to LC-PCFG does not improve performance, reinforcing the claim that multimodal inputs may be unnecessary in this regime. The work highlights the importance of strong vision-free baselines for evaluating multimodal approaches and points toward more efficient, text-driven solutions for grammar induction.

Abstract

Are multimodal inputs necessary for grammar induction? Recent work has shown that multimodal training inputs can improve grammar induction. However, these improvements are based on comparisons to weak text-only baselines that were trained on relatively little textual data. To determine whether multimodal inputs are needed in regimes with large amounts of textual training data, we design a stronger text-only baseline, which we refer to as LC-PCFG. LC-PCFG is a C-PFCG that incorporates em-beddings from text-only large language models (LLMs). We use a fixed grammar family to directly compare LC-PCFG to various multi-modal grammar induction methods. We compare performance on four benchmark datasets. LC-PCFG provides an up to 17% relative improvement in Corpus-F1 compared to state-of-the-art multimodal grammar induction methods. LC-PCFG is also more computationally efficient, providing an up to 85% reduction in parameter count and 8.8x reduction in training time compared to multimodal approaches. These results suggest that multimodal inputs may not be necessary for grammar induction, and emphasize the importance of strong vision-free baselines for evaluating the benefit of multimodal approaches.

Figures (4)

• Figure 1: Comparison with prior multimodal methods on image-assisted grammar induction. Prior works showed that paired images can improve grammar induction ($53.6 \rightarrow 59.3$ Corpus-level F1). We show that a strong text-only baselined line that incorporates embeddings from large language models (LLM) can match (and surpass) multimodal methods, suggesting that multimodal inputs may not be necessary for grammar induction ($53.6 \rightarrow 67.2$).
• Figure 2: Experimental Settings. We explore using large language model features for unsupervised grammar induction. We use three experimental settings. (1) the standard setting in which word representations are learned from scratch (Text Only), (2) prior methods that incorporate a multimodal regularization loss (+Pixels), and (3) our method, which uses pre-trained text-only LLM features (+LLM features). We show that LLM features can obtain state-of-the-art performance, without requiring multimodal regularization.
• Figure 3: LC-PCFG workflow. A sentence $\bm{x}$ is fed to an LLM to obtain a sentence embedding $\mathcal{E}(\bm{x})$. $\mathcal{E}(\bm{x})$ is passed through a fully-connected layer (FC), producing the latent $\bm{z}$. $\bm{z}$ is fed to the C-PCFG to obtain a constituency parse tree. Note that unlike prior work, our approach does not require multimodal data.
• Figure 4: Top: A parse tree generated by LC-PCFG. Bottom: The gold-parse tree. Note how the parser is generally able to form noun phrases. We note that all parsers in our experiments learn a right-branching bias, which follows intuition since English is a right-branching language.