Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing

TL;DR

This study interrogates whether increasing model size reliably improves compositional generalization in semantic parsing. It compares full fine-tuning, prompt tuning, and in-context learning across encoder-decoder and decoder-only architectures up to hundreds of billions of parameters, using CFQ, COGS, GeoQuery, and SMCalFlow-CS. Key findings show flat or negative scaling for fine-tuning, positive but often weaker scaling for in-context learning, and more favorable scaling with prompt tuning, especially at larger sizes. Error analyses reveal that larger models better capture output syntax but are prone to certain overfitting risks, and that retriever quality and prompt design critically influence scaling benefits. The results suggest promising directions around constrained decoding, alternative output representations, and improved retrieval strategies to harness scale for compositional generalization.

Abstract

Despite their strong performance on many tasks, pre-trained language models have been shown to struggle on out-of-distribution compositional generalization. Meanwhile, recent work has shown considerable improvements on many NLP tasks from model scaling. Can scaling up model size also improve compositional generalization in semantic parsing? We evaluate encoder-decoder models up to 11B parameters and decoder-only models up to 540B parameters, and compare model scaling curves for three different methods for applying a pre-trained language model to a new task: fine-tuning all parameters, prompt tuning, and in-context learning. We observe that fine-tuning generally has flat or negative scaling curves on out-of-distribution compositional generalization in semantic parsing evaluations. In-context learning has positive scaling curves, but is generally outperformed by much smaller fine-tuned models. Prompt-tuning can outperform fine-tuning, suggesting further potential improvements from scaling as it exhibits a more positive scaling curve. Additionally, we identify several error trends that vary with model scale. For example, larger models are generally better at modeling the syntax of the output space, but are also more prone to certain types of overfitting. Overall, our study highlights limitations of current techniques for effectively leveraging model scale for compositional generalization, while our analysis also suggests promising directions for future work.

Figures (13)

• Figure 1: Our experimental setup.
• Figure 2: Aggregated scaling curves for compositional splits of different datasets. Note that in-context learning with an oracle retriever (dashed) cannot be compared directly with other methods as it has access to the gold output.
• Figure 3: Percentage of predictions that contain unbalanced parentheses, as an estimate of syntax errors.
• Figure 4: Percentage of predictions where the output exactly matches an output seen in the training set on GeoQuery compositional splits (left). Percentage of predictions where the output only contains functions from a single domain on SMCalFlow-CS cross-domain splits (right).
• Figure 5: Average number of tokens in the prediction. The dotted brown and black lines show the average target lengths in the training and test set, respectively.