DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalization

TL;DR

DiNeR introduces a large-scale, realistic Chinese dataset for evaluating compositional generalization in dish-name recognition, where models predict a dish name formed from food, action, and flavor components using recipe instructions. It combines a maximal-matching dish-name parser with glossaries and TMCD distribution-based splits to create diverse generalization forms and robust evaluation. Baselines include T5 with continual pretraining and compositional prompting, plus GPT-3.5 with selective demonstrations, showing improvements over standard fine-tuning but persistent gaps between ID and OOD performance under large distribution shifts. The work demonstrates dataset controllability across distribution shifts and data scales, provides insights into which components and generalization forms are harder, and highlights directions for future multilingual expansion and methodological advances in compositional generalization in NLP.

Abstract

Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DiNeR) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition. Code and data are available at https://github.com/Jumpy-pku/DiNeR.

Figures (5)

• Figure 1: An example of compositional generalization in dish name prediction. Models are required to learn knowledge about braised and beef brisket and predict the composed dish name braised beef brisket given the corresponding instructions.
• Figure 2: An example of selective demonstrations. GPT-3.5 learns the flavor hot and sour from demonstrations, but fails to figure out the staple ingredient.
• Figure 3: F1-score on TMCD splits of each kind of components. When calculating the F1 score of one kind of components, the samples whose dish name doesn't have this kind of components will be ignored
• Figure 4: F1-score on TMCD splits of each category of generalization forms.
• Figure 5: The F1-score of continue pretrained T5 model under different data scales and different levels of distributional shift.