SPOR: A Comprehensive and Practical Evaluation Method for Compositional Generalization in Data-to-Text Generation

TL;DR

SPOR introduces a practical, four-aspect framework for assessing compositional generalization in data-to-text generation: Systematicity, Productivity, Order Invariance, and Rule Learnability. By constructing automatic, annotation-free datasets on two real-world benchmarks (WebNLG and E2E) and evaluating a spectrum of models from T5/BART/GPT-2 to large LLMs with LoRA, the paper reveals persistent deficiencies across manifestations, even for state-of-the-art LLMs. The approach combines targeted dataset constructions with precise evaluation metrics (notably PARENT, Kendall-based order measures, and copy-rule tests) to yield a holistic view of model capabilities and failure modes. The findings underscore the need for comprehensive evaluation in practical data-to-text systems and provide a reusable framework to diagnose and guide future improvements in compositional generalization.

Abstract

Compositional generalization is an important ability of language models and has many different manifestations. For data-to-text generation, previous research on this ability is limited to a single manifestation called Systematicity and lacks consideration of large language models (LLMs), which cannot fully cover practical application scenarios. In this work, we propose SPOR, a comprehensive and practical evaluation method for compositional generalization in data-to-text generation. SPOR includes four aspects of manifestations (Systematicity, Productivity, Order invariance, and Rule learnability) and allows high-quality evaluation without additional manual annotations based on existing datasets. We demonstrate SPOR on two different datasets and evaluate some existing language models including LLMs. We find that the models are deficient in various aspects of the evaluation and need further improvement. Our work shows the necessity for comprehensive research on different manifestations of compositional generalization in data-to-text generation and provides a framework for evaluation.

Figures (5)

• Figure 1: Examples of data-text pairs in WebNLG (above) and E2E (below).
• Figure 2: An example of datasets for the systematicity evaluation. Each pair of brackets denotes a sample and each letter (A G) denotes a data unit.
• Figure 3: An example of datasets with threshold $N=4$ for the productivity evaluation. Each number represents a sample with a corresponding number of data units.
• Figure 4: An illustration of the order invariance evaluation. Each letter (A D) denotes a data unit. For a certain property, the evaluation checks whether the output has that property. $\checkmark$ means yes and $\times$ means no.
• Figure 5: An example of dataset construction for the rule learnability evaluation.