Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Structural Generalization in Neural Machine Translation

Ryoma Kumon, Daiki Matsuoka, Hitomi Yanaka

TL;DR

This work addresses how neural models generalize compositionally in MT by introducing SGET, a controlled English–Japanese benchmark that tests both lexical and structural generalization. SGET uses PCFG-generated English sentences paired with RBMT-derived Japanese translations, enabling strict control over lexical items and syntactic gaps across seven generalization patterns. Experiments with LSTM, Transformer, and Llama 2 reveal near-perfect in-distribution performance but substantial gaps on the generalization set, with structural generalization proving substantially harder than lexical generalization, though pretrained models show stronger robustness. The findings underscore the importance of evaluating across tasks beyond semantic parsing and suggest practical strategies, such as concatenating training data to alleviate length-generalization pressures, while highlighting limitations and directions for future cross-language and cross-task analyses.

Abstract

Compositional generalization refers to the ability to generalize to novel combinations of previously observed words and syntactic structures. Since it is regarded as a desired property of neural models, recent work has assessed compositional generalization in machine translation as well as semantic parsing. However, previous evaluations with machine translation have focused mostly on lexical generalization (i.e., generalization to unseen combinations of known words). Thus, it remains unclear to what extent models can translate sentences that require structural generalization (i.e., generalization to different sorts of syntactic structures). To address this question, we construct SGET, a machine translation dataset covering various types of compositional generalization with control of words and sentence structures. We evaluate neural machine translation models on SGET and show that they struggle more in structural generalization than in lexical generalization. We also find different performance trends in semantic parsing and machine translation, which indicates the importance of evaluations across various tasks.

Evaluating Structural Generalization in Neural Machine Translation

TL;DR

This work addresses how neural models generalize compositionally in MT by introducing SGET, a controlled English–Japanese benchmark that tests both lexical and structural generalization. SGET uses PCFG-generated English sentences paired with RBMT-derived Japanese translations, enabling strict control over lexical items and syntactic gaps across seven generalization patterns. Experiments with LSTM, Transformer, and Llama 2 reveal near-perfect in-distribution performance but substantial gaps on the generalization set, with structural generalization proving substantially harder than lexical generalization, though pretrained models show stronger robustness. The findings underscore the importance of evaluating across tasks beyond semantic parsing and suggest practical strategies, such as concatenating training data to alleviate length-generalization pressures, while highlighting limitations and directions for future cross-language and cross-task analyses.

Abstract

Compositional generalization refers to the ability to generalize to novel combinations of previously observed words and syntactic structures. Since it is regarded as a desired property of neural models, recent work has assessed compositional generalization in machine translation as well as semantic parsing. However, previous evaluations with machine translation have focused mostly on lexical generalization (i.e., generalization to unseen combinations of known words). Thus, it remains unclear to what extent models can translate sentences that require structural generalization (i.e., generalization to different sorts of syntactic structures). To address this question, we construct SGET, a machine translation dataset covering various types of compositional generalization with control of words and sentence structures. We evaluate neural machine translation models on SGET and show that they struggle more in structural generalization than in lexical generalization. We also find different performance trends in semantic parsing and machine translation, which indicates the importance of evaluations across various tasks.
Paper Structure (51 sections, 3 figures, 11 tables)

This paper contains 51 sections, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Semantic Parsing
  4. Machine Translation
  5. Method
  6. Overview
  7. Data Construction
  8. Generalization Patterns
  9. Tense Alternation
  10. Phrase Recombination
  11. Gap Position Recombination
  12. Recursion Depth Alternation
  13. Sentence Generation with PCFG
  14. Creating Translation Data
  15. Sentence Filtering
  16. ...and 36 more sections

Figures (3)

  • Figure 1: English-Japanese translation
  • Figure 2: Syntactic trees
  • Figure 4: Data construction method.