Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations

TL;DR

This work introduces STEP, an intermediate pre-training regime that strengthens Transformers' structural inductive biases by teaching them to perform syntactic transformations on dependency-tree representations specified by a transformation prefix. The model is pre-trained on a large corpus of synthetic transformations, then fine-tuned with tunable prefixes to activate useful transformations for downstream tasks, improving few-shot performance on syntax-dependent tasks and boosting structural generalization in semantic parsing. Analyses reveal interpretable transformation look-up heads that track which transformation applies to which token and show that these heads are reused during downstream fine-tuning, validating the mechanism. The approach yields strong gains on chunking, passivization, adjective emphasis, and SLOG/ATIS generalization, suggesting a practical path to inject robust structural biases into seq2seq models without requiring explicit parse inputs at inference time.

Abstract

Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can still benefit from enhanced structural inductive biases for seq2seq tasks, especially those involving syntactic transformations, such as converting active to passive voice or semantic parsing. In this paper, we propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform synthetically generated syntactic transformations of dependency trees given a description of the transformation. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking, and also improves structural generalization for semantic parsing. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token, and that the model can leverage these attention heads on downstream tasks.

Figures (8)

• Figure 1: Left: Intermediate pre-training of a Transformer to perform syntactic transformations specified in the prefix; the syntax tree forms the basis of the transformation but is not given to the model. Right: fine-tuning the Transformer and the prefix on a downstream task. Tunable parameters are represented in orange.
• Figure 2: Our procedure of applying a syntactic transformation specified as edgewise transformations (grey box): (1) recursively unfolding a dependency tree into a binary tree where dependency labels serve as labels of internal nodes, (2) annotation dependency relations with edgewise transformations, (3), recursive evaluation of the edgewise transformations with partial results shown.
• Figure 3: Unfolding a head $h$ and its children.
• Figure 4: Change in accuracy of predicting the output of edgewise transformations when masking different attention heads. We show accuracy relative to no masking.
• Figure 5: Effect of masking look-up heads of models that have been fine-tuned on downstream syntactic tasks. For each task, we show the distribution for the 10 fine-tuned models from \ref{['sec:eval-syn-transformation']}.