Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis

TL;DR

The paper tackles how pretraining objectives shape syntactic knowledge in LSTM representations by directly comparing language modeling, translation, skip-thought, and autoencoding under identical data and architectural conditions. Using POS tagging and CCG supertagging as probes, the authors show bidirectional language models yield the strongest syntactic signals, often outperforming translation and skip-thought even with less data. They also reveal that randomly initialized LSTMs can match trained models when auxiliary data is plentiful, highlighting memorization as a potential confound and demonstrating that true syntactic learning resides in the representations beyond simple word identity. Layer-wise analysis indicates lower layers capture local cues while upper layers encode broader context, with depth enabling more abstract syntactic information. These findings inform pretraining choices for transfer learning tasks requiring rich syntactic information and emphasize the nuanced role of data volume and architecture in shaping representations.

Abstract

Recent work using auxiliary prediction task classifiers to investigate the properties of LSTM representations has begun to shed light on why pretrained representations, like ELMo (Peters et al., 2018) and CoVe (McCann et al., 2017), are so beneficial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives---language modeling, translation, skip-thought, and autoencoding---on their ability to induce syntactic and part-of-speech information. We make a fair comparison between the tasks by holding constant the quantity and genre of the training data, as well as the LSTM architecture. We find that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data. These results suggest that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information. We also find that the representations from randomly-initialized, frozen LSTMs perform strikingly well on our syntactic auxiliary tasks, but this effect disappears when the amount of training data for the auxiliary tasks is reduced.

Figures (6)

• Figure 1: An annotated PTB example sentence.
• Figure 2: POS and CCG tagging accuracies for different amounts of LSTM encoder and classifier training data. We show results for the best performing layer of each model. Note, BiLMs are displayed with the attention models and forward LMs are displayed with the models without attention.
• Figure 3: POS and CCG tagging accuracies for different amounts of classifier training data in terms of percentage points over the word-conditional most frequent class (WC-MFC) baseline. We show results for the best performing layer and model for each task.
• Figure 4: POS and CCG tagging accuracies in terms of percentage points over the word-conditional most frequent class baseline. We display results for the best performing models for each task.
• Figure 5: Performance of classifiers trained to predict the identity of the word a fixed number of timesteps away. Note, the forward LM has asymmetrical access to this information in its input.