Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale

TL;DR

GPST introduces an unsupervised syntactic language model that jointly learns explicit syntax and left-to-right generation at scale by combining a composition model with a generative Transformer LM. Training uses a hard-EM style loop with an E-step that induces parse trees via a pruned inside-outside encoder and an M-step that optimizes a joint objective using a representation surrogate to enable parallel processing; a gradient-stop mechanism mitigates left-branching biases. Pre-trained on OpenWebText (~9B tokens) and evaluated across language understanding, generation, grammar induction, and syntactic generalization, GPST shows substantial training speedups and competitive or superior performance compared to GPT-2 of similar size, with notable gains in grammar induction and RTE-style tasks. The work demonstrates that unsupervised syntactic LMs can surpass gold-tree baselines in certain settings and offers richer constituent-level representations that could enhance interpretability and multi-modality in future large-scale systems.

Abstract

A syntactic language model (SLM) incrementally generates a sentence with its syntactic tree in a left-to-right manner. We present Generative Pretrained Structured Transformers (GPST), an unsupervised SLM at scale capable of being pre-trained from scratch on raw texts with high parallelism. GPST circumvents the limitations of previous SLMs such as relying on gold trees and sequential training. It consists of two components, a usual SLM supervised by a uni-directional language modeling loss, and an additional composition model, which induces syntactic parse trees and computes constituent representations, supervised by a bi-directional language modeling loss. We propose a representation surrogate to enable joint parallel training of the two models in a hard-EM fashion. We pre-train GPST on OpenWebText, a corpus with $9$ billion tokens, and demonstrate the superiority of GPST over GPT-2 with a comparable size in numerous tasks covering both language understanding and language generation. Meanwhile, GPST also significantly outperforms existing unsupervised SLMs on left-to-right grammar induction, while holding a substantial acceleration on training.

Figures (8)

• Figure 1: An illustration of the generative process of GPST. $\mathbf{x}_{i:j}$ denotes the sub tree representation spanning from $i$ to $j$. As we use Transformers as the backbone, all previous hidden states are leveraged. At step $t$, the length of historical hidden states is $t$ for the type layers and $w_t$ for the token layers as illustrated with dotted lines for step 3.
• Figure 2: Illustration of the training process. (a) In the E-step, we induce a parse tree and compute constituent representations. (b)(i) Data dependencies within inputs of the generative model. (b)(ii) Illustration of representation surrogates. $\mathbf{x}_{i:j}$ denotes the original input representation spanning over $(i,j)$ composed from left to right.
• Figure 3: An illustration of beam search decoding of size 2. For simplicity, we use ")" to denote $\textsc{Comp}\xspace$ and upper case characters to denote words generated by $\textsc{Gen}\xspace(x)$. Boxes filled in gray are hypotheses with the last action being $\textsc{Comp}\xspace$. Grayed-out boxes are pruned out during beam search.
• Figure 4: Fast encoding follows the order given by a top-down parser, with the merging order $\mathcal{M}$ being the reverse order of the split point sequence $\mathcal{A}$. $x_i$ denotes the $i_{th}$ token in a sentence of length 6. Numbers in $\mathcal{A}$ and $\mathcal{M}$ denote the indices of the split/merge point between tokens. $v_j$ denotes the split score of $j_{th}$ split point, predicted by the top-down parser.
• Figure 5: The initial step of encoding in $O(\log n)$ steps. The numbers in blue correspond to the indices of the split points introduced in Figure \ref{['fig:merge_order']}.