On Eliciting Syntax from Language Models via Hashing

TL;DR

This work investigates unsupervised constituency parsing by eliciting syntactic trees from pre-trained language models through a unified binary representation of lexicon and syntax. It elevates CKY to a first-order formulation and replaces supervised hashing with an unsupervised, span-level contrastive objective, including a novel balanced loss to align substructures across views. Empirical results on PTB and CTB show competitive parsing performance with relatively few bits and low training cost, demonstrating that high-quality syntax can be induced with minimal annotated data. The approach offers a scalable pathway to extract grammatical structure from large language models, with practical implications for syntax-aware NLP applications and analysis of implicit grammar in LMs.

Abstract

Unsupervised parsing, also known as grammar induction, aims to infer syntactic structure from raw text. Recently, binary representation has exhibited remarkable information-preserving capabilities at both lexicon and syntax levels. In this paper, we explore the possibility of leveraging this capability to deduce parsing trees from raw text, relying solely on the implicitly induced grammars within models. To achieve this, we upgrade the bit-level CKY from zero-order to first-order to encode the lexicon and syntax in a unified binary representation space, switch training from supervised to unsupervised under the contrastive hashing framework, and introduce a novel loss function to impose stronger yet balanced alignment signals. Our model shows competitive performance on various datasets, therefore, we claim that our method is effective and efficient enough to acquire high-quality parsing trees from pre-trained language models at a low cost.

Figures (16)

• Figure 1: The model architecture. The hash layer produces scores of all spans, and the following first-order bit-level CKY (§\ref{['sec:first_cky']}) returns marginal probabilities $\ve{\mu}$ and predicts the most probable trees $\hat{\ve{t}}$. Sentences are fed into the network twice, We select span marginal probabilities from one pass according to the predicted trees from the other pass, and perform contrastive hashing (§\ref{['sec:unsup_hash']}, §\ref{['sec:loss']}) on their corresponding score and code vectors. The purple cells represent the marginal probabilities, and the dark purple indicate the selected ones.
• Figure 2: Charts of the zero-order (above §\ref{['sec:zero_cky']}) and the first-order parsing (below §\ref{['sec:first_cky']}). At this time step, zero-order parsers separately determine the splitting positions on the left and right children and predict labels according to the top-most cell. In contrast, first-order parsers make these two decisions jointly by averaging all the cells that cross the left and right children to unify the representation of lexicon and syntax.
• Figure 3: Derivation of the sentence The quick brown fox jumps over the lazy dog. The left side is the ground-truth consistency tree, and the right side is our parsing result with binary labels represented in hexadecimal format.
• Figure 4: Derivation example.
• Figure 5: Derivation example.