Contrastive Learning on LLM Back Generation Treebank for Cross-domain Constituency Parsing

TL;DR

This work tackles cross-domain constituency parsing under data scarcity by introducing LLM back generation, which directly produces labeled cross-domain parse trees from masked inputs guided by domain keywords. It pairs this with a novel span-level contrastive pre-training objective to learn robust span representations from the generated treebank, enabling a single parser to handle five target domains. Empirical results on MCTB show state-of-the-art average F1, outperforming natural-treebank baselines, previous cross-domain approaches, and LLM-only pipelines, with GPT-4 delivering the strongest LLM-based results among those tested. The approach reduces annotation costs and data requirements while achieving fast convergence, though it is currently evaluated only on English, suggesting future work on multilingual extension and alternative LLMs.

Abstract

Cross-domain constituency parsing is still an unsolved challenge in computational linguistics since the available multi-domain constituency treebank is limited. We investigate automatic treebank generation by large language models (LLMs) in this paper. The performance of LLMs on constituency parsing is poor, therefore we propose a novel treebank generation method, LLM back generation, which is similar to the reverse process of constituency parsing. LLM back generation takes the incomplete cross-domain constituency tree with only domain keyword leaf nodes as input and fills the missing words to generate the cross-domain constituency treebank. Besides, we also introduce a span-level contrastive learning pre-training strategy to make full use of the LLM back generation treebank for cross-domain constituency parsing. We verify the effectiveness of our LLM back generation treebank coupled with contrastive learning pre-training on five target domains of MCTB. Experimental results show that our approach achieves state-of-the-art performance on average results compared with various baselines.

Figures (8)

• Figure 1: LLM constituency parsing usually predicts the wrong constituency tree structure $\widehat{Y}$ for the input sentence $X$. However, LLM back generation can generate valid constituency tree structure with the appropriate sentence $\widehat{X}$ based on the masked sentence $X^{*}$ and bare constituency tree structure $Y$.
• Figure 2: Overview of LLM Back Generation. We first extract the target domain constituency tree and domain keywords. Then we mask all sentential words except domain keywords from the constituency tree. Finally, the LLM back generates the whole syntax tree based on the masked tree and some demonstration examples.
• Figure 3: Positive instances (green node) and indexes for the constituent span (red node). LC, RC, PA and BR are left child, right child, parent, and brother, respectively.
• Figure 4: F1 score of three pre-training strategies on the LLM back generation treebanks with different mask rates of sentential word.
• Figure 5: Convergence curve of different pre-training strategies on the LLM back generation treebank.