The Emergence of Chunking Structures with Hierarchical RNN

TL;DR

The paper tackles unsupervised chunking by introducing a Hierarchical RNN (HRNN) that explicitly models word-to-chunk and chunk-to-sentence composition via a trainable gating mechanism. A two-stage training regime uses an unsupervised Compound PCFG to induce chunk labels for pretraining, followed by finetuning on downstream text-generation tasks to refine chunk representations. Empirical results show significant gains over baselines in unsupervised chunking and improved transfer to downstream tasks, with summarization-driven pretraining delivering the largest gains. A key finding is the transient emergence of linguistic structure during finetuning, suggesting that chunk-like representations serve as a useful inductive bias early in training but may be discarded as the model optimizes downstream performance. The work advances unsupervised syntactic structure discovery and opens avenues for multilingual extension and deeper linguistic analysis.

Abstract

In Natural Language Processing (NLP), predicting linguistic structures, such as parsing and chunking, has mostly relied on manual annotations of syntactic structures. This paper introduces an unsupervised approach to chunking, a syntactic task that involves grouping words in a non-hierarchical manner. We present a Hierarchical Recurrent Neural Network (HRNN) designed to model word-to-chunk and chunk-to-sentence compositions. Our approach involves a two-stage training process: pretraining with an unsupervised parser and finetuning on downstream NLP tasks. Experiments on multiple datasets reveal a notable improvement of unsupervised chunking performance in both pretraining and finetuning stages. Interestingly, we observe that the emergence of the chunking structure is transient during the neural model's downstream-task training. This study contributes to the advancement of unsupervised syntactic structure discovery and opens avenues for further research in linguistic theory.

Figures (9)

• Figure 1: An overview of our chunking induction method. (a) Pretraining HRNN using the chunk labels induced from the Compound PCFG parser. (b) HRNN is finetuned with text generation, specifically a summarization task in this example. A weight matrix is created from the switching gate's values and then inserted into the Transformer's encoder-decoder attention modules.
• Figure 2: Analysis of the size of the pretraining dataset.
• Figure 3: Ablation study of our HRNN finetuning method. We plot the learning curves in terms of phrasal F1 (top) and tag accuracy (bottom).
• Figure 4: This analysis examines the effect of the chunking strength hyperparameter $\kappa$ on the actual chunking ratio. The dashed line represents the groundtruth chunking ratio in the dataset. The purple lines indicate that the finetuned chunking performance outperforms the pretrained model. Additionally, the color depth and width of these lines indicate the ranking of the chunking performance achieved, with deeper and wider lines indicating better performance.
• Figure 5: Learning curves of finetuning HRNN with text generation tasks. Phrasal F1 and accuracy reflect the chunking performance, while the Rouge1 score measures the text generation performance. The gray dashed lines mark the steps when HRNN achieves the highest chunking performance.

Theorems & Definitions (2)

• theorem 1
• proof : Existence