Ensemble Distillation for Unsupervised Constituency Parsing

TL;DR

This work tackles unsupervised constituency parsing by exploiting diverse expertise across multiple existing parsers. It introduces tree averaging, a CYK-like dynamic programming method, to compute an AvgTree that most closely aligns with teachers' outputs, and then distills the ensemble into a fast student model to enable efficient inference. The approach achieves state-of-the-art results on Penn Treebank and demonstrates strong robustness under domain shift to SUSANNE, while substantially accelerating inference via the URNNG student. Overall, the ensemble-then-distill framework effectively bridges the gap between supervised and unsupervised parsing and offers practical benefits for low-resource languages and real-world applications.

Abstract

We investigate the unsupervised constituency parsing task, which organizes words and phrases of a sentence into a hierarchical structure without using linguistically annotated data. We observe that existing unsupervised parsers capture differing aspects of parsing structures, which can be leveraged to enhance unsupervised parsing performance. To this end, we propose a notion of "tree averaging," based on which we further propose a novel ensemble method for unsupervised parsing. To improve inference efficiency, we further distill the ensemble knowledge into a student model; such an ensemble-then-distill process is an effective approach to mitigate the over-smoothing problem existing in common multi-teacher distilling methods. Experiments show that our method surpasses all previous approaches, consistently demonstrating its effectiveness and robustness across various runs, with different ensemble components, and under domain-shift conditions.

Figures (8)

• Figure 1: Effect of denoising vs. utilizing different expertise. Results are the $F_1$ scores on the PTB test set. The italic blue annotation is an interpretation of the plot.
• Figure 2: Ensemble performance with different numbers of teachers. The lines are best-performing, average, and worst-performing combinations. These results are averaged over five runs available in the experiments conducted for Table \ref{['tab:main_results']}. The gray shades are the best and worst runs.
• Figure 3: Step-by-step illustration of our CYK algorithm, showing the dynamic changes in the $H$ along with the construction of the corresponding optimal binary constituency tree.
• Figure 4: Performance by sentence lengths. $F_1$ scores are averaged over five different runs.
• Figure 5: Performance by constituency labels on the PTB test set. Results are measured by recall, because the predicted parse trees are unlabeled; thus, precision and $F_1$ scores cannot be computed drozdov-etal-2019-unsupervised-latent. Bars and gray intervals are the mean and standard deviation, respectively, over five runs.