Can we obtain significant success in RST discourse parsing by using Large Language Models?

TL;DR

This paper employs Llama 2 and fine-tune it with QLoRA, which has fewer parameters that can be tuned, and demonstrates generalizability when evaluated on RST-DT, showing that, in spite of being trained with the GUM corpus, it obtained similar performances to those of existing parsers trained with RST-DT.

Abstract

Recently, decoder-only pre-trained large language models (LLMs), with several tens of billion parameters, have significantly impacted a wide range of natural language processing (NLP) tasks. While encoder-only or encoder-decoder pre-trained language models have already proved to be effective in discourse parsing, the extent to which LLMs can perform this task remains an open research question. Therefore, this paper explores how beneficial such LLMs are for Rhetorical Structure Theory (RST) discourse parsing. Here, the parsing process for both fundamental top-down and bottom-up strategies is converted into prompts, which LLMs can work with. We employ Llama 2 and fine-tune it with QLoRA, which has fewer parameters that can be tuned. Experimental results on three benchmark datasets, RST-DT, Instr-DT, and the GUM corpus, demonstrate that Llama 2 with 70 billion parameters in the bottom-up strategy obtained state-of-the-art (SOTA) results with significant differences. Furthermore, our parsers demonstrated generalizability when evaluated on RST-DT, showing that, in spite of being trained with the GUM corpus, it obtained similar performances to those of existing parsers trained with RST-DT.

Figures (6)

• Figure 1: Example of RST tree from WSJ_1100 in RST-DT carlson-etal-2002-rstdt, consisting of six EDUs ($e$): $e_1$: [Westinghouse Electric Corp. said], $e_2$: [it will buy Shaw-Walker Co.], $e_3$: [Terms weren't disclosed.], $e_4$: [Shaw-Walker,], $e_5$: [based in Muskegon, Mich.,], $e_6$: [makes metal files and desks, and seating and office systems furniture.]. N and S represent the Nucleus and Satellite, respectively.
• Figure 2: Bottom-up parsing by shift-reduce operations
• Figure 3: Top-down parsing by span split
• Figure 4: Example of the bottom-up parsing process using an LLM with prompts. In the example, Stack2 stores a text span, an already constructed subtree, consisting of two EDUs: $e_1$: [Westinghouse Electric Corp. said], $e_2$: [it will buy Shaw-Walker Co.]. Stack1 stores a text span of single EDU, $e_3$: [Terms weren't disclosed.]. Queue1 stores an EDU, $e_4$: [Shaw-Walker,]. After this step, the parsing process goes on to the next steps while updating Stack* and Queue.
• Figure 5: Example of the top-down parsing process using an LLM with prompts. In the example, a text span consisting of three EDUs, $e_4$: [Shaw-Walker,], $e_5$: [based in Muskegon, Mich.,], and $e_6$: [makes metal files and desks, and seating and office systems furniture.], are divided and labeled by an LLM's decision. The process is recursively applied until divided spans are identical to single EDUs.