New Evaluation Paradigm for Lexical Simplification

TL;DR

This work addresses the challenge of evaluating lexical simplification when LLMs can generate full-sentence simplifications in a single step. It introduces an all-in-one LS dataset built via human–machine collaboration on CWI data, including automated substitute generation and LLM-assisted human annotation. It then proposes CoLLS, a multi-LLM collaboration framework that simulates CWI, SG, and Validation with majority voting, achieving superior performance over single-prompt LLMs and traditional baselines. The results establish a new evaluation paradigm for LS and suggest strong potential for extending the approach across languages and contexts.

Abstract

Lexical Simplification (LS) methods use a three-step pipeline: complex word identification, substitute generation, and substitute ranking, each with separate evaluation datasets. We found large language models (LLMs) can simplify sentences directly with a single prompt, bypassing the traditional pipeline. However, existing LS datasets are not suitable for evaluating these LLM-generated simplified sentences, as they focus on providing substitutes for single complex words without identifying all complex words in a sentence. To address this gap, we propose a new annotation method for constructing an all-in-one LS dataset through human-machine collaboration. Automated methods generate a pool of potential substitutes, which human annotators then assess, suggesting additional alternatives as needed. Additionally, we explore LLM-based methods with single prompts, in-context learning, and chain-of-thought techniques. We introduce a multi-LLMs collaboration approach to simulate each step of the LS task. Experimental results demonstrate that LS based on multi-LLMs approaches significantly outperforms existing baselines.

Figures (12)

• Figure 1: Flowchart of current LS methods and LLM-based LS method. Current LS methods require not only three models for three steps: complex word identification, substitute generation, and substitute ranking, but also iterative simplification of each complex word. We found that the LLM-based method only needs a single prompting to complete the task.
• Figure 2: An overview of the methodology of the LS corpus we built. In the substitute generation phase, we combine three different methods via majority vote to generate pseudo substitutes. In the annotation stage, we first use LLMs to perform the first round of annotation using direct prompting and chain-of-thought prompting, and then feedback on the results to the annotator for final judgment and addition.
• Figure 3: Prompt template for one-step LS.
• Figure 4: Prompt template for LLM-based LS method (COT).
• Figure 5: The framework of CoLLS. Each step defines the roles and tasks of LLMs.