Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models

Zihong Zhang, Liqi He, Zuchao Li, Lefei Zhang, Hai Zhao, Bo Du

TL;DR

This work reframes word segmentation as a probe of semantic understanding by Large Language Models (LLMs) and introduces the LLM-Word Segmentation (LLM-WS) framework for unsupervised segmentation across languages lacking explicit word boundaries. It then presents LLACA, a dynamic Aho-Corasick automaton informed by LLM derived vocabularies and a variable $n$-gram model with Viterbi decoding, augmented by PMI filtering to minimize hallucinations. Empirical results show that larger LLMs improve segmentation performance and that LLACA delivers faster, more robust unsupervised segmentation, often surpassing direct LLM outputs while reducing computational cost. The study covers multilingual data (Chinese, Japanese, Korean, Thai) and includes thorough OOV handling analysis, highlighting the practical benefits for domain adaptation and downstream NLP tasks. Overall, the paper demonstrates a shift from segment-first to comprehend-first paradigms in NLP and provides a scalable, domain adaptable framework for unsupervised word segmentation with broad implications for language understanding research.

Abstract

Word segmentation stands as a cornerstone of Natural Language Processing (NLP). Based on the concept of "comprehend first, segment later", we propose a new framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs) and evaluate the semantic understanding capabilities of LLMs based on word segmentation. We employ current mainstream LLMs to perform word segmentation across multiple languages to assess LLMs' "comprehension". Our findings reveal that LLMs are capable of following simple prompts to segment raw text into words. There is a trend suggesting that models with more parameters tend to perform better on multiple languages. Additionally, we introduce a novel unsupervised method, termed LLACA ($\textbf{L}$arge $\textbf{L}$anguage Model-Inspired $\textbf{A}$ho-$\textbf{C}$orasick $\textbf{A}$utomaton). Leveraging the advanced pattern recognition capabilities of Aho-Corasick automata, LLACA innovatively combines these with the deep insights of well-pretrained LLMs. This approach not only enables the construction of a dynamic $n$-gram model that adjusts based on contextual information but also integrates the nuanced understanding of LLMs, offering significant improvements over traditional methods. Our source code is available at https://github.com/hkr04/LLACA

Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models

TL;DR

This work reframes word segmentation as a probe of semantic understanding by Large Language Models (LLMs) and introduces the LLM-Word Segmentation (LLM-WS) framework for unsupervised segmentation across languages lacking explicit word boundaries. It then presents LLACA, a dynamic Aho-Corasick automaton informed by LLM derived vocabularies and a variable -gram model with Viterbi decoding, augmented by PMI filtering to minimize hallucinations. Empirical results show that larger LLMs improve segmentation performance and that LLACA delivers faster, more robust unsupervised segmentation, often surpassing direct LLM outputs while reducing computational cost. The study covers multilingual data (Chinese, Japanese, Korean, Thai) and includes thorough OOV handling analysis, highlighting the practical benefits for domain adaptation and downstream NLP tasks. Overall, the paper demonstrates a shift from segment-first to comprehend-first paradigms in NLP and provides a scalable, domain adaptable framework for unsupervised word segmentation with broad implications for language understanding research.

Abstract

Word segmentation stands as a cornerstone of Natural Language Processing (NLP). Based on the concept of "comprehend first, segment later", we propose a new framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs) and evaluate the semantic understanding capabilities of LLMs based on word segmentation. We employ current mainstream LLMs to perform word segmentation across multiple languages to assess LLMs' "comprehension". Our findings reveal that LLMs are capable of following simple prompts to segment raw text into words. There is a trend suggesting that models with more parameters tend to perform better on multiple languages. Additionally, we introduce a novel unsupervised method, termed LLACA (arge anguage Model-Inspired ho-orasick utomaton). Leveraging the advanced pattern recognition capabilities of Aho-Corasick automata, LLACA innovatively combines these with the deep insights of well-pretrained LLMs. This approach not only enables the construction of a dynamic -gram model that adjusts based on contextual information but also integrates the nuanced understanding of LLMs, offering significant improvements over traditional methods. Our source code is available at https://github.com/hkr04/LLACA

Paper Structure

This paper contains 33 sections, 4 equations, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Approach
  4. Preliminary Experiments on LLM-WS
  5. LLACA: Large Language Model-Inspired Aho-Corasick Automaton
  6. Time Complexity
  7. Experiments
  8. Experimental Setup
  9. LLM-WS: Evaluating the Word Segmentation Capabilities of LLMs
  10. From LLM-WS to LLACA
  11. On OOV Handling
  12. Conclusion
  13. Experimental Details
  14. Algorithm
  15. Datasets
  16. ...and 18 more sections

Figures (7)

  • Figure 1: Examples of LLM-WS
  • Figure 2: Evaluations on datasets MSR (Simplified Chinese), KWDLC (Japanese) and BEST (Thai).
  • Figure 3: The illustration of an Aho-Corasick automaton with patterns "she", "he", and "her", with final states in double circles and failure links in red.
  • Figure 4: The construction of LLACA.
  • Figure 5: A Directed Acyclic Graph (DAG) for sequence segmentation, starting from "BOS" (Beginning of Sentence) and ending at "EOS" (End of Sentence). Edges denote recognized patterns. Paths through the graph represent potential segmentations, with the preferred path highlighted in frames.
  • ...and 2 more figures