Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StringLLM: Understanding the String Processing Capability of Large Language Models

Xilong Wang, Hao Fu, Jindong Wang, Neil Zhenqiang Gong

TL;DR

StringLLM introduces a comprehensive benchmark construction framework to evaluate LLMs on string processing via StringBench, spanning multilingual, hash, and random strings. The study finds that current LLMs exhibit limited character-level understanding due to tokenization and embedding limitations, performing far below human capabilities. Prompt engineering, especially Program of Thought (PoT), yields notable performance gains, yet gaps remain. The authors demonstrate that supervised fine-tuning with PoT-oriented data and LoRA can boost string-processing accuracy by at least $38.80\%$ while largely preserving foundational and coding abilities, suggesting dataset-guided fine-tuning as a practical path to improve string processing in LLMs.

Abstract

String processing, which mainly involves the analysis and manipulation of strings, is a fundamental component of modern computing. Despite the significant advancements of large language models (LLMs) in various natural language processing (NLP) tasks, their capability in string processing remains underexplored and underdeveloped. To bridge this gap, we present a comprehensive study of LLMs' string processing capability. In particular, we first propose StringLLM, a method to construct datasets for benchmarking string processing capability of LLMs. We use StringLLM to build a series of datasets, referred to as StringBench. It encompasses a wide range of string processing tasks, allowing us to systematically evaluate LLMs' performance in this area. Our evaluations indicate that LLMs struggle with accurately processing strings compared to humans. To uncover the underlying reasons for this limitation, we conduct an in-depth analysis and subsequently propose an effective approach that significantly enhances LLMs' string processing capability via fine-tuning. This work provides a foundation for future research to understand LLMs' string processing capability. Our code and data are available at https://github.com/wxl-lxw/StringLLM.

StringLLM: Understanding the String Processing Capability of Large Language Models

TL;DR

StringLLM introduces a comprehensive benchmark construction framework to evaluate LLMs on string processing via StringBench, spanning multilingual, hash, and random strings. The study finds that current LLMs exhibit limited character-level understanding due to tokenization and embedding limitations, performing far below human capabilities. Prompt engineering, especially Program of Thought (PoT), yields notable performance gains, yet gaps remain. The authors demonstrate that supervised fine-tuning with PoT-oriented data and LoRA can boost string-processing accuracy by at least while largely preserving foundational and coding abilities, suggesting dataset-guided fine-tuning as a practical path to improve string processing in LLMs.

Abstract

String processing, which mainly involves the analysis and manipulation of strings, is a fundamental component of modern computing. Despite the significant advancements of large language models (LLMs) in various natural language processing (NLP) tasks, their capability in string processing remains underexplored and underdeveloped. To bridge this gap, we present a comprehensive study of LLMs' string processing capability. In particular, we first propose StringLLM, a method to construct datasets for benchmarking string processing capability of LLMs. We use StringLLM to build a series of datasets, referred to as StringBench. It encompasses a wide range of string processing tasks, allowing us to systematically evaluate LLMs' performance in this area. Our evaluations indicate that LLMs struggle with accurately processing strings compared to humans. To uncover the underlying reasons for this limitation, we conduct an in-depth analysis and subsequently propose an effective approach that significantly enhances LLMs' string processing capability via fine-tuning. This work provides a foundation for future research to understand LLMs' string processing capability. Our code and data are available at https://github.com/wxl-lxw/StringLLM.
Paper Structure (18 sections, 3 figures, 13 tables)

This paper contains 18 sections, 3 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Constructing Datasets via StringLLM
  4. Overview
  5. Constructing String Processing Tasks
  6. Generating Question-answer Pairs for Each Task
  7. Post-processing
  8. Benchmarking LLMs' String Processing Capabilities
  9. Experimental Setup
  10. Experimental Results
  11. Understanding Why LLMs Struggle with String Processing
  12. Tokenization Cannot Split Strings into Characters
  13. Tokenization Makes LLMs Lack Character-Level Understanding
  14. LLMs' Understanding of Strings is Limited
  15. Improving String Processing Capability via Fine-tuning
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Overview of how StringLLM builds the benchmark datasets. Code represents Python code expression for each string processing task, Template denotes the generated question template for each task, Variables are the placeholders within both Code and Template, and Answer refers to the groundtruth for each sample.
  • Figure 2: Examples of GPT-4o solving a string processing task using different prompt engineering techniques.
  • Figure 3: Examples of GPT-4o solving a string processing task using different prompt engineering techniques.