Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CS-Sum: A Benchmark for Code-Switching Dialogue Summarization and the Limits of Large Language Models

Sathya Krishnan Suresh, Tanmay Surana, Lim Zhi Hao, Eng Siong Chng

TL;DR

CS-Sum introduces the first benchmark for code-switching dialogue summarization across EN-ZH, EN-TA, and EN-MS, using 900–1300 CS dialogue–summary pairs per pair created by translating English DialogSum and SAMSum. The study evaluates ten LLMs under few-shot, translate-summarize, and LoRA/QLoRA fine-tuning with Gemini-2 synthetic data, and finds that high automated metrics often mask subtle, meaning-altering errors, characterized by three persistent error types (CSL, MST, SMA). Results show language-pair variability and that synthetic fine-tuning can degrade CS understanding when distributions shift, underscoring the need for CS-aware pretraining and evaluation. The authors release CS-Sum and the synthetic dataset to catalyze further research, highlighting the practical impact for multilingual NLP and safer, more faithful CS processing in LLMs.

Abstract

Code-switching (CS) poses a significant challenge for Large Language Models (LLMs), yet its comprehensibility remains underexplored in LLMs. We introduce CS-Sum, to evaluate the comprehensibility of CS by the LLMs through CS dialogue to English summarization. CS-Sum is the first benchmark for CS dialogue summarization across Mandarin-English (EN-ZH), Tamil-English (EN-TA), and Malay-English (EN-MS), with 900-1300 human-annotated dialogues per language pair. Evaluating ten LLMs, including open and closed-source models, we analyze performance across few-shot, translate-summarize, and fine-tuning (LoRA, QLoRA on synthetic data) approaches. Our findings show that though the scores on automated metrics are high, LLMs make subtle mistakes that alter the complete meaning of the dialogue. To this end, we introduce 3 most common type of errors that LLMs make when handling CS input. Error rates vary across CS pairs and LLMs, with some LLMs showing more frequent errors on certain language pairs, underscoring the need for specialized training on code-switched data.

CS-Sum: A Benchmark for Code-Switching Dialogue Summarization and the Limits of Large Language Models

TL;DR

CS-Sum introduces the first benchmark for code-switching dialogue summarization across EN-ZH, EN-TA, and EN-MS, using 900–1300 CS dialogue–summary pairs per pair created by translating English DialogSum and SAMSum. The study evaluates ten LLMs under few-shot, translate-summarize, and LoRA/QLoRA fine-tuning with Gemini-2 synthetic data, and finds that high automated metrics often mask subtle, meaning-altering errors, characterized by three persistent error types (CSL, MST, SMA). Results show language-pair variability and that synthetic fine-tuning can degrade CS understanding when distributions shift, underscoring the need for CS-aware pretraining and evaluation. The authors release CS-Sum and the synthetic dataset to catalyze further research, highlighting the practical impact for multilingual NLP and safer, more faithful CS processing in LLMs.

Abstract

Code-switching (CS) poses a significant challenge for Large Language Models (LLMs), yet its comprehensibility remains underexplored in LLMs. We introduce CS-Sum, to evaluate the comprehensibility of CS by the LLMs through CS dialogue to English summarization. CS-Sum is the first benchmark for CS dialogue summarization across Mandarin-English (EN-ZH), Tamil-English (EN-TA), and Malay-English (EN-MS), with 900-1300 human-annotated dialogues per language pair. Evaluating ten LLMs, including open and closed-source models, we analyze performance across few-shot, translate-summarize, and fine-tuning (LoRA, QLoRA on synthetic data) approaches. Our findings show that though the scores on automated metrics are high, LLMs make subtle mistakes that alter the complete meaning of the dialogue. To this end, we introduce 3 most common type of errors that LLMs make when handling CS input. Error rates vary across CS pairs and LLMs, with some LLMs showing more frequent errors on certain language pairs, underscoring the need for specialized training on code-switched data.
Paper Structure (27 sections, 1 equation, 6 figures, 11 tables)

This paper contains 27 sections, 1 equation, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. CS-Sum Benchmark
  4. Dataset Construction
  5. Corpus Overview and Analysis
  6. Quantifying Code-Switching
  7. Gemini-Generated Training Data
  8. LLM performance on CS-Sum
  9. Result Analysis
  10. Translate-Summarize % Improvement
  11. LoRA and QLoRA Results
  12. Error Analysis
  13. CSL persists as the dominant failure mode in Few-Shot evaluation.
  14. Scaling within a model family does not reliably reduce errors.
  15. EN-TA exhibits consistently higher error rates across models.
  16. ...and 12 more sections

Figures (6)

  • Figure 1: An instance from the benchmark
  • Figure 2: Wrong summary with high BERTScore $0.903$
  • Figure 3: CSL error example
  • Figure 4: Distribution b/w filtered CS-Sum-Syn and CS-Sum for EN-ZH
  • Figure 5: Filtered data % improvement over CS-Sum-Syn for EN-ZH
  • ...and 1 more figures