PingPong: A Natural Benchmark for Multi-Turn Code-Switching Dialogues

TL;DR

PingPong presents a natural, open benchmark for code-switching in multi-party dialogues across five language combinations, including trilingual setups. The dataset is crowdsourced with 2–4 participants and supports Question Answering, Dialogue Summarization, and Topic Classification, capturing long-range references and varied speaker dynamics. Across multiple models, performance remains limited on code-switched inputs, illustrating a gap between current NLP systems and real-world multilingual discourse. The work demonstrates that human-authored dialogues are more natural than machine-generated ones and shows how reasoning traces can improve tasks, establishing PingPong as a valuable platform for evaluating and advancing robust multilingual dialogue processing.

Abstract

Code-switching is a widespread practice among the world's multilingual majority, yet few benchmarks accurately reflect its complexity in everyday communication. We present PingPong, a benchmark for natural multi-party code-switching dialogues covering five language-combination variations, some of which are trilingual. Our dataset consists of human-authored conversations among 2 to 4 participants covering authentic, multi-threaded structures where replies frequently reference much earlier points in the dialogue. We demonstrate that our data is significantly more natural and structurally diverse than machine-generated alternatives, offering greater variation in message length, speaker dominance, and reply distance. Based on these dialogues, we define three downstream tasks: Question Answering, Dialogue Summarization, and Topic Classification. Evaluations of several state-of-the-art language models on PingPong reveal that performance remains limited on code-switched inputs, underscoring the urgent need for more robust NLP systems capable of addressing the intricacies of real-world multilingual discourse.

Figures (20)

• Figure 1: Illustration of dialogue samples from $\textcolor{black}{PingPong}$. The bottom panel highlights how each turn corresponds to previous turns, as labeled in our dataset.
• Figure 2: Overview of $\textcolor{black}{PingPong}$ dataset construction. We recruit annotators who speak five language combinations and group them into small teams of 2 to 4 participants. Each group engages in multi-party dialogue creation, after which downstream annotations are performed. The final output is the curated $\textcolor{black}{PingPong}$ dataset, focusing on natural code-switched dialogues for multiple tasks.
• Figure 3: Comparison between human-written and machine-generated conversation texts. Text with a color denotes the original conversation, while text with a color represents the English translation. Segments highlighted in indicate Indonesian words, whereas segments highlighted in indicate English words.
• Figure 4: Comparison of average model performance (Acc. %) across languages for Answerable vs. Unanswerable cases, from the perspectives of N-shot prompting (Left) and reasoning performance (Right).
• Figure 5: Reasoning vs No-Reasoning performance in Dialogue Summarization (blue), Topic Classification (green), and QA (red).