Can Small Language Models Handle Context-Summarized Multi-Turn Customer-Service QA? A Synthetic Data-Driven Comparative Evaluation

TL;DR

The paper tackles multi-turn customer-service QA under privacy constraints by evaluating instruction-tuned small language models (SLMs) that leverage context-summarized histories. It introduces a synthetic data pipeline to convert single-turn QA into context-aware multi-turn interactions and applies QLoRA-based fine-tuning across nine SLMs, comparing them to three commercial LLMs using a mix of automatic metrics and qualitative judgments. A novel conversation stage-based analysis and LLM-as-a-judge framework provide fine-grained insights into model behavior across early, middle, and late interaction stages. Findings show several SLMs achieving near-LLM performance on lexical and semantic metrics and strong results in human-like qualities, while highlighting persistent gaps in dialogue continuity for some models. The work demonstrates a feasible, privacy-preserving path for deploying efficient CS QA systems in resource-constrained or on-premise environments and offers a reproducible dataset and evaluation framework for future research.

Abstract

Customer-service question answering (QA) systems increasingly rely on conversational language understanding. While Large Language Models (LLMs) achieve strong performance, their high computational cost and deployment constraints limit practical use in resource-constrained environments. Small Language Models (SLMs) provide a more efficient alternative, yet their effectiveness for multi-turn customer-service QA remains underexplored, particularly in scenarios requiring dialogue continuity and contextual understanding. This study investigates instruction-tuned SLMs for context-summarized multi-turn customer-service QA, using a history summarization strategy to preserve essential conversational state. We also introduce a conversation stage-based qualitative analysis to evaluate model behavior across different phases of customer-service interactions. Nine instruction-tuned low-parameterized SLMs are evaluated against three commercial LLMs using lexical and semantic similarity metrics alongside qualitative assessments, including human evaluation and LLM-as-a-judge methods. Results show notable variation across SLMs, with some models demonstrating near-LLM performance, while others struggle to maintain dialogue continuity and contextual alignment. These findings highlight both the potential and current limitations of low-parameterized language models for real-world customer-service QA systems.

Figures (8)

• Figure 1: Overview of the evolution of customer-service question answering research.
• Figure 2: Example of a context-summarized multi-turn customer-service QA instance.
• Figure 3: Dataset construction pipeline for context-summarized customer-service QA.
• Figure 4: Token count distributions for agent answers across train, validation and test splits. Vertical lines indicate the first quartile (Q1), median and third quartile (Q3).
• Figure 5: Total token count distributions across dataset splits using the GPT-4 tokeniser, illustrating overall input length variability and quartile statistics computed over the combined instruction, history_summary, client_question and refined_agent_answer fields.