Call2Instruct: Automated Pipeline for Generating Q&A Datasets from Call Center Recordings for LLM Fine-Tuning

TL;DR

The paper addresses the bottleneck of creating high-quality instructional Q&A data from unstructured call-center audio for LLM fine-tuning. It introduces Call2Instruct, an end-to-end pipeline that performs audio preprocessing (diarization, denoising, IVR removal), transcription, textual cleaning/anonymization, semantic extraction and embedding-based matching, followed by Q&A generation and validation. The authors implement the pipeline on thousands of telecom recordings, produce an instruction-format dataset, and validate its utility via fine-tuning the Llama 2 7B model using Lamini, demonstrating domain adaptation with qualitative success and highlighting areas for improvement. Public release of the code promotes reproducibility and provides a practical path to transform raw customer conversations into domain-specific training data for Q&A tasks in customer service.

Abstract

The adaptation of Large-Scale Language Models (LLMs) to specific domains depends on high-quality fine-tuning datasets, particularly in instructional format (e.g., Question-Answer - Q&A). However, generating these datasets, particularly from unstructured sources such as call center audio recordings, poses a significant challenge due to the noisy and disorganized nature of the data. This paper presents a solution to this challenge by offering an end-to-end automated pipeline for generating Q&A instructional datasets from such recordings. The methodology developed comprises sequential steps of audio processing (including diarization, noise removal and automatic transcription), textual processing (cleaning, normalization, and anonymization), semantic extraction of customer demands and attendant responses using vector embeddings, and matching via semantic search to form the final Q&A pairs. As a result, the complete pipeline was successfully implemented, generating a dataset specifically formatted for Instruct Fine Tuning. The practical value and feasibility of the generated dataset were substantiated and functionally demonstrated through the successful fine-tuning of an LLM model (based on Llama 2 7B). The conclusion of the paper states that the proposed approach is viable for converting unstructured conversational data from call centers into valuable resources for training LLMs. This development has the potential to open up avenues for creating more effective AI systems for Q&A tasks in the customer service domain. The developed codes have been made publicly available to promote reproducibility and future research.

Figures (1)

• Figure 1: Conceptual overview of the Call2Instruct automated pipeline.