LingVarBench: Benchmarking LLMs on Entity Recognitions and Linguistic Verbalization Patterns in Phone-Call Transcripts

TL;DR

LingVarBench tackles the challenge of robust entity extraction from privacy-constrained phone transcripts by synthetically generating linguistically varied data governed by explicit field schemas and verbalization patterns. It combines a three-module data generation pipeline with DSPy+SIMBA prompt optimization to create robust extraction prompts without real PHI, achieving near-human performance on core entities and improvements on subjective items. The framework demonstrates that synthetic, controlled linguistic variation can yield effective prompts that transfer to real production data, enabling cost-efficient, HIPAA-compliant deployment. This approach offers scalable, domain- and language-agnostic opportunities for rapid iteration and safe data-driven development in voice-based healthcare and customer-support systems.

Abstract

We study structured entity extraction from phone-call transcripts in customer-support and healthcare settings, where annotation is costly, and data access is limited by privacy and consent. Existing methods degrade under disfluencies, interruptions, and speaker overlap, yet large real-call corpora are rarely shareable. We introduce LingVarBench, a benchmark and semantic synthetic data generation pipeline that generates linguistically varied training data via (1) LLM-sampled entity values, (2) curated linguistic verbalization patterns covering diverse disfluencies and entity-specific readout styles, and (3) a value-transcript consistency filter. Using this dataset, DSPy's SIMBA automatically synthesizes and optimizes extraction prompts, reducing manual prompt engineering and targeting robustness to verbal variation. On real customer transcripts, prompts optimized solely on LingVarBench outperform zero-shot baselines and match or closely approach human-tuned prompts for structured entities such as ZIP code, date of birth, and name (F1 approximately 94-95 percent). For subjective questionnaire items, optimized prompts substantially improve over zero-shot performance and approach human-tuned prompts. LingVarBench offers a practical and cost-efficient path to deployment in a direct-answer setting, with real annotations later enabling additional refinement.

Figures (7)

• Figure 1: Motivation for LingVarBench: (a) Traditional entity extraction starts with little or no usable transcript data, leading to poor performance and requiring manual prompt tuning as transcripts arrive. (b) LingVarBench instead synthesizes diverse transcript verbalization patterns and uses DSPy + SIMBA to optimize prompts and generate robust, scalable evaluation data.
• Figure 2: Overview of the LingVarBench Synthetic Data Generation Framework: A three-step pipeline for generating linguistically varied transcripts of structured fields. Canonical values (e.g., zip code) are transformed into diverse spoken-style utterances with consistency checks to ensure accurate recovery.
• Figure 3: Abstracted Prompt for Value Generation
• Figure 4: Abstracted Prompt for Transcript Generation
• Figure 5: Abstracted Prompt for Validation