NC-Bench: An LLM Benchmark for Evaluating Conversational Competence
Robert J. Moore, Sungeun An, Farhan Ahmed, Jay Pankaj Gala
TL;DR
NC-Bench presents a theory-grounded, extensible framework to evaluate conversational competence in LLMs by focusing on the form and sequence of natural dialogue rather than content alone. Grounded in the IBM Natural Conversation Framework, it defines three pattern sets—Basic, RAG, and Complex Request—and a four-stage process (select pattern, create example, generate prompts, judge/evaluate) to operationalize hundreds of generic conversation patterns. Through open-source data and an automated judging pipeline, the benchmark reveals that basic question answering is relatively robust across models, while repair and complex multi-turn tasks remain challenging, with grounding in RAG helping some tasks but also exposing grounding limitations under ungrounded conditions. The work demonstrates the utility of a pattern-centered approach for diagnosing and improving conversational abilities in LLMs, and provides a foundation for systematic, cross-domain evaluation and future expansion of conversational patterns.
Abstract
The Natural Conversation Benchmark (NC-Bench) introduce a new approach to evaluating the general conversational competence of large language models (LLMs). Unlike prior benchmarks that focus on the content of model behavior, NC-Bench focuses on the form and structure of natural conversation. Grounded in the IBM Natural Conversation Framework (NCF), NC-Bench comprises three distinct sets. The Basic Conversation Competence set evaluates fundamental sequence management practices, such as answering inquiries, repairing responses, and closing conversational pairs. The RAG set applies the same sequence management patterns as the first set but incorporates retrieval-augmented generation (RAG). The Complex Request set extends the evaluation to complex requests involving more intricate sequence management patterns. Each benchmark tests a model's ability to produce contextually appropriate conversational actions in response to characteristic interaction patterns. Initial evaluations across 6 open-source models and 14 interaction patterns show that models perform well on basic answering tasks, struggle more with repair tasks (especially repeat), have mixed performance on closing sequences, and find complex multi-turn requests most challenging, with Qwen models excelling on the Basic set and Granite models on the RAG set and the Complex Request set. By operationalizing fundamental principles of human conversation, NC-Bench provides a lightweight, extensible, and theory-grounded framework for assessing and improving the conversational abilities of LLMs beyond topical or task-specific benchmarks.