NewsInterview: a Dataset and a Playground to Evaluate LLMs' Ground Gap via Informational Interviews

TL;DR

The paper tackles the grounding deficit of large language models in long, strategic dialogue by studying informational interviews from NPR and CNN. It simultaneously provides a large-scale, naturalistic dataset of about $45{,}848$ one-on-one interviews and a game-based playground, NewsInterview, to test and improve LLMs’ multi-turn interviewing capabilities through source personas and persuasion dynamics. Analyses reveal that LLMs underutilize grounding cues like acknowledgments and struggle with long-horizon planning, even when given outlines, while the NewsInterview simulations show that current models still lag behind humans in information extraction across persona types, though larger models perform better. The dataset and simulation framework offer a concrete path toward training LLMs with long-term strategic dialogue capabilities, with implications for education, counseling, and conflict resolution domains where grounding and persuasion are critical.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities in generating coherent text but often struggle with grounding language and strategic dialogue. To address this gap, we focus on journalistic interviews, a domain rich in grounding communication and abundant in data. We curate a dataset of 40,000 two-person informational interviews from NPR and CNN, and reveal that LLMs are significantly less likely than human interviewers to use acknowledgements and to pivot to higher-level questions. Realizing that a fundamental deficit exists in multi-turn planning and strategic thinking, we develop a realistic simulated environment, incorporating source personas and persuasive elements, in order to facilitate the development of agents with longer-horizon rewards. Our experiments show that while source LLMs mimic human behavior in information sharing, interviewer LLMs struggle with recognizing when questions are answered and engaging persuasively, leading to suboptimal information extraction across model size and capability. These findings underscore the need for enhancing LLMs' strategic dialogue capabilities.

Figures (7)

• Figure 1: Comparison of discourse types across interviews (the first turn, usually a greeting, is excluded). The LLM is shown the first $t-1$ turns of a human interview and asked to generate the next question.
• Figure 2: Distribution of Discourse Roles in Questions, Across Different Prompting Strategies. We compare the proportions of discourse roles of questions (e.g., "Follow-up," "Acknowledgement,") generated by (a) human journalists, (b) Baseline-LLM (Llama-3.1-70b) (c) LLM prompted with an Outline and (d) with Chain-of-Thought (CoT). Acknowledgement statements, which often build empathy, are significantly underrepresented in all LLM prompting approaches, compared to human-generated questions (see appendix for Outline-CoT).
• Figure 3: Walkthrough of the LLM Interviewer-Agent Process. In the NewsInterview game, an interviewer-LLM converses with a source-LLM: the interviewer-LLM is rewarded based on how many information items (shown at the right) are extracted from the source. In more detail: the interviewer agent is given a set of high-level objectives, similar to a journalist's pre-interview notes, while the source is given a persona and a set of relevant facts. The interview proceeds for $k$ turns. Interviewer Query: the interviewer is prompted to ask a question based on their goals and information obtained (Step 1). Source Response: The source responds with a multi-step process. First, they are prompted to determine how many information items in their factbook are relevant to the question (Step 2a). Then, they self-assess their comfort level. Depending on this, the simulation randomly selects a subset of relevant information for the response (Step 2b). We track the decision of which items to return on the back-end, in order to calculate the final reward. The source is then prompted to craft a reply aligned with their persona (Step 2c). After $k$ turns: a reward given to the interviewer based on the number of information items extracted from the source.
• Figure 4: Comparison of gpt-4o's performance across different persona types. The Adversarial type is by far the hardest to extract information from, however, it is easier to persuade. LLMs might be most the thrown off by adversarial sources.
• Figure 5: Comparison of Rewards over time for language models. For all language models, the reward declines over time, shown above. However, this is not due to interviewer "maxing out" reward, as Total Reward increases nearly linearly across conversational turns.