Disclose with Care: Designing Privacy Controls in Interview Chatbots

TL;DR

Disclose with Care investigates privacy controls for chatbot-based interviews on sensitive topics. It introduces EDIT, featuring free editing and AI-aided editing, to manage disclosures after the interview. In a between-subjects study with $N=188$, AI-aided editing significantly reduces residual PII without harming data quality or engagement. The work demonstrates that post-hoc, AI-guided privacy scaffolding can balance ethical data collection with analytic utility and offers design guidelines for privacy-by-design in chatbot research and deployment.

Abstract

Collecting data on sensitive topics remains challenging in HCI, as participants often withhold information due to privacy concerns and social desirability bias. While chatbots' perceived anonymity may reduce these barriers, research paradoxically suggests people tend to over-share personal or sensitive information with chatbots. In this work, we explore privacy controls in chatbot interviews to address this problem. The privacy control allows participants to revise their transcripts at the end of the interview, featuring two design variants: free editing and AI-aided editing. In a between-subjects study \red{($N=188$)}, we compared no-editing, free-editing, and AI-aided editing conditions in a chatbot-based interview on a sensitive topic. Our results confirm the prevalent issue of oversharing in chatbot-based interviews and show that AI-aided editing serves as an effective privacy-control mechanism, reducing PII disclosure while maintaining data quality and user engagement, thereby offering a promising approach to balancing ethical practice and data quality in such interviews.

Figures (13)

• Figure 1: A snapshot of EDIT user experience in AI-aided Editing during an interview about using AI for job interviews. The interface starts with a chatbot interface (Top) which includes a scrollable interface to demonstrate the chat logs between users and the interview chatbot (A) and the text area (B) that contains the text box (B1) and the send button (B2). The interface in the privacy control phase (Bottom) consists of a conspicuous prompt (D) to notify participants about the following steps, a free-editing area (C) with interview logs (C1) that enable the in-text highlights of PIIs (C2), and a button in warning signs that redirect participants to the corresponding privacy issue card by clicking it (C3). The interface flags potentially sensitive text with a red wavy underline and a privacy tooltip (F) including privacy issue cards (F1) that explain the risk and offer three edit buttons: replace the text with placeholders (F2) or blur sensitive details (F3), with an ignore control if users want to revert to their original logs (F4). For the free-editing group, the participants will only get access to (C) and (D) while the participants in Control group will not do post-interview editing.
• Figure 2: A diagram for conversational orchestration visualization. The process begins when the orchestration engine selects the next main question and prompts the Executor to generate a structured JSON response, which is delivered to the user as the chatbot’s question. Once the user answers, the Auditing Layer evaluates the exchange through two LLM-driven audits: a quality audit ensuring the response is on-topic, well-formed, and free of topic drift, and a completion audit determining whether sufficient follow-ups have been covered. Based on these verdicts, the orchestrator either triggers an additional follow-up, asks for clarification, or advances to the next main question. This loop continues until no further questions remain, concluding the interview.
• Figure 3: Main stages and conditions of the study workflow: 1) Participants are shown the initial consent form for a semi-structured chatbot interview; 2) After the consent form, participants are randomly assigned to one of the three conditions with equal chances to conduct the interview; 3) Depending on the assigned condition, they are provided with one of three post-interview privacy controls after the interview: AI-aided editing, free editing, or no editing (control); 4) Participants complete additional consent inquiries; and 5) All participants finish with a post-task survey
• Figure 4: Mean PII reduction rate by experimental condition. The AI-aided editing condition shows a significant net reduction in PII, while the free-editing condition shows a slight increase of PII after editing. Error bars represent standard deviation.
• Figure 5: Relationship between a participant's initial PII count and the effectiveness of the AI-aided intervention. (Top) Users who started with more PII removed more tokens in absolute terms. (Bottom) Users who started with more PII removed a smaller proportion of their total PII.