A Multi-Agent Framework for Mitigating Dialect Biases in Privacy Policy Question-Answering Systems

TL;DR

The paper tackles dialect bias in privacy policy QA by introducing a two-agent, prompt-based framework: a Dialect Agent translates dialectal queries to Standard American English and validates results, while a Privacy Policy Agent reasons over policy text to generate answers. The collaboration, driven by human-centered prompting and iterative refinement, achieves substantial zero-shot gains and reduces cross-dialect disparities on PrivacyQA and PolicyQA without task-specific retraining. Formalizing the objective, the authors minimize the disparity $\Delta(f)=\max_{d_i,d_j \in \mathcal{D}} |\Phi_{d_i}(f) - \Phi_{d_j}(f)|$ while preserving average accuracy, demonstrating that dialect-aware prompting can rival few-shot baselines. These results highlight the practicality and impact of structured multi-agent reasoning for equitable NLP in high-stakes domains like privacy policy understanding.

Abstract

Privacy policies inform users about data collection and usage, yet their complexity limits accessibility for diverse populations. Existing Privacy Policy Question Answering (QA) systems exhibit performance disparities across English dialects, disadvantaging speakers of non-standard varieties. We propose a novel multi-agent framework inspired by human-centered design principles to mitigate dialectal biases. Our approach integrates a Dialect Agent, which translates queries into Standard American English (SAE) while preserving dialectal intent, and a Privacy Policy Agent, which refines predictions using domain expertise. Unlike prior approaches, our method does not require retraining or dialect-specific fine-tuning, making it broadly applicable across models and domains. Evaluated on PrivacyQA and PolicyQA, our framework improves GPT-4o-mini's zero-shot accuracy from 0.394 to 0.601 on PrivacyQA and from 0.352 to 0.464 on PolicyQA, surpassing or matching few-shot baselines without additional training data. These results highlight the effectiveness of structured agent collaboration in mitigating dialect biases and underscore the importance of designing NLP systems that account for linguistic diversity to ensure equitable access to privacy information.

Figures (3)

• Figure 1: Illustration of dialect-based disparities in Privacy Question Answering (QA). The QA model correctly answers a query phrased in Standard American English (SAE) but produces an incorrect response when the same query is asked in African American Vernacular English (AAVE).
• Figure 2: Our multi-agent framework for mitigating dialect biases in privacy QA. The Dialect Agent translates queries into Standard American English (SAE) and validates responses. The Privacy Policy Agent generates answers based on policy text. Disagreements trigger refinement, ensuring accurate and inclusive responses across dialects.
• Figure 3: Comparison of the few-shot baseline performance (grey) $F_1$ scores with the improvements achieved by our method (colored bars) for each model on PrivacyQA. We compare SAE with the two highest-performing dialects for each model.