The Good, the Bad, and the Ugly: The Role of AI Quality Disclosure in Lie Detection
Haimanti Bhattacharya, Subhasish Dugar, Sanchaita Hazra, Bodhisattwa Prasad Majumder
TL;DR
The study investigates how disclosing AI advisor quality affects truth-detection when using AI to identify text-based lies. Using a dataset of neutral, text-based transcripts from To Tell The Truth and GPT-4-generated AI guesses, the authors simulate low-, medium-, and high-quality AI advisors and compare outcomes under black-box versus information disclosure conditions. The results show that high-quality AI boosted final truth detection regardless of disclosure, while low- and medium-quality AIs can depress performance when their quality is unknown, but disclosure mitigates this risk. The findings imply that AI transparency, aligned with user expectations, can enhance welfare and reduce the spread of deceptive content online, with important policy implications for AI governance and platform design. The work advances deception detection literature by linking AI quality, user beliefs, and information design in a high-stakes textual setting, offering actionable guidance for mitigating online misinformation.
Abstract
We investigate how low-quality AI advisors, lacking quality disclosures, can help spread text-based lies while seeming to help people detect lies. Participants in our experiment discern truth from lies by evaluating transcripts from a game show that mimicked deceptive social media exchanges on topics with objective truths. We find that when relying on low-quality advisors without disclosures, participants' truth-detection rates fall below their own abilities, which recovered once the AI's true effectiveness was revealed. Conversely, high-quality advisor enhances truth detection, regardless of disclosure. We discover that participants' expectations about AI capabilities contribute to their undue reliance on opaque, low-quality advisors.