The Voice: Lessons on Trustworthy Conversational Agents from "Dune"
Philip Feldman
TL;DR
The paper addresses the risk of weaponized generative AI enabling individualized manipulation across text, audio, image, and video. Using the Bene Gesserit Voice from Dune as a provocation, it connects fictional capabilities to real-world cues in voice and media manipulation. It surveys evidence that vocal cues and deepfakes can influence perception and that recommender systems magnify manipulation through scale and duration. It then proposes a proactive defense framework, notably White Hat AI or Truthsayers, along with the concept of Black Hat AI for adversarial testing, highlighting the need for threat modeling, safeguards, and ethical considerations to mitigate near-term risks.
Abstract
The potential for untrustworthy conversational agents presents a significant threat for covert social manipulation. Taking inspiration from Frank Herbert's "Dune", where the Bene Gesserit Sisterhood uses the Voice for influence, manipulation, and control of people, we explore how generative AI provides a way to implement individualized influence at industrial scales. Already, these models can manipulate communication across text, image, speech, and most recently video. They are rapidly becoming affordable enough for any organization of even moderate means to train and deploy. If employed by malicious actors, they risk becoming powerful tools for shaping public opinion, sowing discord, and undermining organizations from companies to governments. As researchers and developers, it is crucial to recognize the potential for such weaponization and to explore strategies for prevention, detection, and defense against these emerging forms of sociotechnical manipulation.