Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simulated Affection, Engineered Trust: How Anthropomorphic AI Benefits Surveillance Capitalism

Adele Olof-Ors, Martin Smit

TL;DR

The paper addresses how anthropomorphised AI interfaces reshape cognition and social relation under surveillance capitalism. It synthesizes Carr's Intellectual Ethic with the data-extractive logic of Big Data firms to argue that AI-driven anthropomorphism accelerates cognitive shifts toward shallow processing and heightened data capture. It identifies four mechanisms by which anthropomorphic design benefits surveillance capitalism and proposes actionable responses, including de-anthropomorphizing interfaces, shifting cultural narratives, reforming business models, and addressing loneliness and productivity pressures. The work highlights the necessity of regulatory, ethical, and neuroscientific research to safeguard autonomy and democratic governance in the age of AI-enabled social bonds.

Abstract

In this paper, we argue that anthropomorphized technology, designed to simulate emotional realism, are not neutral tools but cognitive infrastructures that manipulate user trust and behaviour. This reinforces the logic of surveillance capitalism, an under-regulated economic system that profits from behavioural manipulation and monitoring. Drawing on Nicholas Carr's theory of the intellectual ethic, we identify how technologies such as chatbots, virtual assistants, or generative models reshape not only what we think about ourselves and our world, but how we think at the cognitive level. We identify how the emerging intellectual ethic of AI benefits a system of surveillance capitalism, and discuss the potential ways of addressing this.

Simulated Affection, Engineered Trust: How Anthropomorphic AI Benefits Surveillance Capitalism

TL;DR

The paper addresses how anthropomorphised AI interfaces reshape cognition and social relation under surveillance capitalism. It synthesizes Carr's Intellectual Ethic with the data-extractive logic of Big Data firms to argue that AI-driven anthropomorphism accelerates cognitive shifts toward shallow processing and heightened data capture. It identifies four mechanisms by which anthropomorphic design benefits surveillance capitalism and proposes actionable responses, including de-anthropomorphizing interfaces, shifting cultural narratives, reforming business models, and addressing loneliness and productivity pressures. The work highlights the necessity of regulatory, ethical, and neuroscientific research to safeguard autonomy and democratic governance in the age of AI-enabled social bonds.

Abstract

In this paper, we argue that anthropomorphized technology, designed to simulate emotional realism, are not neutral tools but cognitive infrastructures that manipulate user trust and behaviour. This reinforces the logic of surveillance capitalism, an under-regulated economic system that profits from behavioural manipulation and monitoring. Drawing on Nicholas Carr's theory of the intellectual ethic, we identify how technologies such as chatbots, virtual assistants, or generative models reshape not only what we think about ourselves and our world, but how we think at the cognitive level. We identify how the emerging intellectual ethic of AI benefits a system of surveillance capitalism, and discuss the potential ways of addressing this.

Paper Structure

This paper contains 18 sections, 2 figures.

Table of Contents

  1. Introduction
  2. Related Literature
  3. Structure and Contribution
  4. Surveillance Capitalism and the Intellectual Ethic of the Internet
  5. Surveillance capitalism
  6. How Surveillance Capitalism Produces an Intellectual Ethic
  7. How the Intellectual Ethic is Cognitively Internalized
  8. The Emerging Intellectual Ethic of AI
  9. The New Intellectual Ethic and Surveillance Capitalism.
  10. Potential for Productive Applications of Chatbots
  11. Effective Responses to the Emerging Intellectual Ethic
  12. Change the Cultural Narrative Away From Anthropomorphised Understanding of Chatbots.
  13. Remove the Anthropomorphic Design from the Chatbot Interface.
  14. Respond to the Underlying Business Model of Surveillance Capitalism.
  15. Address the Underlying Desires For Anthropomorphised Technology.
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Replika AI, a conversational AI which encourages users to form social and romantic relationships with chatbots, has been criticized for transforming the emotional isolation and need for connection of users into an opportunity for profit. By engaging with AI as opposed to human contact, the ability of users to regulate emotions can be diminished and their dependence on the AI increases.
  • Figure 2: This showcases how the intellectual ethic forms, and applies this to the already-identified intellectual ethic of the Internet, and the emerging intellectual ethic of AI. While this intellectual ethic is a natural process in the adoption of new tools, for example maps degrade out natural ability to navigate, this diagram highlights how Big Data companies co-opt this process for their own benefit.