Mapping the Stochastic Penal Colony
Robert Grimm
TL;DR
The paper investigates the punitive dimensions of online content moderation by framing algorithmic interventions as a stochastic penal colony, a concept that sits between punishment as performance and discipline. It develops a novel methodology that combines auto-ethnography with procedural justice, and reworks Foucault's penal theory for the algorithmic age, applying this framework to three case studies of pre-Musk Twitter, OpenAI's dall•e 2, and Pinterest. The analysis highlights injustices across voice, agency, dignity, neutrality, and trust, demonstrating how platforms’ punishment practices can resemble transportation-like banishment without transparent governance. The work underscores the need for greater transparency, accountability, and normative checks in platform moderation to prevent abusive, opaque, and potentially discriminatory enforcement from eroding democratic discourse.
Abstract
With peak content moderation seemingly behind us, this paper revisits its punitive side. But instead of focusing on who is being (disproportionately) moderated, it focuses on the punishment itself and makes three contributions. First, it develops a novel methodology that combines auto-ethnography for collecting experiences and artifacts with procedural justice for analyzing them. Second, it reworks Foucault's model of the penal system for the algorithmic age, restoring the penal colony as the historically liminal practice between punishment as performance and punishment as discipline, i.e., the stochastic penal colony. Finally, it applies this methodological and conceptual framing to three case studies, one on the gallingly performative moderation by pre-Musk Twitter, one on the exhaustively punitive content moderation for OpenAI's DALLE~2, and one on the relatively light touch but still rather precious moderation by Pinterest. While substantially different, all three feature the pervasive threat of account suspension, thereby banishing users to the stochastic penal colony.