LLMs on Drugs: Language Models Are Few-Shot Consumers
Alexander Doudkin
TL;DR
The paper assesses how psychoactive-style persona prompts influence LLM reasoning on ARC-Challenge, revealing that such framings can undermine strict output templates and reduce accuracy without altering model weights. Using deterministic decoding and rigorous statistics, it shows alcohol severely degrades performance while cannabis and other framings also impair results in distinct ways. The findings highlight the vulnerability of formatting and task compliance to surface-level prompt cues, underscoring the need for persona benchmarking in production systems. An open-source framework is provided to enable reproducible evaluation of persona effects across tasks and models.
Abstract
Large language models (LLMs) are sensitive to the personas imposed on them at inference time, yet prompt-level "drug" interventions have never been benchmarked rigorously. We present the first controlled study of psychoactive framings on GPT-5-mini using ARC-Challenge. Four single-sentence prompts -- LSD, cocaine, alcohol, and cannabis -- are compared against a sober control across 100 validation items per condition, with deterministic decoding, full logging, Wilson confidence intervals, and Fisher exact tests. Control accuracy is 0.45; alcohol collapses to 0.10 (p = 3.2e-8), cocaine to 0.21 (p = 4.9e-4), LSD to 0.19 (p = 1.3e-4), and cannabis to 0.30 (p = 0.041), largely because persona prompts disrupt the mandated "Answer: <LETTER>" template. Persona text therefore behaves like a "few-shot consumable" that can destroy reliability without touching model weights. All experimental code, raw results, and analysis scripts are available at https://github.com/lexdoudkin/llms-on-drugs.