Bringing Comparative Cognition To Computers

TL;DR

The paper addresses how to rigorously assess AI cognition without over- or under-stating its capabilities. It argues for a comparative cognition framework that borrows tools from animal cognition to design valid tests and interpretations for AI, highlighting risks of misattribution such as tokenization-induced failures and Clever Hans-like cues. By outlining under- and over-attribution pitfalls and proposing a cross-species comparative program, the authors seek to integrate AI cognition into the broader cognitive sciences. This approach promises clearer definitions of cognition and more robust, safe, and scientifically grounded assessments of AI systems across diverse architectures.

Abstract

Researchers are increasingly subjecting artificial intelligence systems to psychological testing. But to rigorously compare their cognitive capacities with humans and other animals, we must avoid both over- and under-stating our similarities and differences. By embracing a comparative approach, we can integrate AI cognition research into the broader cognitive sciences.