Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Toward best research practices in AI Psychology

Anna A. Ivanova

TL;DR

14 methodological considerations are discussed that can help design more robust, generalizable studies evaluating the cognitive abilities of language-based AI systems, as well as to accurately interpret the results of these studies.

Abstract

Language models have become an essential part of the burgeoning field of AI Psychology. I discuss 14 methodological considerations that can help design more robust, generalizable studies evaluating the cognitive abilities of language-based AI systems, as well as to accurately interpret the results of these studies.

Toward best research practices in AI Psychology

TL;DR

14 methodological considerations are discussed that can help design more robust, generalizable studies evaluating the cognitive abilities of language-based AI systems, as well as to accurately interpret the results of these studies.

Abstract

Language models have become an essential part of the burgeoning field of AI Psychology. I discuss 14 methodological considerations that can help design more robust, generalizable studies evaluating the cognitive abilities of language-based AI systems, as well as to accurately interpret the results of these studies.
Paper Structure (6 sections, 1 figure)

This paper contains 6 sections, 1 figure.

Table of Contents

  1. Cognitive evaluations of AI models
  2. Experimental Design
  3. Interpreting the results
  4. Testing open vs. closed models
  5. Acknowledgements
  6. Competing interests

Figures (1)

  • Figure 1: The do's and don'ts of AI model evaluation on cognitive tests.