The Shiny Scary Future of Automated Research Synthesis in HCI

TL;DR

The paper examines how automation, including LLMs, can reshape research synthesis in HCI, focusing on the tension between efficiency gains and maintaining methodological rigour. It distinguishes two stages of secondary research—search/selection and analysis/synthesis—and reviews where automation has the strongest potential and where it risks undermining depth. It argues that search and screening may benefit most from automation, while synthesis requires careful human interpretation to preserve nuance and critical appraisal. The authors advocate cautious, transparent integration of AI tools as complementary to human reviewers, with explicit attention to uncertainty and subjectivity.

Abstract

Automation and semi-automation through computational tools like LLMs are also making their way to deployment in research synthesis and secondary research, such as systematic reviews. In some steps of research synthesis, this has the opportunity to provide substantial benefits by saving time that previously was spent on repetitive tasks. The screening stages in particular may benefit from carefully vetted computational support. However, this position paper argues for additional caution when bringing in such tools to the analysis and synthesis phases, where human judgement and expertise should be paramount throughout the process.