Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reasoning About Reasoning: Towards Informed and Reflective Use of LLM Reasoning in HCI

Ramaravind Kommiya Mothilal, Sally Zhang, Syed Ishtiaque Ahmed, Shion Guha

TL;DR

This paper analyzes how LLM reasoning is framed, executed, and evaluated in HCI by surveying 258 CHI papers from 2020 to 2025, revealing a predominance of framing LLM reasoning as a motivational tool and a heavy reliance on prompting, often neglecting the underlying sociotechnical mechanisms. By contextualizing HCI practices within NLP/ML foundations, the authors show that HCI abstracts away the structured nature of reasoning, potentially misrepresenting capabilities and limitations. They propose a constructive tool of reflection prompts to foster informed and reflective engagement with LLM reasoning, and they make the prompts open-source as a living document to bridge NLP/ML and HCI communities. The work highlights opportunities to improve evaluation, data practices, and training approaches (SFT/RLHF) in HCI, aiming for more rigorous, transparent, and responsible deployment of reasoning-enabled systems.

Abstract

Reasoning is a distinctive human-like characteristic attributed to LLMs in HCI due to their ability to simulate various human-level tasks. However, this work argues that the reasoning behavior of LLMs in HCI is often decontextualized from the underlying mechanics and subjective decisions that condition the emergence and human interpretation of this behavior. Through a systematic survey of 258 CHI papers from 2020-2025 on LLMs, we discuss how HCI hardly perceives LLM reasoning as a product of sociotechnical orchestration and often references it as an object of application. We argue that such abstraction leads to oversimplification of reasoning methodologies from NLP/ML and results in a distortion of LLMs' empirically studied capabilities and (un)known limitations. Finally, drawing on literature from both NLP/ML and HCI, as a constructive step forward, we develop reflection prompts to support HCI practitioners engage with LLM reasoning in an informed and reflective way.

Reasoning About Reasoning: Towards Informed and Reflective Use of LLM Reasoning in HCI

TL;DR

This paper analyzes how LLM reasoning is framed, executed, and evaluated in HCI by surveying 258 CHI papers from 2020 to 2025, revealing a predominance of framing LLM reasoning as a motivational tool and a heavy reliance on prompting, often neglecting the underlying sociotechnical mechanisms. By contextualizing HCI practices within NLP/ML foundations, the authors show that HCI abstracts away the structured nature of reasoning, potentially misrepresenting capabilities and limitations. They propose a constructive tool of reflection prompts to foster informed and reflective engagement with LLM reasoning, and they make the prompts open-source as a living document to bridge NLP/ML and HCI communities. The work highlights opportunities to improve evaluation, data practices, and training approaches (SFT/RLHF) in HCI, aiming for more rigorous, transparent, and responsible deployment of reasoning-enabled systems.

Abstract

Reasoning is a distinctive human-like characteristic attributed to LLMs in HCI due to their ability to simulate various human-level tasks. However, this work argues that the reasoning behavior of LLMs in HCI is often decontextualized from the underlying mechanics and subjective decisions that condition the emergence and human interpretation of this behavior. Through a systematic survey of 258 CHI papers from 2020-2025 on LLMs, we discuss how HCI hardly perceives LLM reasoning as a product of sociotechnical orchestration and often references it as an object of application. We argue that such abstraction leads to oversimplification of reasoning methodologies from NLP/ML and results in a distortion of LLMs' empirically studied capabilities and (un)known limitations. Finally, drawing on literature from both NLP/ML and HCI, as a constructive step forward, we develop reflection prompts to support HCI practitioners engage with LLM reasoning in an informed and reflective way.
Paper Structure (30 sections, 2 figures, 2 tables)

This paper contains 30 sections, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Methods
  4. LLM Reasoning in HCI
  5. Framing of Reasoning
  6. Motivators for Application.
  7. Sequential Progression of Responses.
  8. Execution of Reasoning
  9. Prompting as Gold Standard.
  10. Interactions Within System
  11. Evaluation of Reasoning
  12. Subjective Utility-Based Validation
  13. Latent Validation
  14. HCI's Abstraction of LLM Reasoning
  15. Framing of Reasoning
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Total LLM-related papers at CHI over the years, including the total that engage with LLM reasoning and the total that use LLM reasoning as motivators.
  • Figure 2: Total LLM-related papers that apply LLMs to build systems across different domains, including the total that use Chain-of-Thought prompting.