Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

Ben Wang, Jiqun Liu, Jamshed Karimnazarov, Nicolas Thompson

TL;DR

The paper tackles prompt formulation difficulties and cognitive biases in information-seeking tasks by embedding task context and user perceptions into a ChatGPT-like interface via three supportive functions. It uses a GPT-3.5-turbo-based platform together with a pre-task questionnaire to enrich prompts through perception articulation, prompt suggestions, and conversation explanations, validated by a naturalistic study with 16 participants. Results show that these supports improve expectation management, reduce cognitive load, guide prompt refinement, and boost engagement, contributing to proactive, user-centric human-LLM interaction design and informing evaluation of such interactions. The work offers design guidelines for inclusive, task-aware LLM interfaces and outlines directions for scalable studies, interface enhancements, and fine-tuning strategies to balance engagement and task efficiency.

Abstract

Large language model (LLM) applications, such as ChatGPT, are a powerful tool for online information-seeking (IS) and problem-solving tasks. However, users still face challenges initializing and refining prompts, and their cognitive barriers and biased perceptions further impede task completion. These issues reflect broader challenges identified within the fields of IS and interactive information retrieval (IIR). To address these, our approach integrates task context and user perceptions into human-ChatGPT interactions through prompt engineering. We developed a ChatGPT-like platform integrated with supportive functions, including perception articulation, prompt suggestion, and conversation explanation. Our findings of a user study demonstrate that the supportive functions help users manage expectations, reduce cognitive loads, better refine prompts, and increase user engagement. This research enhances our comprehension of designing proactive and user-centric systems with LLMs. It offers insights into evaluating human-LLM interactions and emphasizes potential challenges for under served users.

Task Supportive and Personalized Human-Large Language Model Interaction: A User Study

TL;DR

The paper tackles prompt formulation difficulties and cognitive biases in information-seeking tasks by embedding task context and user perceptions into a ChatGPT-like interface via three supportive functions. It uses a GPT-3.5-turbo-based platform together with a pre-task questionnaire to enrich prompts through perception articulation, prompt suggestions, and conversation explanations, validated by a naturalistic study with 16 participants. Results show that these supports improve expectation management, reduce cognitive load, guide prompt refinement, and boost engagement, contributing to proactive, user-centric human-LLM interaction design and informing evaluation of such interactions. The work offers design guidelines for inclusive, task-aware LLM interfaces and outlines directions for scalable studies, interface enhancements, and fine-tuning strategies to balance engagement and task efficiency.

Abstract

Large language model (LLM) applications, such as ChatGPT, are a powerful tool for online information-seeking (IS) and problem-solving tasks. However, users still face challenges initializing and refining prompts, and their cognitive barriers and biased perceptions further impede task completion. These issues reflect broader challenges identified within the fields of IS and interactive information retrieval (IIR). To address these, our approach integrates task context and user perceptions into human-ChatGPT interactions through prompt engineering. We developed a ChatGPT-like platform integrated with supportive functions, including perception articulation, prompt suggestion, and conversation explanation. Our findings of a user study demonstrate that the supportive functions help users manage expectations, reduce cognitive loads, better refine prompts, and increase user engagement. This research enhances our comprehension of designing proactive and user-centric systems with LLMs. It offers insights into evaluating human-LLM interactions and emphasizes potential challenges for under served users.
Paper Structure (15 sections, 1 figure, 4 tables)

This paper contains 15 sections, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Capability of ChatGPT
  4. LLMs as Information Access Systems
  5. Evaluating Human-LLM-Interaction
  6. Research Questions
  7. Methodology
  8. System design and supportive functions
  9. Participant recruitment
  10. Analysis
  11. Results
  12. Participants and tasks
  13. Interview case study on task experience
  14. Discussion and conclusion
  15. Acknowledgment

Figures (1)

  • Figure 1: User study platform and prompt templates for supportive functions. Yellow boxes highlight the components for the questionnaires and supportive functions. Grey boxes contain features (including generative features) collected through the questionnaires. Solid arrows indicate the features collected in the pre-task questionnaire, subsequently utilized in prompt suggestions and conversation explanations through prompt engineering. Dashed and dotted boxes contain prompt templates, with {variable features}. Dotted arrows indicate the application of prompt templates in implementing the supportive functions.