Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Supporting Effective Goal Setting with LLM-Based Chatbots

Michel Schimpf, Sebastian Maier, Anton Wyrowski, Lara Christoforakos, Stefan Feuerriegel, Thomas Bohné

TL;DR

The paper addresses how to scale goal setting theory and implementation intentions through LLM-based chatbots. It employs a preregistered randomized controlled trial (N = 543) to test three design features—guidance, adaptive suggestions, and feedback—across five chatbot conditions, focusing on goal difficulty, specificity, and implementation-intention quality. Results show that guidance and especially feedback improve goal quality, while adaptive suggestions offer limited benefits and highly sophisticated GenBot designs may dampen commitment, though they can increase perceived social presence. The findings illuminate a trade-off between output quality and motivation, and provide concrete design guidelines for scalable, domain-agnostic behavior-change interventions using AI chatbots.

Abstract

Each day, individuals set behavioral goals such as eating healthier, exercising regularly, or increasing productivity. While psychological frameworks (i.e., goal setting and implementation intentions) can be helpful, they often need structured external support, which interactive technologies can provide. We thus explored how large language model (LLM)-based chatbots can apply these frameworks to guide users in setting more effective goals. We conducted a preregistered randomized controlled experiment ($N = 543$) comparing chatbots with different combinations of three design features: guidance, suggestions, and feedback. We evaluated goal quality using subjective and objective measures. We found that, while guidance is already helpful, it is the addition of feedback that makes LLM-based chatbots effective in supporting participants' goal setting. In contrast, adaptive suggestions were less effective. Altogether, our study shows how to design chatbots by operationalizing psychological frameworks to provide effective support for reaching behavioral goals.

Supporting Effective Goal Setting with LLM-Based Chatbots

TL;DR

The paper addresses how to scale goal setting theory and implementation intentions through LLM-based chatbots. It employs a preregistered randomized controlled trial (N = 543) to test three design features—guidance, adaptive suggestions, and feedback—across five chatbot conditions, focusing on goal difficulty, specificity, and implementation-intention quality. Results show that guidance and especially feedback improve goal quality, while adaptive suggestions offer limited benefits and highly sophisticated GenBot designs may dampen commitment, though they can increase perceived social presence. The findings illuminate a trade-off between output quality and motivation, and provide concrete design guidelines for scalable, domain-agnostic behavior-change interventions using AI chatbots.

Abstract

Each day, individuals set behavioral goals such as eating healthier, exercising regularly, or increasing productivity. While psychological frameworks (i.e., goal setting and implementation intentions) can be helpful, they often need structured external support, which interactive technologies can provide. We thus explored how large language model (LLM)-based chatbots can apply these frameworks to guide users in setting more effective goals. We conducted a preregistered randomized controlled experiment () comparing chatbots with different combinations of three design features: guidance, suggestions, and feedback. We evaluated goal quality using subjective and objective measures. We found that, while guidance is already helpful, it is the addition of feedback that makes LLM-based chatbots effective in supporting participants' goal setting. In contrast, adaptive suggestions were less effective. Altogether, our study shows how to design chatbots by operationalizing psychological frameworks to provide effective support for reaching behavioral goals.
Paper Structure (38 sections, 6 figures, 13 tables)

This paper contains 38 sections, 6 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Theoretical Foundations of Goal Attainment
  4. Technology for Behavioral Interventions
  5. Hypotheses: Design Features in Behavioral Interventions
  6. Methodology
  7. Prototype Design
  8. Intervention Groups
  9. Procedure
  10. Measures
  11. Study Population
  12. Statistical Analysis
  13. Ethics
  14. Results
  15. Main Analyses of Design Features
  16. ...and 23 more sections

Figures (6)

  • Figure 1: System architecture of the goal setting chatbot application.(A) The mobile app serves as the user interface, enabling participants to interact with the chatbot. (B) Requests are routed through the backend API, which handles communication between the app, database, and chatbot module. (C) The database module stores user information (e.g., messages, goal states) and manages group assignment and questionnaire tracking. (D) The LLM chat module implements a state-based architecture for structured goal setting conversations. Each conversational state corresponds to a step in the procedure. (E) The system connects to the OpenAI API for LLM-generated outputs.
  • Figure 2: Example interactions with the goal assistant chatbots demonstrating different design features.(A) Guidance with a general example: The chatbot uses a general swimming example to help the user think about learning-oriented goals. (B) Suggestions with personalization: Adaptive strategies tailored to a marathon goal are provided. (C) No feedback: The chatbot accepts an underspecified cue (“this evening”) without prompting for clarification. (D) Feedback: The FeedbackBot detects that the cue is vague and encourages the user to make it more concrete and situational.
  • Figure 3: Example of a system prompt (left) and function call (right) for the "specific goal" state of the FeedbackBot. Each state in the FeedbackBot has a similar prompt, with an explicit instruction not to give suggestions, ensuring it is differentiated from the SuggestionBot condition. This instruction is removed in the GenBot.
  • Figure 4: Mediation model showing that social presence mediates the relationship between chatbot type and goal commitment. Compared to the ControlBot, the GenBot increased perceived social presence, which in turn predicted higher goal commitment. The direct effect of chatbot type on goal commitment remained negative, indicating a suppression effect.
  • Figure 5: The two messages of Control Bot that are sent to the user. These are hard-coded, but delivered through the same chat interface as all other conditions. The first message on the left explains goal setting theory and what makes a good goal. The second message on the right then instructs the user to set an implementation intention for the goal.
  • ...and 1 more figures