How Users Perceive Mixed-Initiative AI: Attitudes Toward Assistance in Problem Solving

TL;DR

The problem addressed is how the timing of AI assistance affects user experience in mixed-initiative problem solving. The authors implement two modes—on-demand (Button) and inactivity-triggered (Timer)—within a budgeted Rush Hour task and compare them to a no-AI control, using BFS-based AI to guarantee optimal moves and a monetary cost structure ($0.01$ per second, $0.05$ per AI move). The study demonstrates that while task performance is similar across AI-delivery modes, the Timer mode yields more positive user perceptions, highlighting that assistance timing and delivery mechanisms can shape user experience independently of outcomes. The findings imply design guidelines for human-AI collaboration: support calibrated, configurable proactivity, improve interpretability, and consider how the mode of AI action affects perceived autonomy and engagement, especially in cost-constrained environments.

Abstract

In mixed-initiative systems, the mode of AI assistance delivery can be as consequential as the assistance itself. We investigated two assistance delivery modes: on-demand help (users request via Button) and pre-scheduled help (assistance delivered at user-selected intervals, with user actions resetting the Timer). To evaluate these modes, we selected Rush Hour puzzles as the human-AI collaborative task because they capture elements of real-world problem solving such as analysis, resource management, and decision-making under constraints. To enhance ecological validity, we imposed monetary costs for both time and AI assistance, simulating scenarios where people must balance implicit or explicit trade-offs such as time pressure, financial limitations, or opportunity costs. Although task performance was comparable across modes, participants who used the pre-scheduled (Timer) mode reported more positive perceptions of the AI, even when their ending budget was low. This suggests that assistance delivery mode can shape user experience independent of task outcomes, indicating that human-AI systems may need to consider how AI assistance is delivered alongside improving task performance.

Figures (9)

• Figure 1: Study System Design. (a) Users can use their mouse to select a car (any colored rectangle), which will be highlighted with a dashed outline, and use the keyboard arrow keys to move the selected car. The two modes of AI assistance are: (b) Button mode: Users may request help at any time during puzzle solving by clicking the Help Button and specifying the number of moves they want AI to help with. The chosen number of moves is preserved for future requests but can be adjusted at any time. (c) Timer mode: Before starting a puzzle, users define an idle-time threshold and the number of moves they want AI to help with. If the user remains inactive (neither selecting nor moving a car) past the idle-time threshold, AI assistance is triggered. If the user selects or moves a puzzle piece before the idle-time threshold is reached, the Timer resets. These parameters cannot be changed during puzzle solving. (d) When AI assistance is triggered, the AI takes control of the board and performs the requested number of moves, with each move associated with a cost of $\$0.05$, deducted from the budget upon activation, in both conditions. While the AI assistant is active, the user may interrupt assistance by clicking on the Stop Help Button available in both conditions. There is a time cost of $\$0.01$/second in both assistance conditions.
• Figure 2: Study Procedure. Participants began by reading an introduction to the Rush Hour puzzle and the study objectives, followed by a pre-study questionnaire. They then completed two practice puzzles: the first without AI assistance to learn the puzzle and controls, and the second with the AI assistant from their assigned condition. After practice, all participants solved two puzzles of different difficulty levels without assistance to establish baseline puzzle-solving skills. They then proceeded to the main study phase, which involved solving five puzzles of varying difficulty, working with the AI corresponding to their assigned condition. The session concluded with a post-study questionnaire.
• Figure 3: User interface used in the user study. (a1) Instruction panel for the Button condition. (a2) User in control in the Button condition. Users can adjust how many moves of help to receive for the next AI assistance. (a3) AI in control; users can click on the Stop Help Button to terminate assistance. (b1) Instruction panel for the Timer condition, with a configuration module below for users to set the idle-time threshold and the number of moves to help. (b2) User in control in the Timer condition. (b3) AI in control; as in the Button condition, assistance can be terminated by clicking on the Stop Help Button. (c) Interface for the control condition and baseline phase. Users may click on the "Reset Puzzle" button to reset the puzzle to its initial state. This reset button is also available in both the Button and Timer conditions.
• Figure 4: Moves precision for main study puzzles across conditions (left). Asterisks denote significant differences between the Button and Control conditions and between the Timer and Control conditions. Comparisons between conditions for each puzzle in the main study phase (right). $O$ denotes the optimum number of moves to solve each puzzle from its initial state. Bars represent means, and error bars denote 95% confidence intervals; these conventions apply to all subsequent bar charts.
• Figure 5: Maximum progress achieved for main study puzzles across conditions (left) and comparisons between conditions for each puzzle in the main study phase (right).