Investigating the Effects of LLM Use on Critical Thinking Under Time Constraints: Access Timing and Time Availability

Abstract

The impact of large language models (LLMs) on critical thinking has provoked growing attention, yet this impact on actual performance may not be uniformly negative or positive. Particularly, the role of time -- the temporal context under which an LLM is provided -- remains overlooked. In a between-subjects experiment (n=393), we examined two types of time constraints for a critical thinking task requiring participants to make a reasoned decision for a real-world scenario based on diverse documents: (1) LLM access timing -- an LLM available only at the beginning (early), throughout (continuous), near the end (late), or not at all (no LLM), and (2) time availability -- insufficient or sufficient time for the task. We found a temporal reversal: LLM access from the start (early, continuous) improved performance under time pressure but impaired it with sufficient time, whereas beginning the task independently (late, no LLM) showed the opposite pattern. These findings demonstrate that time constraints fundamentally shape whether an LLM augments or undermines critical thinking, making time a central consideration when designing LLM support and evaluating human-AI collaboration in cognitive tasks.

Figures (9)

• Figure 1: Interface for the main task in the six conditions having LLM access (earlyblueEarly, contblueContinuous, lateblueLate LLM access under Insufficient and Sufficient time availability): (a) instructions and (b) an example of the active interface during the task. Participants in two conditions without LLM access (noblueNo LLM access under Insufficient and Sufficient time availability) only have the document viewer and the essay textbox.
• Figure 2: Interface for assessing (a) Recall and (b) Evaluation performance.
• Figure 3: Estimated means of the main score for each measure across all conditions. Circles and diamonds show the estimated means when time availability is Insufficient and Sufficient, respectively. Error bars represent standard errors. Dashed lines indicate the reference of noblueNo LLM access. The effects of LLM access timing and time availability were most pronounced for (a) Essay score and (b) Recall score. See Table \ref{['tab:ancova-posthoc-insufficient']} and \ref{['tab:ancova-posthoc-sufficient']} for detailed pairwise comparisons between LLM access timings.
• Figure 4: Estimated means of (a) Myside Bias score and (b) number of arguments across conditions. Error bars represent standard errors. Dashed lines indicate the reference of noblueNo LLM access. Comparing (a) and (b), only lateblueLate LLM access appeared to reduce Myside Bias while maintaining argument quantity under Sufficient time, whereas the stable low Myside Bias for earlyblueEarly and contblueContinuous LLM access may reflect fewer arguments rather than balanced reasoning. See Section \ref{['myside_bias_score']}.
• Figure 5: Estimated means of Evaluation for all three components across conditions. Error bars represent standard errors. Dashed lines indicate the reference of noblueNo LLM access. Sufficient time generally improved Evaluation with modest differences between LLM access timings. The exception was Evaluation (trustworthiness), where contblueContinuous LLM access impaired it under Sufficient time. See Section \ref{['evaluation_results']}.