When Copilot Becomes Autopilot: Generative AI's Critical Risk to Knowledge Work and a Critical Solution

TL;DR

This work argues that the deepest risk of generative AI in knowledge work lies not merely in factual hallucinations but in the potential degradation of human critical thinking as users defer to AI output. It proposes a design paradigm in which AI acts as a provocateur, exposing divergent viewpoints and highlighting risks to keep users engaged in evaluative reasoning. The authors present a Critical Shortlisting Prototype for spreadsheets that generates factors and accompanying provocations, enabling an integrated testbed for critical thinking interventions via factor analyses, per-factor shortlists, and a global weighted shortlist. They outline a comprehensive research agenda addressing interaction design and technical implementation to understand when and how critiques improve thinking, and to guide the development of AI experiences—Tools for Thought—that augment, rather than replace, human reasoning across knowledge-work domains.

Abstract

Generative AI, with its tendency to "hallucinate" incorrect results, may pose a risk to knowledge work by introducing errors. On the other hand, it may also provide unprecedented opportunities for users, particularly non-experts, to learn and apply advanced software features and greatly increase the scope and complexity of tasks they can successfully achieve. As an example of a complex knowledge workflow that is subject to risks and opportunities from generative AI, we consider the spreadsheet. AI hallucinations are an important challenge, but they are not the greatest risk posed by generative AI to spreadsheet workflows. Rather, as more work can be safely delegated to AI, the risk is that human critical thinking -- the ability to holistically and rigorously evaluate a problem and its solutions -- is degraded in the process. The solution is to design the interfaces of generative AI systems deliberately to foster and encourage critical thinking in knowledge work, building primarily on a long history of research on critical thinking tools for education. We discuss a prototype system for the activity of critical shortlisting in spreadsheets. The system uses generative AI to suggest shortlisting criteria and applies these criteria to sort rows in a spreadsheet. It also generates "provocations": short text snippets that critique the AI-generated criteria, highlighting risks, shortcomings, and alternatives. Our prototype opens up a rich and completely unexplored design space of critical thinking tools for modern AI-assisted knowledge work. We outline a research agenda for AI as a critic or provocateur, including questions about where and when provocations should appear, their form and content, and potential design trade-offs.

Figures (5)

• Figure 1: Dataset loading and initial prompt.
• Figure 2: Factor cards
• Figure 3: Factor analysis
• Figure 4: Global list
• Figure 5: Scenario interface (in TypeScript). Implementations of Scenario are used to automate user interactions, and to intercept back end requests for caching and experimental control.