Generative AI in Knowledge Work: Design Implications for Data Navigation and Decision-Making

TL;DR

Knowledge workers struggle to synthesize unstructured data across platforms, hindering decision-making. The authors design Yodeai, an AI-enabled, widget-based system, and evaluate it through formative and PM-focused user studies to uncover how AI can assist data navigation and prioritization while revealing limitations. They identify three design imperatives—workflow adaptability, accountability through cross-verification and audit trails, and context-aware interoperability—to guide future human-centered AI tools. The work demonstrates that AI can provide starting points at varying abstraction levels, support transparent collaboration, and intermix background knowledge with external information, but requires careful management of data freshness, privacy, and user control. Collectively, these insights offer practical guidance for designing AI tools that augment knowledge-work practitioners without diminishing critical human judgment.

Abstract

Our study of 20 knowledge workers revealed a common challenge: the difficulty of synthesizing unstructured information scattered across multiple platforms to make informed decisions. Drawing on their vision of an ideal knowledge synthesis tool, we developed Yodeai, an AI-enabled system, to explore both the opportunities and limitations of AI in knowledge work. Through a user study with 16 product managers, we identified three key requirements for Generative AI in knowledge work: adaptable user control, transparent collaboration mechanisms, and the ability to integrate background knowledge with external information. However, we also found significant limitations, including overreliance on AI, user isolation, and contextual factors outside the AI's reach. As AI tools become increasingly prevalent in professional settings, we propose design principles that emphasize adaptability to diverse workflows, accountability in personal and collaborative contexts, and context-aware interoperability to guide the development of human-centered AI systems for product managers and knowledge workers.

Figures (11)

• Figure 1: Yodeai hosts three AI-powered workflow templates (widgets) that transform unstructured data into actionable insights. 1) Q&A: ask questions and retrieve relevant information with citations (conversational). 2) Pain Point Tracker: cluster and quantify key issues over time (quantitative). 3) User Insights: view multi-granular summaries on a whiteboard (visual). With these widgets, users can explore data, identify key points, and visualize metrics to support exploration and decision-making.
• Figure 2: A voting interface for PMs to either submit their own ideas in response to the prompt or vote on existing ideas for PM tasks that AI could potentially assist with.
• Figure 3: The top 10 results from the public poll, with each idea’s score representing its estimated likelihood of winning against a randomly selected idea (100 indicating a guaranteed win, 0 indicating a guaranteed loss).
• Figure 4: Yodeai's space and page structure. Left: A space containing multiple pages including raw data (user interviews) and widget outputs (user insights). Right: Individual page view showing an interview transcript. This organization allows users to maintain connections between source data and derived analyses.
• Figure 5: The Q&A widget provides conversational access to data across different contexts. Left: Direct querying of notion reviews about pain points, with sources linked. Right: Analysis of User Insights widget output, where users can ask follow-up questions about patterns and themes identified in the sticky notes.