Facilitating Proactive and Reactive Guidance for Decision Making on the Web: A Design Probe with WebSeek

TL;DR

WebSeek tackles the limitations of text-driven web agents by introducing a data-first, mixed-initiative browser extension that treats data artifacts as first-class citizens on an interactive canvas. It defines a design space for proactive and reactive AI guidance across Discovery, Extraction/Wrangling, Profiling/Cleaning, and Modeling/Visualization, and grounds actions in a tool-based execution architecture to preserve reliability and user control. In a 15-participant exploratory study, WebSeek enabled end-to-end web data sensemaking with high perceived usefulness, strong autonomy, and transparent AI behavior, while revealing varied workflows and a preference for direct data manipulation. Technical evaluation on a 50-task benchmark shows fast guidance generation (approx. 20 seconds) with high accuracy (≈97%), validating the approach and highlighting remaining challenges in data-type inference and context management. Overall, the work argues for moving beyond chatbot-style web agents toward data-centric collaboration where AI assists within a persistent, manipulable workspace, strengthening trust and enabling verifiable, reproducible web-based decision making.

Abstract

Web AI agents such as ChatGPT Agent and GenSpark are increasingly used for routine web-based tasks, yet they still rely on text-based input prompts, lack proactive detection of user intent, and offer no support for interactive data analysis and decision making. We present WebSeek, a mixed-initiative browser extension that enables users to discover and extract information from webpages to then flexibly build, transform, and refine tangible data artifacts-such as tables, lists, and visualizations-all within an interactive canvas. Within this environment, users can perform analysis-including data transformations such as joining tables or creating visualizations-while an in-built AI both proactively offers context-aware guidance and automation, and reactively responds to explicit user requests. An exploratory user study (N=15) with WebSeek as a probe reveals participants' diverse analysis strategies, underscoring their desire for transparency and control during human-AI collaboration.

Figures (7)

• Figure 1: The table editor (A) and visualization editor (B) in WebSeek. In the table editor, an in-situ suggestion is provided suggesting the completion of the next few rows in green after users fill the initial ones. In the visualization editor, users may select a chart type and drag data attributes to the shelves (x-axis, y-axis, color, and size) to create a visualization.
• Figure 2: Illustration of the data capture and source tracing interactions. (A) The user click on an empty cell and clicks on the "capture" button. (B) The user enters the selection mode and is enabled to capture DOM elements. (C) The captured image is saved in the cell. (D) The "source" button is clicked to locate the data source on the web. (E) The browser is automatically navigated to the source page with the source DOM highlighted.
• Figure 3: An illustration of the usage scenario. (A) The user names the workspace before entering the interface. (B) A peripheral proactive AI suggestion is generated. (C) The user captures data from Amazon into cells. (D) An in-situ proactive AI suggestion is generated twice, and the user taps on the tab key to accept it. The user repeats this to get the complete table. (E) The user chats to add new columns. (F) The extracted table from Amazon. (G) The user opens eBay and a new table is extracted simiarly. (H) An proactive AI suggestion for joining two tables is generated. (I) The results after the table join. (J) The user chats to create a visualization. (K) The user checks the generated visualization and navigates to the data source of a chosen item. (L) The item's source highlighted on eBay.
• Figure 4: Participants' ratings on (A) their perceived confidence on the final results and sense of control during the process, and (B) the perceived helpfulness of each system feature.
• Figure 5: Participants' detailed ratings on the four features of the system regarding 1) whether the guidance/feature was reliable, 2) whether the guidance matched the user's intent, 3) whether the guidance/feature helped avoid errors, and 4) whether the user regretted following the guidance.