Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ECHO: An Open Research Platform for Evaluation of Chat, Human Behavior, and Outcomes

Jiqun Liu, Nischal Dinesh, Ran Yu

TL;DR

ECHO addresses the need for cohesive, longitudinal evaluation of modern information-access systems by offering an open, low-code platform that supports end-to-end mixed-method studies of chat and Web search interactions. The system enables administrators to configure studies via a dashboard, while participants engage in consent, surveys, and task sessions using either a chat interface powered by LLM APIs or a search interface powered by a Search API, with all interactions logged and exportable. Key contributions include a serverless, dual-modality architecture; flexible API integration; in-situ surveys; and comprehensive data export, all designed to lower barriers for cross-disciplinary, reproducible human-centered AI research. The platform facilitates trajectory-level analyses of user experience, learning, and judgment formation, enabling scalable studies across IR, HCI, cognitive and social sciences with practical impact for evaluating and improving information-access technologies.

Abstract

ECHO (Evaluation of Chat, Human behavior, and Outcomes) is an open research platform designed to support reproducible, mixed-method studies of human interaction with both conversational AI systems and Web search engines. It enables researchers from varying disciplines to orchestrate end-to-end experimental workflows that integrate consent and background surveys, chat-based and search-based information-seeking sessions, writing or judgment tasks, and pre- and post-task evaluations within a unified, low-coding-load framework. ECHO logs fine-grained interaction traces and participant responses, and exports structured datasets for downstream analysis. By supporting both chat and search alongside flexible evaluation instruments, ECHO lowers technical barriers for studying learning, decision making, and user experience across different information access paradigms, empowering researchers from information retrieval, HCI, and the social sciences to conduct scalable and reproducible human-centered AI evaluations.

ECHO: An Open Research Platform for Evaluation of Chat, Human Behavior, and Outcomes

TL;DR

ECHO addresses the need for cohesive, longitudinal evaluation of modern information-access systems by offering an open, low-code platform that supports end-to-end mixed-method studies of chat and Web search interactions. The system enables administrators to configure studies via a dashboard, while participants engage in consent, surveys, and task sessions using either a chat interface powered by LLM APIs or a search interface powered by a Search API, with all interactions logged and exportable. Key contributions include a serverless, dual-modality architecture; flexible API integration; in-situ surveys; and comprehensive data export, all designed to lower barriers for cross-disciplinary, reproducible human-centered AI research. The platform facilitates trajectory-level analyses of user experience, learning, and judgment formation, enabling scalable studies across IR, HCI, cognitive and social sciences with practical impact for evaluating and improving information-access technologies.

Abstract

ECHO (Evaluation of Chat, Human behavior, and Outcomes) is an open research platform designed to support reproducible, mixed-method studies of human interaction with both conversational AI systems and Web search engines. It enables researchers from varying disciplines to orchestrate end-to-end experimental workflows that integrate consent and background surveys, chat-based and search-based information-seeking sessions, writing or judgment tasks, and pre- and post-task evaluations within a unified, low-coding-load framework. ECHO logs fine-grained interaction traces and participant responses, and exports structured datasets for downstream analysis. By supporting both chat and search alongside flexible evaluation instruments, ECHO lowers technical barriers for studying learning, decision making, and user experience across different information access paradigms, empowering researchers from information retrieval, HCI, and the social sciences to conduct scalable and reproducible human-centered AI evaluations.
Paper Structure (18 sections, 5 figures, 2 tables)

This paper contains 18 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. System Overview
  3. Deployment and Resources Requirement
  4. Participant Interface
  5. Administrator Dashboard
  6. System Architecture
  7. Overview
  8. Data Flow
  9. Interaction Logging
  10. Study Flow Management
  11. Key Features
  12. Dual-Modality Support
  13. Flexible API Integration
  14. Multi-Mode Survey Configuration
  15. In-Situ Survey System
  16. ...and 3 more sections

Figures (5)

  • Figure 1: The ECHO participant interaction interface. (a) Task description panel displaying current research task. (b) Chat conversation area with text input for prompts. (c) Optional note taking sidebar for participant responses.
  • Figure 2: ECHO administrator dashboard for study configuration and management. The study components can be customized through the admin interface rather than code.
  • Figure 3: System architecture of ECHO. Administrators configure studies through a Web interface. All data is stored in Firebase and exportable as CSV files for further analysis.
  • Figure 4: The study flow management interface. Workflow steps can be turned on/off and customized individually and reordered through drag-and-drop interaction.
  • Figure 5: In-situ survey popup triggered during interaction.