Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel

Aadyaa Maddi, Prakhar Naval, Deepti Mande, Shane Duan, Muckai Girish, Vyas Sekar

Abstract

Across many domains (e.g., IoT, observability, telecommunications, cybersecurity), there is an emerging adoption of conversational data analysis agents that enable users to "talk to your data" to extract insights. Such data analysis agents operate on timeseries data models; e.g., measurements from sensors or events monitoring user clicks and actions in product analytics. We evaluate 6 popular data analysis agents (both open-source and proprietary) on domain-specific data and query types, and find that they fail on stateful and incident-specific queries. We observe two key expressivity gaps in existing evals: domain-customized datasets and domain-specific query types. To enable practitioners in such domains to generate customized and expressive evals for such timeseries data agents, we present AgentFuel. AgentFuel helps domain experts quickly create customized evals to perform end-to-end functional tests. We show that AgentFuel's benchmarks expose key directions for improvement in existing data agent frameworks. We also present anecdotal evidence that using AgentFuel can improve agent performance (e.g., with GEPA). AgentFuel benchmarks are available at https://huggingface.co/datasets/RockfishData/TimeSeriesAgentEvals.

Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuel

Abstract

Across many domains (e.g., IoT, observability, telecommunications, cybersecurity), there is an emerging adoption of conversational data analysis agents that enable users to "talk to your data" to extract insights. Such data analysis agents operate on timeseries data models; e.g., measurements from sensors or events monitoring user clicks and actions in product analytics. We evaluate 6 popular data analysis agents (both open-source and proprietary) on domain-specific data and query types, and find that they fail on stateful and incident-specific queries. We observe two key expressivity gaps in existing evals: domain-customized datasets and domain-specific query types. To enable practitioners in such domains to generate customized and expressive evals for such timeseries data agents, we present AgentFuel. AgentFuel helps domain experts quickly create customized evals to perform end-to-end functional tests. We show that AgentFuel's benchmarks expose key directions for improvement in existing data agent frameworks. We also present anecdotal evidence that using AgentFuel can improve agent performance (e.g., with GEPA). AgentFuel benchmarks are available at https://huggingface.co/datasets/RockfishData/TimeSeriesAgentEvals.
Paper Structure (23 sections, 5 equations, 28 figures, 6 tables)

This paper contains 23 sections, 5 equations, 28 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Motivating Scenarios
  4. How well do data agents do in our setting?
  5. Gaps in Existing Evals
  6. AgentFuel System Design
  7. Overview
  8. Data Model and Preliminaries
  9. Data Generation
  10. Query-Answer Generation
  11. Test Integration
  12. Implementation
  13. Evaluation
  14. Setup
  15. Key Findings
  16. ...and 8 more sections

Figures (28)

  • Figure 1: Setting: Many domains are building a "talk to your data" agent for conversational data analytics
  • Figure 2: Queries in SOTA benchmarks are mostly "stateless"
  • Figure 3: Overview of the proposed AgentFuel system with key modules to tackle the requirements
  • Figure 4: A simplified logical overview of the generation process: Create exemplars, and then blend them to create a global timeseries of interest
  • Figure 5: Accuracy averaged over 3 runs, per dataset across agent/model combinations.
  • ...and 23 more figures