Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NOVAID: Natural-language Observability Visualization Assistant for ITOps Dashboard Widget Generation

Pratik Mishra, Caner Gözübüyük, Seema Nagar, Prateeti Mohapatra, Raya Wittich, Arthur de Magalhaes

TL;DR

NOVAID addresses the challenge of slow and error-prone manual widget creation in IT Operations dashboards by enabling natural-language to widget generation tailored to ITOps needs. It combines a domain-aware semantic parser, fuzzy matching, and a knowledge-base–driven JSON schema pipeline to produce production-ready widget configurations from user queries, with interactive clarification to resolve underspecifications. Evaluations on a curated 271-query dataset show strong core-component extraction across multiple LLMs, and a user study reports a SUS of 74.2, indicating good usability and practical viability. The work demonstrates deployment potential within enterprise monitoring platforms like Instana and outlines a concrete path for broader adoption and future improvements across widget types and data sources.

Abstract

Manual creation of IT monitoring dashboard widgets is slow, error-prone, and a barrier for both novice and expert users. We present NOVAID, an interactive chatbot that leverages Large Language Models (LLMs) to generate IT monitoring widgets directly from natural language queries. Unlike general natural language-to-visualization tools, NOVAID addresses IT operations-specific challenges: specialized widget types like SLO charts, dynamic API-driven data retrieval, and complex contextual filters. The system combines a domain-aware semantic parser, fuzzy entity matching, and schema completion to produce standardized widget JSON specifications. An interactive clarification loop ensures accuracy in underspecified queries. On a curated dataset of 271 realistic queries, NOVAID achieves promising accuracy (up to 94.10% in metric extraction) across multiple LLMs. A user study with IT engineers yielded a System Usability Scale score of 74.2 for NOVAID, indicating good usability. By bridging natural language intent with operational dashboards, NOVAID demonstrates clear potential and a path for deployment in enterprise ITOps monitoring platforms.

NOVAID: Natural-language Observability Visualization Assistant for ITOps Dashboard Widget Generation

TL;DR

NOVAID addresses the challenge of slow and error-prone manual widget creation in IT Operations dashboards by enabling natural-language to widget generation tailored to ITOps needs. It combines a domain-aware semantic parser, fuzzy matching, and a knowledge-base–driven JSON schema pipeline to produce production-ready widget configurations from user queries, with interactive clarification to resolve underspecifications. Evaluations on a curated 271-query dataset show strong core-component extraction across multiple LLMs, and a user study reports a SUS of 74.2, indicating good usability and practical viability. The work demonstrates deployment potential within enterprise monitoring platforms like Instana and outlines a concrete path for broader adoption and future improvements across widget types and data sources.

Abstract

Manual creation of IT monitoring dashboard widgets is slow, error-prone, and a barrier for both novice and expert users. We present NOVAID, an interactive chatbot that leverages Large Language Models (LLMs) to generate IT monitoring widgets directly from natural language queries. Unlike general natural language-to-visualization tools, NOVAID addresses IT operations-specific challenges: specialized widget types like SLO charts, dynamic API-driven data retrieval, and complex contextual filters. The system combines a domain-aware semantic parser, fuzzy entity matching, and schema completion to produce standardized widget JSON specifications. An interactive clarification loop ensures accuracy in underspecified queries. On a curated dataset of 271 realistic queries, NOVAID achieves promising accuracy (up to 94.10% in metric extraction) across multiple LLMs. A user study with IT engineers yielded a System Usability Scale score of 74.2 for NOVAID, indicating good usability. By bridging natural language intent with operational dashboards, NOVAID demonstrates clear potential and a path for deployment in enterprise ITOps monitoring platforms.
Paper Structure (22 sections, 6 figures, 6 tables)

This paper contains 22 sections, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. NOVAID System Overview
  4. Knowledge Base Initialization
  5. Semantic Parser
  6. Fuzzy Matching of Extracted Elements
  7. JSON Schema Population and Widget Rendering
  8. Experiments and Results
  9. Evaluation Methodology
  10. Model Details and Configuration
  11. Results
  12. Deployment
  13. NOVAID Frontend
  14. NOVAID Backend
  15. Operational Status and Rollout
  16. ...and 7 more sections

Figures (6)

  • Figure 1: Framework of our proposed method of NOVAID for generating widgets for custom dashboards using LLMs
  • Figure 1: User workflow illustration: (a) landing page, (b) example prompts, (c) query input, (d) dropdown selection, (e) widget preview, and (f) dashboard output.
  • Figure 2: NOVAID Deployment and Integration with Instana
  • Figure 2: User journeys (a–c) for query "show the latency for appdata-writer service over time" where NOVAID infers the widget type as a time series chart even though not explicitly mentioned
  • Figure 3: NOVAID Frontend with query and widget preview
  • ...and 1 more figures