Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support

Haowen Xu, Jose Tupayachi, Xiao-Ying Yu

TL;DR

This paper tackles the difficulty of building scalable geospatial risk dashboards from large environmental datasets by introducing a context-aware, knowledge-guided AI framework. It combines context-aware visual prompting (CAVP), an ontological knowledge graph, and a self-validating AI agent (with Pass@k and semantic metrics) to automatically generate multi-page React MVVM dashboards from UI wireframes and data sources. The approach leverages retrieval-augmented generation, structured prompts, and automated repair to improve code reliability and maintainability, demonstrated in a meteorological dashboard case study. Results show improved syntactic validity and functional correctness over baselines, with robust validation pipelines enabling rapid deployment for decision support in environmental risk analysis. Overall, the framework lowers the barrier for domain scientists to produce interactive, validated GIS dashboards, enhancing risk assessment and policy-making workflows.

Abstract

The development of web-based geospatial dashboards for risk analysis and decision support is often challenged by the difficulty in visualization of big, multi-dimensional environmental data, implementation complexity, and limited automation. We introduce a generative AI framework that harnesses Large Language Models (LLMs) to automate the creation of interactive geospatial dashboards from user-defined inputs including UI wireframes, requirements, and data sources. By incorporating a structured knowledge graph, the workflow embeds domain knowledge into the generation process and enable accurate and context-aware code completions. A key component of our approach is the Context-Aware Visual Prompting (CAVP) mechanism, which extracts encodes and interface semantics from visual layouts to guide LLM driven generation of codes. The new framework also integrates a self-validation mechanism that uses an agent-based LLM and Pass@k evaluation alongside semantic metrics to assure output reliability. Dashboard snippets are paired with data visualization codebases and ontological representations, enabling a pipeline that produces scalable React-based completions using the MVVM architectural pattern. Our results demonstrate improved performance over baseline approaches and expanded functionality over third party platforms, while incorporating multi-page, fully functional interfaces. We successfully developed a framework to implement LLMs, demonstrated the pipeline for automated code generation, deployment, and performed chain-of-thought AI agents in self-validation. This integrative approach is guided by structured knowledge and visual prompts, providing an innovative geospatial solution in enhancing risk analysis and decision making.

Context-Aware Visual Prompting: Automating Geospatial Web Dashboards with Large Language Models and Agent Self-Validation for Decision Support

TL;DR

This paper tackles the difficulty of building scalable geospatial risk dashboards from large environmental datasets by introducing a context-aware, knowledge-guided AI framework. It combines context-aware visual prompting (CAVP), an ontological knowledge graph, and a self-validating AI agent (with Pass@k and semantic metrics) to automatically generate multi-page React MVVM dashboards from UI wireframes and data sources. The approach leverages retrieval-augmented generation, structured prompts, and automated repair to improve code reliability and maintainability, demonstrated in a meteorological dashboard case study. Results show improved syntactic validity and functional correctness over baselines, with robust validation pipelines enabling rapid deployment for decision support in environmental risk analysis. Overall, the framework lowers the barrier for domain scientists to produce interactive, validated GIS dashboards, enhancing risk assessment and policy-making workflows.

Abstract

The development of web-based geospatial dashboards for risk analysis and decision support is often challenged by the difficulty in visualization of big, multi-dimensional environmental data, implementation complexity, and limited automation. We introduce a generative AI framework that harnesses Large Language Models (LLMs) to automate the creation of interactive geospatial dashboards from user-defined inputs including UI wireframes, requirements, and data sources. By incorporating a structured knowledge graph, the workflow embeds domain knowledge into the generation process and enable accurate and context-aware code completions. A key component of our approach is the Context-Aware Visual Prompting (CAVP) mechanism, which extracts encodes and interface semantics from visual layouts to guide LLM driven generation of codes. The new framework also integrates a self-validation mechanism that uses an agent-based LLM and Pass@k evaluation alongside semantic metrics to assure output reliability. Dashboard snippets are paired with data visualization codebases and ontological representations, enabling a pipeline that produces scalable React-based completions using the MVVM architectural pattern. Our results demonstrate improved performance over baseline approaches and expanded functionality over third party platforms, while incorporating multi-page, fully functional interfaces. We successfully developed a framework to implement LLMs, demonstrated the pipeline for automated code generation, deployment, and performed chain-of-thought AI agents in self-validation. This integrative approach is guided by structured knowledge and visual prompts, providing an innovative geospatial solution in enhancing risk analysis and decision making.

Paper Structure

This paper contains 29 sections, 5 equations, 5 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Current Implementations & Literature Review
  3. A Review of LLM-Based Approaches in Software Development
  4. Evaluating LLMs for Software Development
  5. Front-end App Development from UI Prototypes
  6. Limitations and Knowledge Gap
  7. Methodology
  8. Base (Difficulty: 1)
  9. homePage (Difficulty: 2)
  10. Geovisualization (Difficulty: 3)
  11. Context Aware Visual Prompting
  12. Ontological Knowledge and Codebase
  13. Repair Agent Using Functional Output
  14. Implementation & Evaluation
  15. Ablation Study
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Knowledge-driven framework and annotated wireframes transformation to code for automated platform development and temporal and spatial data visualization.
  • Figure 2: Structured knowledge representations for converting plain-language annotations from UI wireframes into structured prompts with technical terminology.
  • Figure 3: Generation and repair procedure. This process is executed after the ontological knowledge-augmented pipeline produces components aligned with framework best practices.
  • Figure 4: Visual comparison between the proposed CAVP method and the baseline ScreenShots2Code. CAVP uses mockups, knowledge graphs, and prompt-based embeddings for semantically-aware UI generation, whereas ScreenShots2Code depends on static screenshots and file page structures.
  • Figure :