Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Broadening Access to Simulations for End-Users via Large Language Models: Challenges and Opportunities

Philippe J. Giabbanelli, Jose J. Padilla, Ameeta Agrawal

TL;DR

This paper tackles enabling non-expert end-users to perform what-if analyses on simulations via Large Language Models (LLMs). It proposes an end-to-end framework where an LLM mediates between user queries and simulation engines through three phases: model alignment, query reformulation with clarifying questions, and contextualized, ethically-grounded result presentation. It outlines a concrete research agenda spanning modeling & simulation, information retrieval, and ethics, and details evaluation metrics for faithfulness, bias, explanations, and user satisfaction. The work aims to democratize access to complex simulations while highlighting challenges in model composition, interpretability, cost, and grounding outputs in validated domain knowledge.

Abstract

Large Language Models (LLMs) are becoming ubiquitous to create intelligent virtual assistants that assist users in interacting with a system, as exemplified in marketing. Although LLMs have been discussed in Modeling & Simulation (M&S), the community has focused on generating code or explaining results. We examine the possibility of using LLMs to broaden access to simulations, by enabling non-simulation end-users to ask what-if questions in everyday language. Specifically, we discuss the opportunities and challenges in designing such an end-to-end system, divided into three broad phases. First, assuming the general case in which several simulation models are available, textual queries are mapped to the most relevant model. Second, if a mapping cannot be found, the query can be automatically reformulated and clarifying questions can be generated. Finally, simulation results are produced and contextualized for decision-making. Our vision for such system articulates long-term research opportunities spanning M&S, LLMs, information retrieval, and ethics.

Broadening Access to Simulations for End-Users via Large Language Models: Challenges and Opportunities

TL;DR

This paper tackles enabling non-expert end-users to perform what-if analyses on simulations via Large Language Models (LLMs). It proposes an end-to-end framework where an LLM mediates between user queries and simulation engines through three phases: model alignment, query reformulation with clarifying questions, and contextualized, ethically-grounded result presentation. It outlines a concrete research agenda spanning modeling & simulation, information retrieval, and ethics, and details evaluation metrics for faithfulness, bias, explanations, and user satisfaction. The work aims to democratize access to complex simulations while highlighting challenges in model composition, interpretability, cost, and grounding outputs in validated domain knowledge.

Abstract

Large Language Models (LLMs) are becoming ubiquitous to create intelligent virtual assistants that assist users in interacting with a system, as exemplified in marketing. Although LLMs have been discussed in Modeling & Simulation (M&S), the community has focused on generating code or explaining results. We examine the possibility of using LLMs to broaden access to simulations, by enabling non-simulation end-users to ask what-if questions in everyday language. Specifically, we discuss the opportunities and challenges in designing such an end-to-end system, divided into three broad phases. First, assuming the general case in which several simulation models are available, textual queries are mapped to the most relevant model. Second, if a mapping cannot be found, the query can be automatically reformulated and clarifying questions can be generated. Finally, simulation results are produced and contextualized for decision-making. Our vision for such system articulates long-term research opportunities spanning M&S, LLMs, information retrieval, and ethics.
Paper Structure (7 sections, 4 figures)

This paper contains 7 sections, 4 figures.

Table of Contents

  1. INTRODUCTION
  2. Conceptual Framework
  3. What is a Question? Aligning Constructs with Model Parameters
  4. Can (A)I Have a Word with You? Clarifying Questions from the LLM
  5. Responsible Responses: Contextualizing Answers and Promoting Ethics
  6. What is a Good System? Measuring Faithfulness, Biases, and Explanations
  7. Discussion and Conclusions

Figures (4)

  • Figure 1: Overview of our conceptual framework including textual questions, alignment with the right model (or request for clarification), and translation of structured simulation results into text. We envision the system as composed of three phases (indicated by colors), detailed in dedicated subsections 3 to 5.
  • Figure 2: The user starts by formulating a what-if question. The extraction step identifies and categorizes key constructs. The user constructs are then matched with semantically equivalent parameters from available models. As a result, we select an acceptable model that operates in a context including the user's interests, and that is sufficiently informative as its outputs directly feed into the desired measurement outcomes.
  • Figure 3: A user's query may need to be rewritten so that it becomes self-contained, then it can be rewritten to better align with simulation models. If the alignment is satisfactory, the simulation can proceed. Otherwise, the system generates a question and interprets the user's answer through the same rewriting process.
  • Figure 4: Running a simulation requires mapping user constructs to model parameters as well as converting linguistic modifiers (e.g., 'modest increase') into quantitative values for each parameter's domain via fuzzy logic. Several simulation runs can then be performed, but a comprehensive interpretation of the structured results (e.g., CSV or HDF5) would require access to the model's context and internal structure.