Automating UI Optimization through Multi-Agentic Reasoning

TL;DR

AutoOptimization introduces a sequential multi-agent framework that automates UI adaptation by translating verbal user instructions into a parameterized multi-objective optimization problem, generating Pareto-optimal UI layouts, and validating the best design against user intent. By employing vision-language model agents for ambiguity detection, problem configuration, and post-optimization validation, the approach reduces manual design effort and enables dynamic personalization in Mixed Reality interfaces. The framework is evaluated through an MR use case, data-collection studies, and end-to-end user experiments, demonstrating strong alignment with user instructions, lower physical workload, and comparable satisfaction to manual methods. The work advances adaptive UIs by combining grounding, reasoning, and optimization in a fully automated pipeline, while identifying avenues for future improvements such as direct manipulation integration and richer objective-function design.

Abstract

We present AutoOptimization, a novel multi-objective optimization framework for adapting user interfaces. From a user's verbal preferences for changing a UI, our framework guides a prioritization-based Pareto frontier search over candidate layouts. It selects suitable objective functions for UI placement while simultaneously parameterizing them according to the user's instructions to define the optimization problem. A solver then generates a series of optimal UI layouts, which our framework validates against the user's instructions to adapt the UI with the final solution. Our approach thus overcomes the previous need for manual inspection of layouts and the use of population averages for objective parameters. We integrate multiple agents sequentially within our framework, enabling the system to leverage their reasoning capabilities to interpret user preferences, configure the optimization problem, and validate optimization outcomes.

Figures (11)

• Figure 1: Examples of scenarios from the data collection survey, showcasing virtual widgets and the simulated environment. Participants are asked to imagine themselves in the given background and to provide instructions to an intelligent assistant to arrange the virtual widgets that completes the specified task.
• Figure 2: Illustrative demonstration of our Configuration component in a Mixed Reality context. Left: The user aims to reply to messages using information from virtual widgets while avoiding overlaying physical monitors. The Configuration module in our framework selects the objective terms that relate to the user's expressed preferences, such that the Optimizer produces layouts that keep all widgets within the user's field of view, the messenger anchored to the desk for haptic feedback, and the monitors visible and unoccluded. Right: The user plans a trip using a map and books a hotel while keeping the whiteboard sketch unobstructed. Our framework selects and displays the widgets relevant to the user's instructions on the clean whiteboard.
• Figure 3: Classification accuracy of few-shot optimized VLMs with different number of participants and scenarios in training.
• Figure 4: Examples of scenarios from the layout comparison survey, each displaying four candidate solutions (a–d), the prompt it was generated with, and the design preferred by participants of the survey. The scenario numbers and candidate solutions (a–d) correspond to the respective scenarios and categorical values (A–D) displayed in Figure \ref{['fig:comparison_results']}.
• Figure 5: Categorical distributions of the MR layout voted as best aligned by vlms and ptps in each scenario.