AutoGameUI: Constructing High-Fidelity GameUI via Multimodal Correspondence Matching

TL;DR

This work tackles the challenge of aligning visual UI design with functional UX design in game UI development. It introduces AutoGameUI, a two-stage multimodal framework that learns rich UI/UX representations and performs hierarchical, group-based correspondence matching guided by constrained optimization, followed by an interactive web tool and a universal data protocol for cross-platform deployment. The authors validate their approach on the newly collected GAMEUI dataset and the public RICO dataset, demonstrating improved matching accuracy, visual fidelity, and deployment-time efficiency, including a notable 3x speed-up in mobile game workflows. The contributions include the GAMEUI dataset, the grouped cross-attention and constrained integer programming pipeline, and an interactive tool with universal protocol support, collectively enabling high-fidelity, scalable GameUI construction across platforms.

Abstract

Game UI development is essential to the game industry. However, the traditional workflow requires substantial manual effort to integrate pairwise UI and UX designs into a cohesive game user interface (GameUI). The inconsistency between the aesthetic UI design and the functional UX design typically results in mismatches and inefficiencies. To address the issue, we present an automatic system, AutoGameUI, for efficiently and accurately constructing GameUI. The system centers on a two-stage multimodal learning pipeline to obtain the optimal correspondences between UI and UX designs. The first stage learns the comprehensive representations of UI and UX designs from multimodal perspectives. The second stage incorporates grouped cross-attention modules with constrained integer programming to estimate the optimal correspondences through top-down hierarchical matching. The optimal correspondences enable the automatic GameUI construction. We create the GAMEUI dataset, comprising pairwise UI and UX designs from real-world games, to train and validate the proposed method. Besides, an interactive web tool is implemented to ensure high-fidelity effects and facilitate human-in-the-loop construction. Extensive experiments on the GAMEUI and RICO datasets demonstrate the effectiveness of our system in maintaining consistency between the constructed GameUI and the original designs. When deployed in the workflow of several mobile games, AutoGameUI achieves a 3$\times$ improvement in time efficiency, conveying significant practical value for game UI development.

Figures (8)

• Figure 1: Comparison between the traditional manual workflow and AutoGameUI in game UI development.
• Figure 2: Overview of the AutoGameUI system. Left: multimodal correspondence matching between pairwise UI and UX designs. The first stage focuses on acquiring the multimodal representation of matchable nodes in UI and UX designs. The second stage aims to estimate the optimal correspondences between matchable nodes. Right: interactive showcase in Figma. The user launches the web tool, initiates automatic matching, and manually refines results with matching probabilities.
• Figure 3: Illustration of the grouped cross-attention module.
• Figure 4: Illustration of the GAMEUI dataset. (a) Distribution of semantic label. (b) Comparison with the RICO dataset. (c) Examples of correspondence annotation.
• Figure 5: Qualitative results on the GAMEUI dataset.