VGA: Vision GUI Assistant -- Minimizing Hallucinations through Image-Centric Fine-Tuning

TL;DR

VGA is introduced, a fine-tuned model designed for comprehensive GUI understanding that enhances the model's ability to extract information from images and achieves state-of-the-art results in GUI understanding tasks.

Abstract

Recent advances in Large Vision-Language Models (LVLMs) have significantly improve performance in image comprehension tasks, such as formatted charts and rich-content images. Yet, Graphical User Interface (GUI) pose a greater challenge due to their structured format and detailed textual information. Existing LVLMs often overly depend on internal knowledge and neglect image content, resulting in hallucinations and incorrect responses in GUI comprehension. To address these issues, we introduce VGA, a fine-tuned model designed for comprehensive GUI understanding. Our model aims to enhance the interpretation of visual data of GUI and reduce hallucinations. We first construct a Vision Question Answering (VQA) dataset of 63.8k high-quality examples with our propose Referent Method, which ensures the model's responses are highly depend on visual content within the image. We then design a two-stage fine-tuning method called Foundation and Advanced Comprehension (FAC) to enhance both the model's ability to extract information from image content and alignment with human intent. Experiments show that our approach enhances the model's ability to extract information from images and achieves state-of-the-art results in GUI understanding tasks. Our dataset and fine-tuning script will be released soon.

Figures (20)

• Figure 1: Data generation Method
• Figure 2: Loss convergence of model trained with foundation task and without during advanced task training.
• Figure 3: Loss convergence of model trained with mixed foundation task and advanced task data.
• Figure 4: VGA-7b-stage2
• Figure 5: VGA-7b-v1 (with FAC method)