Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

mForms : Multimodal Form-Filling with Question Answering

Larry Heck, Simon Heck, Anirudh Sundar

TL;DR

This work reframes GUI form-filling as a multimodal question answering task, translating visual UI elements into natural-language questions and using a pre-trained extractive QA model to extract answers from user utterances. A CV-based semantic labeling pipeline provides rich visual meaning for 25 UI categories, 197 text buttons, and 99 icons, enabling accurate, zero-shot form filling. The authors introduce the mForms dataset and an ATIS-form filling extension, showing strong performance, especially in sparse data regimes, and achieving a state-of-the-art F1 of 0.97 on ATIS with only about 1/10th of the full training data. They further demonstrate gains from multi-task training across domains and discuss directions to scale to more GUI screens and to leverage generative models in future work.

Abstract

This paper presents a new approach to form-filling by reformulating the task as multimodal natural language Question Answering (QA). The reformulation is achieved by first translating the elements on the GUI form (text fields, buttons, icons, etc.) to natural language questions, where these questions capture the element's multimodal semantics. After a match is determined between the form element (Question) and the user utterance (Answer), the form element is filled through a pre-trained extractive QA system. By leveraging pre-trained QA models and not requiring form-specific training, this approach to form-filling is zero-shot. The paper also presents an approach to further refine the form-filling by using multi-task training to incorporate a potentially large number of successive tasks. Finally, the paper introduces a multimodal natural language form-filling dataset Multimodal Forms (mForms), as well as a multimodal extension of the popular ATIS dataset to support future research and experimentation. Results show the new approach not only maintains robust accuracy for sparse training conditions but achieves state-of-the-art F1 of 0.97 on ATIS with approximately 1/10th of the training data.

mForms : Multimodal Form-Filling with Question Answering

TL;DR

This work reframes GUI form-filling as a multimodal question answering task, translating visual UI elements into natural-language questions and using a pre-trained extractive QA model to extract answers from user utterances. A CV-based semantic labeling pipeline provides rich visual meaning for 25 UI categories, 197 text buttons, and 99 icons, enabling accurate, zero-shot form filling. The authors introduce the mForms dataset and an ATIS-form filling extension, showing strong performance, especially in sparse data regimes, and achieving a state-of-the-art F1 of 0.97 on ATIS with only about 1/10th of the full training data. They further demonstrate gains from multi-task training across domains and discuss directions to scale to more GUI screens and to leverage generative models in future work.

Abstract

This paper presents a new approach to form-filling by reformulating the task as multimodal natural language Question Answering (QA). The reformulation is achieved by first translating the elements on the GUI form (text fields, buttons, icons, etc.) to natural language questions, where these questions capture the element's multimodal semantics. After a match is determined between the form element (Question) and the user utterance (Answer), the form element is filled through a pre-trained extractive QA system. By leveraging pre-trained QA models and not requiring form-specific training, this approach to form-filling is zero-shot. The paper also presents an approach to further refine the form-filling by using multi-task training to incorporate a potentially large number of successive tasks. Finally, the paper introduces a multimodal natural language form-filling dataset Multimodal Forms (mForms), as well as a multimodal extension of the popular ATIS dataset to support future research and experimentation. Results show the new approach not only maintains robust accuracy for sparse training conditions but achieves state-of-the-art F1 of 0.97 on ATIS with approximately 1/10th of the training data.

Paper Structure

This paper contains 10 sections, 9 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Approach
  3. Multimodal form-filling
  4. Single- and Multi-Task Training
  5. Experiments
  6. Setup
  7. Multimodal Forms Dataset
  8. Simulated ATIS Form-Filling Dataset
  9. Conclusions and Future Work
  10. Appendix

Figures (9)

  • Figure 1: An example semantic representation with domain, intent, and semantic slot annotations.
  • Figure 2: Semantically annotated mobile Graphical User Interface (GUI) using computer vision to identify 25 UI component categories, 197 text button concepts, and 99 icon classes (Liu et al. 2018)
  • Figure 3: The mForms pipeline. The rule template uses the semantified UI to trigger a question template. The visual information of the GUI element drives the generation of the actual question. Then, using the user's request as evidence, the questions are answered to fill the form with the appropriate information.
  • Figure 4: mForms as QA Approach
  • Figure 5: Simulated ATIS form-filling fields translated to Questions
  • ...and 4 more figures