Second Place Solution of WSDM2023 Toloka Visual Question Answering Challenge

TL;DR

This work tackles the Toloka Visual Question Answering Challenge by recasting VQA as a visual grounding task and building a three-stage pipeline around the OFA multimodal model. It first generates a large synthetic dataset for coarse tuning to foster generalizable semantic understanding, then fine-tunes on the competition data using multiple templates to produce bounding-box outputs, and finally applies IoU-based post-processing to refine predictions with object detector boxes and model ensembles. The approach achieves 76.342 on the private test, ranking second, and demonstrates that synthetic data, task framing, and precise post-processing can substantially boost visual grounding performance in VQA-related tasks. The methods offer a practical path for robust, bounding-box–driven VQA in multimodal systems and underscore the value of staged training plus refinement for cross-modal reasoning tasks.

Abstract

In this paper, we present our solution for the WSDM2023 Toloka Visual Question Answering Challenge. Inspired by the application of multimodal pre-trained models to various downstream tasks(e.g., visual question answering, visual grounding, and cross-modal retrieval), we approached this competition as a visual grounding task, where the input is an image and a question, guiding the model to answer the question and display the answer as a bounding box on the image. We designed a three-stage solution for this task. Specifically, we used the visual-language pre-trained model OFA as the foundation. In the first stage, we constructed a large-scale synthetic dataset similar to the competition dataset and coarse-tuned the model to learn generalized semantic information. In the second stage, we treated the competition task as a visual grounding task, loaded the weights from the previous stage, and continued to fine-tune the model on the competition dataset, transferring the semantic information learned in the first stage to the competition task. Finally, we designed a bounding box matching and replacing post-processing strategy to correct the model's prediction results. Our team achieved a score of 76.342 on the final leaderboard, ranking second.

Figures (3)

• Figure 1: Coarse Tuning Stage.
• Figure 2: Fine Tuning Stage.
• Figure 3: Postprocessing Stage.