From Pen to Pixel: Translating Hand-Drawn Plots into Graphical APIs via a Novel Benchmark and Efficient Adapter

Abstract

As plots play a critical role in modern data visualization and analysis, Plot2API is launched to help non-experts and beginners create their desired plots by directly recommending graphical APIs from reference plot images by neural networks. However, previous works on Plot2API have primarily focused on the recommendation for standard plot images, while overlooking the hand-drawn plot images that are more accessible to non-experts and beginners. To make matters worse, both Plot2API models trained on standard plot images and powerful multi-modal large language models struggle to effectively recommend APIs for hand-drawn plot images due to the domain gap and lack of expertise. To facilitate non-experts and beginners, we introduce a hand-drawn plot dataset named HDpy-13 to improve the performance of graphical API recommendations for hand-drawn plot images. Additionally, to alleviate the considerable strain of parameter growth and computational resource costs arising from multi-domain and multi-language challenges in Plot2API, we propose Plot-Adapter that allows for the training and storage of separate adapters rather than requiring an entire model for each language and domain. In particular, Plot-Adapter incorporates a lightweight CNN block to improve the ability to capture local features and implements projection matrix sharing to reduce the number of fine-tuning parameters further. Experimental results demonstrate both the effectiveness of HDpy-13 and the efficiency of Plot-Adapter.

Figures (6)

• Figure 1: (a) Example of API recommendations by large language models. (b) API recommendation results based on hand-drawn plot images using ResNet-50 and EfficientNet models trained on a standard plot dataset.
• Figure 2: Accuracy comparison in the task of API recommendation based on hand-drawn plot images: orange denotes models trained on our dataset, blue denotes models trained on a standard plot dataset.
• Figure 3: Dataset distribution of each API in HDpy-13.
• Figure 4: Sample hand-drawn plot images from HDpy-13 dataset showcasing various APIs and styles.
• Figure 5: (a) Example of distinction relying on local features. (b) Example of distinction relying on global features.