CodeSCAN: ScreenCast ANalysis for Video Programming Tutorials

TL;DR

CodeSCAN introduces a large-scale, richly annotated dataset of 12,000 Visual Studio Code screenshots spanning $24$ programming languages, $90$ themes, and $25$ fonts to enable source-code extraction from coding screencasts. It benchmarks an object-detection baseline (Mask R-CNN) for IDE-element localization, a Pix2PixHD-based binarization approach, and five OCR engines to study how binarization and image quality affect text recognition. The findings show strong localization for several IDE components, meaningful gains from binarization for OCR in some engines, and varying robustness across input modalities, highlighting the dataset's utility for improving search and hands-on exploration in screencasts. The work lays the groundwork for end-to-end code retrieval from tutorials and points to future enhancements such as full video pipelines, tracking and synchronization, and coding-grid-oriented advancements. Overall, CodeSCAN addresses a critical gap in screencast analysis by enabling precise code-aware retrieval and interactive code experimentation from video tutorials.

Abstract

Programming tutorials in the form of coding screencasts play a crucial role in programming education, serving both novices and experienced developers. However, the video format of these tutorials presents a challenge due to the difficulty of searching for and within videos. Addressing the absence of large-scale and diverse datasets for screencast analysis, we introduce the CodeSCAN dataset. It comprises 12,000 screenshots captured from the Visual Studio Code environment during development, featuring 24 programming languages, 25 fonts, and over 90 distinct themes, in addition to diverse layout changes and realistic user interactions. Moreover, we conduct detailed quantitative and qualitative evaluations to benchmark the performance of Integrated Development Environment (IDE) element detection, color-to-black-and-white conversion, and Optical Character Recognition (OCR). We hope that our contributions facilitate more research in coding screencast analysis, and we make the source code for creating the dataset and the benchmark publicly available on this website.

Figures (5)

• Figure 1: Source code extraction pipeline: Given an image from a coding screencast (1), the IDE elements need to be detected (2). This is followed by an optional binarization step (3) which converts the color image into a black-and-white image, and finally OCR is applied (4) to read out the source code.
• Figure 2: Example of the different available annotation and data types showing the Github repository https://github.com/rrousselGit/flutter_hooks with file https://github.com/rrousselGit/flutter_hooks/blob/master/packages/flutter_hooks/test/use_automatic_keep_alive_test.dart. Note that we omit line, word and character annotations in (b) for better readability.
• Figure 3: Qualitative examples of the binarization showing the input image on the left and the predicted binarization on the right.
• Figure 4: Challenging qualitative examples of the binarization showing the input image on the left and the predicted binarization on the right.
• Figure 5: Quantitative evaluation of OCR performance for different image qualities.