LivePyxel: Accelerating image annotations with a Python-integrated webcam live streaming

TL;DR

LivePyxel addresses the annotation bottleneck in scientific imaging by delivering an open-source Python GUI that enables on-site, real-time image labeling through integrated video devices. It combines Bézier spline-based contours, binary masks, and a multi-layer, non-destructive editing framework built on OpenCV and Qt to produce precise, easily exportable masks and per-instance annotations. The paper demonstrates two data-generation pipelines: one using real-world water-microbial images for U-Net segmentation and another using controlled-background binary masks to synthesize large datasets, achieving competitive performance with established tools. By enabling on-site labeling, device-agnostic input, and seamless integration with downstream vision tasks, LivePyxel accelerates AI model deployment in experimental workflows and broadens participation from domain experts with limited coding expertise.

Abstract

The lack of flexible annotation tools has hindered the deployment of AI models in some scientific areas. Most existing image annotation software requires users to upload a precollected dataset, which limits support for on-demand pipelines and introduces unnecessary steps to acquire images. This constraint is particularly problematic in laboratory environments, where on-site data acquisition from instruments such as microscopes is increasingly common. In this work, we introduce \texttt{LivePixel}, a Python-based graphical user interface that integrates with imaging systems, such as webcams, microscopes, and others, to enable on-site image annotation. LivePyxel is designed to be easy to use through a simple interface that allows users to precisely delimit areas for annotation using tools commonly found in commercial graphics editing software. Of particular interest is the availability of Bézier splines and binary masks, and the software's capacity to work with non-destructive layers that enable high-performance editing. LivePyxel also integrates a wide compatibility across video devices, and it's optimized for object detection operations via the use of OpenCV in combination with high-performance libraries designed to handle matrix and linear algebra operations via Numpy effectively. LivePyxel facilitates seamless data collection and labeling, accelerating the development of AI models in experimental workflows. LivePyxel is freely available at https://github.com/UGarCil/LivePyxel

Figures (10)

• Figure 1: Overview of the LivePyxel architecture, illustrating the integration of webcam and image inputs with OpenCV for on-site image annotation. The system supports input from live microscopy or camera feeds as well as pre-existing datasets, processes them through the LivePyxel interface, and outputs both annotated images and segmentation masks for storage or further analysis by a vision model.
• Figure 2: Preview of the LivePyxel graphical user interface (GUI) with its main components labeled: (A) sliders for adjusting mask opacity and binary mask threshold, (B) labels panel for managing annotation categories, (C) annotation panel with main control buttons, (D) toolbar for selecting drawing and editing tools, (E) central canvas where objects are annotated on microscopy images in real time, and (F) navigation controls for browsing through image frames or dataset entries.
• Figure 3: Comparison between polygon (panels A and C) and Bézier-splines (panels B and D) in the delimitation of boundaries of a tardigrade's body. One Bézier unit is defined by three points. ptA and ptB constitute the start and end of the unit, whereas ptG serves as a point of gravity to pull the line away from the user's cursor, forcing it to become a curve.
• Figure 4: Illustration of the multi-layer annotation process. Each annotation is stored as a separate layer within a stack, which is subsequently merged into a single raster image from bottom to top upon saving.
• Figure 5: The F1 scores achieved by a U-Net model, highlighting the performance across the eight different microorganisms and the background. The U-Net was initialized with the VGG-19 weights. For more details regarding the training, see the main text.