Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BioVL-QR: Egocentric Biochemical Vision-and-Language Dataset Using Micro QR Codes

Tomohiro Nishimoto, Taichi Nishimura, Koki Yamamoto, Keisuke Shirai, Hirotaka Kameko, Yuto Haneji, Tomoya Yoshida, Keiya Kajimura, Taiyu Cui, Chihiro Nishiwaki, Eriko Daikoku, Natsuko Okuda, Fumihito Ono, Shinsuke Mori

TL;DR

BioVL-QR tackles reproducibility in biochemistry by introducing an egocentric vision–language dataset where Micro QR Codes attached to lab objects enable automated object labeling. The authors couple a Micro QR Code detector with a hand-object detector to label objects and align video events with procedural steps, then extend StepFormer with object-label embeddings via EgoVLPv2 to localize steps in instructional videos. Key contributions include the BioVL-QR dataset, the automated object labeling method, and improved step localization baselines that leverage QR-code information, achieving notable gains in MoF, precision, recall, and tIoU. This work provides a scalable approach for multimodal biochemical video understanding, potentially enhancing reproducibility and automated protocol execution in lab settings.

Abstract

This paper introduces BioVL-QR, a biochemical vision-and-language dataset comprising 23 egocentric experiment videos, corresponding protocols, and vision-and-language alignments. A major challenge in understanding biochemical videos is detecting equipment, reagents, and containers because of the cluttered environment and indistinguishable objects. Previous studies assumed manual object annotation, which is costly and time-consuming. To address the issue, we focus on Micro QR Codes. However, detecting objects using only Micro QR Codes is still difficult due to blur and occlusion caused by object manipulation. To overcome this, we propose an object labeling method combining a Micro QR Code detector with an off-the-shelf hand object detector. As an application of the method and BioVL-QR, we tackled the task of localizing the procedural steps in an instructional video. The experimental results show that using Micro QR Codes and our method improves biochemical video understanding. Data and code are available through https://nishi10mo.github.io/BioVL-QR/

BioVL-QR: Egocentric Biochemical Vision-and-Language Dataset Using Micro QR Codes

TL;DR

BioVL-QR tackles reproducibility in biochemistry by introducing an egocentric vision–language dataset where Micro QR Codes attached to lab objects enable automated object labeling. The authors couple a Micro QR Code detector with a hand-object detector to label objects and align video events with procedural steps, then extend StepFormer with object-label embeddings via EgoVLPv2 to localize steps in instructional videos. Key contributions include the BioVL-QR dataset, the automated object labeling method, and improved step localization baselines that leverage QR-code information, achieving notable gains in MoF, precision, recall, and tIoU. This work provides a scalable approach for multimodal biochemical video understanding, potentially enhancing reproducibility and automated protocol execution in lab settings.

Abstract

This paper introduces BioVL-QR, a biochemical vision-and-language dataset comprising 23 egocentric experiment videos, corresponding protocols, and vision-and-language alignments. A major challenge in understanding biochemical videos is detecting equipment, reagents, and containers because of the cluttered environment and indistinguishable objects. Previous studies assumed manual object annotation, which is costly and time-consuming. To address the issue, we focus on Micro QR Codes. However, detecting objects using only Micro QR Codes is still difficult due to blur and occlusion caused by object manipulation. To overcome this, we propose an object labeling method combining a Micro QR Code detector with an off-the-shelf hand object detector. As an application of the method and BioVL-QR, we tackled the task of localizing the procedural steps in an instructional video. The experimental results show that using Micro QR Codes and our method improves biochemical video understanding. Data and code are available through https://nishi10mo.github.io/BioVL-QR/
Paper Structure (15 sections, 4 figures, 4 tables)

This paper contains 15 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. BioVL-QR Dataset
  4. Video Recording
  5. Alignment Annotation
  6. Statistics
  7. Object labeling based on QR Codes
  8. Method
  9. Evaluation
  10. Application: Step Localization
  11. Problem Formulation
  12. Our Approach
  13. Experimental Settings
  14. Results
  15. Conclusion

Figures (4)

  • Figure 1: Overview of BioVL-QR containing experiment videos and corresponding protocols. We label objects appearing in the videos using Micro QR Codes.
  • Figure 2: Overview of our object labeling process, which consists of two steps: object dictionary construction (step 1) and dictionary-based object linking (step 2). Step 1 constructs an object dictionary by associating object names obtained from Micro QR Codes with their visual feature vectors. Step 2 links hand-interacted objects in the video to their names using an object dictionary.
  • Figure 3: Success case (left) and failure case (right) of the Micro QR Code detection.
  • Figure 4: Sample of step localization for DNA extraction.