OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments

TL;DR

OpenMarcie is the biggest multimodal dataset designed for human action monitoring in manufacturing environments, featuring over 37 hours of egocentric and exocentric, multimodal, and multipositional data, featuring eight distinct data types and more than 200 independent information channels.

Abstract

Smart factories use advanced technologies to optimize production and increase efficiency. To this end, the recognition of worker activity allows for accurate quantification of performance metrics, improving efficiency holistically while contributing to worker safety. OpenMarcie is, to the best of our knowledge, the biggest multimodal dataset designed for human action monitoring in manufacturing environments. It includes data from wearables sensing modalities and cameras distributed in the surroundings. The dataset is structured around two experimental settings, involving a total of 36 participants. In the first setting, twelve participants perform a bicycle assembly and disassembly task under semi-realistic conditions without a fixed protocol, promoting divergent and goal-oriented problem-solving. The second experiment involves twenty-five volunteers (24 valid data) engaged in a 3D printer assembly task, with the 3D printer manufacturer's instructions provided to guide the volunteers in acquiring procedural knowledge. This setting also includes sequential collaborative assembly, where participants assess and correct each other's progress, reflecting real-world manufacturing dynamics. OpenMarcie includes over 37 hours of egocentric and exocentric, multimodal, and multipositional data, featuring eight distinct data types and more than 200 independent information channels. The dataset is benchmarked across three human activity recognition tasks: activity classification, open vocabulary captioning, and cross-modal alignment.

Figures (12)

• Figure 1: OpenMarcie is an open dataset for multimodal action recognition in industrial environments. Top Its applications include industrial process optimization, human activity understanding, and context awareness. Middle two assembly scenarios represent ad-hoc goal-oriented action and a procedural scenario to promote natural knowledge acquisition. Bottom multimodal data is collected with more than 200 independent channels, and a validation with three different benchmarks is done.
• Figure 2: Experiment room setting with example views of the exocentric RGBD cameras.
• Figure 3: Participant wearable setup and sensor signals. (a) Participant in the ad-hoc Scenario: bicycle assembly. (b) Participant in the procedural Scenario: 3D printer assembly. Wrist-mounted ZEDs provided only IMU and barometer data (stereo cameras disabled), while head and chest cameras captured visual streams. Sensor placements were adapted to the ergonomics and task demands of each scenario.
• Figure 4: Egocentric and exocentric views of activity examples, accompanied by annotations (see \ref{['sec:SectionAnnotation']}). These include both soft and hard labels for two scenarios: (a) the ad-hoc bicycle assembly/disassembly task, and (b) the procedural 3D printer construction task.
• Figure 5: Participants statistic. Top left Height distribution ranging from 150 to 193 cm. Top right Age distribution spanning 22 to 37 years. Bottom left Self-reported experience levels in assembly tasks, categorized as beginner, intermediate, and advanced. Bottom right academic level of participants, with engineers representing the majority and managerial roles accounting for only 3% of the sample.