Multi-modal video data-pipelines for machine learning with minimal human supervision

TL;DR

This paper tackles scalable multi-modal machine learning by focusing on automated data acquisition and processing to produce rich multi-modal video representations with minimal human supervision. It introduces the Video Representations Extractor (VRE) data-pipeline and the PHG-MAE framework, including lightweight distilled variants, and demonstrates real-time semantic segmentation and depth estimation on commodity hardware. A key contribution is the Dronescapes2 dataset, generated fully automatically with VRE, and the demonstration that small-distilled models can approach the performance of far larger baselines. The work provides an open-source toolkit and practical deployment insights for on-device and cloud-based real-time multi-modal ML.

Abstract

The real-world is inherently multi-modal at its core. Our tools observe and take snapshots of it, in digital form, such as videos or sounds, however much of it is lost. Similarly for actions and information passing between humans, languages are used as a written form of communication. Traditionally, Machine Learning models have been unimodal (i.e. rgb -> semantic or text -> sentiment_class). Recent trends go towards bi-modality, where images and text are learned together, however, in order to truly understand the world, we need to integrate all these independent modalities. In this work we try to combine as many visual modalities as we can using little to no human supervision. In order to do this, we use pre-trained experts and procedural combinations between them on top of raw videos using a fully autonomous data-pipeline, which we also open-source. We then make use of PHG-MAE, a model specifically designed to leverage multi-modal data. We show that this model which was efficiently distilled into a low-parameter (<1M) can have competitive results compared to models of ~300M parameters. We deploy this model and analyze the use-case of real-time semantic segmentation from handheld devices or webcams on commodity hardware. Finally, we deploy other off-the-shelf models using the same framework, such as DPT for near real-time depth estimation.

Figures (12)

• Figure 1: High-level overview of an end-to-end machine learning system: from raw data and data processing, to training and optimizing models and lastly by deploying it to interact and control a real hardware autonomously with intelligent actions.
• Figure 2: VRE showcase. We present six exported representations on top of the RGB frame. The first two are pre-trained experts (DPT ranftl2021vision and Marigold ke2024repurposing). Next, we derive two camera normals representations using a SVD-based algorithm hartley2003multiple. Lastly, we derive safe-landing areas by thresholding the camera normals maps like in the newly introduced Dronescapes2 mihaicristian2025probabilistichypergraphsusingmultiple dataset.
• Figure 3: VRE processing strategies. Left: the standard batched strategy. We split the frames in batches and then each batch is passed through the algorithm of the representation, followed by a step where the results are stored on the disk. Right: the streaming strategy. In this mode the input is a live video stream (webcam, camera phone etc.) Each frame, or nearby ones (if needed), are processed sequentially by all representations.
• Figure 4: VRE streaming architecture. We read frame by frame from the source (i.e. drone camera), process it on the VRE streaming client (i.e. cloud or local GPU), analyze the results and pass the actions to the target (i.e. drone controller). Notably, all these components can live on the same machine but they can also communicate through the network.
• Figure 5: VRE multi-gpu batching strategies. Strategy 1: Slice the video in multiple independent chunks. Strategy 2: split the video's representations in sub-groups.