Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GAZE:Governance-Aware pre-annotation for Zero-shot World Model Environments

Leela Krishna, Mengyang Zhao, Saicharithreddy Pasula, Harshit Rajgarhia, Abhishek Mukherji

TL;DR

GAZE tackles the bottleneck of assembling large, multimodal, labeled datasets for robust world-model training by introducing a governance-first pre-annotation pipeline. It automates multi-task pre-annotation (scene understanding, object tracking, audio diarization/transcription with PII, age/NSFW assessment, motion analysis) and fuses these signals into an explainable timeline to enable review-by-exception, all while enforcing privacy and chain-of-custody. The study reports substantial efficiency gains, notably ~19 minutes saved per review hour and >80% reduction in review volume for long-form videos, with documented gains in label density and consistency. While costs and model limitations remain, the approach provides a scalable blueprint for generating high-quality, privacy-aware, cross-modal data streams suitable for cross-modal dynamics learning and action-conditioned prediction in world-model environments.

Abstract

Training robust world models requires large-scale, precisely labeled multimodal datasets, a process historically bottlenecked by slow and expensive manual annotation. We present a production-tested GAZE pipeline that automates the conversion of raw, long-form video into rich, task-ready supervision for world-model training. Our system (i) normalizes proprietary 360-degree formats into standard views and shards them for parallel processing; (ii) applies a suite of AI models (scene understanding, object tracking, audio transcription, PII/NSFW/minor detection) for dense, multimodal pre-annotation; and (iii) consolidates signals into a structured output specification for rapid human validation. The GAZE workflow demonstrably yields efficiency gains (~19 minutes saved per review hour) and reduces human review volume by >80% through conservative auto-skipping of low-salience segments. By increasing label density and consistency while integrating privacy safeguards and chain-of-custody metadata, our method generates high-fidelity, privacy-aware datasets directly consumable for learning cross-modal dynamics and action-conditioned prediction. We detail our orchestration, model choices, and data dictionary to provide a scalable blueprint for generating high-quality world model training data without sacrificing throughput or governance.

GAZE:Governance-Aware pre-annotation for Zero-shot World Model Environments

TL;DR

GAZE tackles the bottleneck of assembling large, multimodal, labeled datasets for robust world-model training by introducing a governance-first pre-annotation pipeline. It automates multi-task pre-annotation (scene understanding, object tracking, audio diarization/transcription with PII, age/NSFW assessment, motion analysis) and fuses these signals into an explainable timeline to enable review-by-exception, all while enforcing privacy and chain-of-custody. The study reports substantial efficiency gains, notably ~19 minutes saved per review hour and >80% reduction in review volume for long-form videos, with documented gains in label density and consistency. While costs and model limitations remain, the approach provides a scalable blueprint for generating high-quality, privacy-aware, cross-modal data streams suitable for cross-modal dynamics learning and action-conditioned prediction in world-model environments.

Abstract

Training robust world models requires large-scale, precisely labeled multimodal datasets, a process historically bottlenecked by slow and expensive manual annotation. We present a production-tested GAZE pipeline that automates the conversion of raw, long-form video into rich, task-ready supervision for world-model training. Our system (i) normalizes proprietary 360-degree formats into standard views and shards them for parallel processing; (ii) applies a suite of AI models (scene understanding, object tracking, audio transcription, PII/NSFW/minor detection) for dense, multimodal pre-annotation; and (iii) consolidates signals into a structured output specification for rapid human validation. The GAZE workflow demonstrably yields efficiency gains (~19 minutes saved per review hour) and reduces human review volume by >80% through conservative auto-skipping of low-salience segments. By increasing label density and consistency while integrating privacy safeguards and chain-of-custody metadata, our method generates high-fidelity, privacy-aware datasets directly consumable for learning cross-modal dynamics and action-conditioned prediction. We detail our orchestration, model choices, and data dictionary to provide a scalable blueprint for generating high-quality world model training data without sacrificing throughput or governance.
Paper Structure (12 sections, 1 equation, 2 figures, 2 tables)

This paper contains 12 sections, 1 equation, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. The GAZE Pipeline
  3. Governance-first Video collection
  4. Video pre-processing and standardization
  5. AI understanding (multi-task pre-annotation)
  6. Human-in-the-loop review
  7. Metadata and Compliance Validation Questionnaire
  8. Results and Discussion
  9. Annotation Trial Results
  10. Related Work
  11. Limitations and Future Work
  12. Conclusion

Figures (2)

  • Figure 1: The GAZE Pipeline.
  • Figure 2: Example of 360$^\circ$ video unwrapping and multi-view rendering. Top: equi-rectangular projection (ERP) obtained by dewarping a dual-fisheye capture. Bottom: four rectilinear $\sim$90$^\circ$ FOV views (Back, Left, Front, Right) extracted from the ERP. These views are fed to downstream modules (detection/tracking, age/NSFW, ASR+PII sync) to reduce fisheye distortion and stabilize inference across directions.