Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control

TL;DR

This work addresses the challenge of aligning 2D-trained vision-language models with 3D multi-object scenes by introducing MI-ZO, a zeroth-order online method to estimate multivariate mutual information and guide an in-scene camera via active regret minimisation. The controller, built without backpropagation through the VLM, leverages a weighted mixture of entropy sources to select informative viewpoints, with theoretical bounds ensuring bounded regret. Empirical evaluation on three cross-modal 3D benchmarks (GeoProperties-3DS, FeatureID-3DS, PartialView-3DS) demonstrates substantial improvements in cross-modal reasoning and occlusion handling over standard control methods and ablations, even with limited demonstrations. This approach offers a scalable, derivative-free pathway to enhance VLM reasoning in complex 3D environments, with potential impact on planetary science and automated analysis of 3D visual data.

Abstract

Cross-modal systems trained on 2D visual inputs are presented with a dimensional shift when processing 3D scenes. An in-scene camera bridges the dimensionality gap but requires learning a control module. We introduce a new method that improves multivariate mutual information estimates by regret minimisation with derivative-free optimisation. Our algorithm enables off-the-shelf cross-modal systems trained on 2D visual inputs to adapt online to object occlusions and differentiate features. The pairing of expressive measures and value-based optimisation assists control of an in-scene camera to learn directly from the noisy outputs of vision-language models. The resulting pipeline improves performance in cross-modal tasks on multi-object 3D scenes without resorting to pretraining or finetuning.

Figures (20)

• Figure 1: Textures of objects in 3D scenes vary in appearance depending on the position of the in-scene camera. Aligning a description with a scene is even harder when referenced objects belong to a group such as a single boulder in an outcrop on Mars. An optimal sequence of viewpoints improves the decisions of VLMs trained on 2D data where understanding a 3D reconstructed scene relies on a set of views.
• Figure 2: Evaluation consists of two rounds with a set number of viewpoints. In the correction round, an in-scene camera controller uses multiple sources of information to predict a sequence of camera actions. In each round a VLM makes decisions based on viewpoints and descriptions.
• Figure 3: Variants of our two multi-information metrics with active regret minimisation ($MI\text{-}ar$) compare favourably with versions calculated with standard methods (MI) on how the measures distribute scores in relation to the decisions of a VLM. $MI\text{-}ar$ groups the respective centres of mass for correctness decisions over separate ranges of the distributions.
• Figure 4: Systems trained on 2D data reason over 3D scenes using a set of viewpoints from the in‑scene camera. The objective for camera control methods is to predict the sequence of viewpoints with the highest likelihood of returning a correct assessment of the scene by the VLM. Scenes in all our benchmarks contain multiple objects from a single class.
• Figure 5: Samples from our GeoProperties-3DS benchmark with close-ups of properties correctly identifying a single object in the scene. Descriptions refer to the largest rock or boulder and match scenes on the right. VLMs return false positives and this weakness is identified by presenting the same descriptions for the samples on the left where the property does not apply to objects in view.

Theorems & Definitions (20)

• Theorem 1: Function with upper bound on nonpositive contribution
• Definition 1
• Definition 2
• Definition 3
• Lemma 1: Nonpositivity
• proof : Proof of Lemma \ref{['lemma:nonpos']}
• Definition 4
• Definition 5
• Definition 6
• Remark 1