ReVision: A Dataset and Baseline VLM for Privacy-Preserving Task-Oriented Visual Instruction Rewriting

TL;DR

This work tackles privacy concerns in vision-language systems by converting multimodal instructions into text-only rewrites that can be processed on devices. It introduces ReVision, a compact 250M-parameter VLM built on SigLIP/Perceiver Sampler with an instruction-tuned language model, pretrained on large image–text corpora and finetuned on a newly constructed Visual Instruction Rewriting dataset. The dataset comprises 39,023 original–rewritten instruction pairs from 1,734 images across 14 domains, generated via GPT-4o and validated through human subtasks. Empirical results show strong rewriting quality across BLEU, METEOR, ROUGE, and semantic parsing metrics, with metadata augmentation and 8-bit quantization preserving performance while maintaining a tiny on-device footprint. This approach enables secure, real-time, on-device multimodal interaction for AR/VR and mobile devices, bridging large-scale multimodal AI with privacy-centric deployment.

Abstract

Efficient and privacy-preserving multimodal interaction is essential as AR, VR, and modern smartphones with powerful cameras become primary interfaces for human-computer communication. Existing powerful large vision-language models (VLMs) enabling multimodal interaction often rely on cloud-based processing, raising significant concerns about (1) visual privacy by transmitting sensitive vision data to servers, and (2) their limited real-time, on-device usability. This paper explores Visual Instruction Rewriting, a novel approach that transforms multimodal instructions into text-only commands, allowing seamless integration of lightweight on-device instruction rewriter VLMs (250M parameters) with existing conversational AI systems, enhancing vision data privacy. To achieve this, we present a dataset of over 39,000 examples across 14 domains and develop a compact VLM, pretrained on image captioning datasets and fine-tuned for instruction rewriting. Experimental results, evaluated through NLG metrics such as BLEU, METEOR, and ROUGE, along with semantic parsing analysis, demonstrate that even a quantized version of the model (<500MB storage footprint) can achieve effective instruction rewriting, thus enabling privacy-focused, multimodal AI applications.

Figures (7)

• Figure 1: Mindmap showing Data Collection and Rewrite Desiderata. O = Original Query. R = Rewritten Query.
• Figure 2: Dataset Distributions By Intent
• Figure 3: Revision Model Architecture
• Figure 4: Class-wise F1 Scores for Intent Classification
• Figure 5: Intent and Argument Labels Considered for Data Bootstrapping