Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GLIMPSE : Real-Time Text Recognition and Contextual Understanding for VQA in Wearables

Akhil Ramachandran, Ankit Arun, Ashish Shenoy, Abhay Harpale, Srihari Jayakumar, Debojeet Chatterjee, Mohsen Moslehpour, Pierce Chuang, Yichao Lu, Vikas Bhardwaj, Peyman Heidari

TL;DR

This work exploits asymmetry in text recognition and visual reasoning with a hybrid architecture that performs selective high-resolution OCR on-device while streaming low-resolution video for visual context, enabling sustained VQA sessions on resource-constrained wearables without sacrificing text understanding quality.

Abstract

Video Large Language Models (Video LLMs) have shown remarkable progress in understanding and reasoning about visual content, particularly in tasks involving text recognition and text-based visual question answering (Text VQA). However, deploying Text VQA on wearable devices faces a fundamental tension: text recognition requires high-resolution video, but streaming high-quality video drains battery and causes thermal throttling. Moreover, existing models struggle to maintain coherent temporal context when processing text across multiple frames in real-time streams. We observe that text recognition and visual reasoning have asymmetric resolution requirements - OCR needs fine detail while scene understanding tolerates coarse features. We exploit this asymmetry with a hybrid architecture that performs selective high-resolution OCR on-device while streaming low-resolution video for visual context. On a benchmark of text-based VQA samples across five task categories, our system achieves 72% accuracy at 0.49x the power consumption of full-resolution streaming, enabling sustained VQA sessions on resource-constrained wearables without sacrificing text understanding quality.

GLIMPSE : Real-Time Text Recognition and Contextual Understanding for VQA in Wearables

TL;DR

This work exploits asymmetry in text recognition and visual reasoning with a hybrid architecture that performs selective high-resolution OCR on-device while streaming low-resolution video for visual context, enabling sustained VQA sessions on resource-constrained wearables without sacrificing text understanding quality.

Abstract

Video Large Language Models (Video LLMs) have shown remarkable progress in understanding and reasoning about visual content, particularly in tasks involving text recognition and text-based visual question answering (Text VQA). However, deploying Text VQA on wearable devices faces a fundamental tension: text recognition requires high-resolution video, but streaming high-quality video drains battery and causes thermal throttling. Moreover, existing models struggle to maintain coherent temporal context when processing text across multiple frames in real-time streams. We observe that text recognition and visual reasoning have asymmetric resolution requirements - OCR needs fine detail while scene understanding tolerates coarse features. We exploit this asymmetry with a hybrid architecture that performs selective high-resolution OCR on-device while streaming low-resolution video for visual context. On a benchmark of text-based VQA samples across five task categories, our system achieves 72% accuracy at 0.49x the power consumption of full-resolution streaming, enabling sustained VQA sessions on resource-constrained wearables without sacrificing text understanding quality.
Paper Structure (18 sections, 4 figures, 8 tables)

This paper contains 18 sections, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Detailed Architecture
  3. On-Device Components
  4. Smart Frame Selection
  5. Blur Detection
  6. Region of Interest and Text Detection
  7. Server-Side Components
  8. OCR session manager
  9. Video LLM
  10. Inference Context
  11. Experiments
  12. Datasets
  13. Results
  14. Comparison with Baselines
  15. Per-Category Analysis
  16. ...and 3 more sections

Figures (4)

  • Figure 1: Illustrative wearable VQA scenarios: a user poses natural-language questions about text in their environment (e.g., signs, labels, menus) and receives real-time contextual answers through the GLIMPSE pipeline.
  • Figure 2: Overview of the GLIMPSE hybrid architecture. On-device components (Smart Frame Selection, high-resolution OCR) process video locally and transmit sparse text payloads, while a low-resolution video stream is sent to the server where the OCR Session Manager and Video LLM perform fusion and reasoning for VQA.
  • Figure 3: Impact of blur on text legibility in wearable video frames. (Left) Motion blur from rapid device movement renders text unreadable. (Right) Low-light conditions produce underexposed, noisy frames. The blur detection stage uses IMU-derived motion energy and camera exposure time to reject such frames before OCR processing.
  • Figure 4: ROI and text detection outputs across four interaction scenarios. (a) Hand-holding a text-bearing object: the model crops the detected text region at full resolution for OCR. (b) Finger-pointing at a text area: the pointed-at region is selected as the primary ROI. (c) Hand-holding a non-text object: the model correctly identifies the object but skips OCR since no text is present. (d) No salient region detected: the frame is passed without text processing. Bounding boxes and keypoints are shown as model outputs.