Adaptive Camera Sensor for Vision Models

TL;DR

The paper tackles domain shift by introducing Lens, a post-hoc adaptive camera-sensor control system that optimizes sensor parameters for each architecture and scene using a training-free quality estimator, VisiT. Lens selects the optimal parameter via $\hat{p} = \arg\max_{p \in \mathbf{P}} Q(x_{s,p}; M)$ and uses lightweight candidate selection to maintain real-time performance, achieving up to 0.16s per capture and substantial accuracy gains. A new benchmark, ImageNet-ES Diverse, with 192,000 real-world perturbed images, demonstrates Lens's effectiveness across diverse models and environments, including synergy with domain-generalization techniques and resilience to large model size gaps (up to 50×). Ablation studies show VisiT outperforms OOD-based proxies, reinforcing the value of model-perspective quality assessment. Overall, Lens shifts the focus from model-centric improvement to data acquisition quality, enabling real-time, adaptable perception with broad practical implications.

Abstract

Domain shift remains a persistent challenge in deep-learning-based computer vision, often requiring extensive model modifications or large labeled datasets to address. Inspired by human visual perception, which adjusts input quality through corrective lenses rather than over-training the brain, we propose Lens, a novel camera sensor control method that enhances model performance by capturing high-quality images from the model's perspective rather than relying on traditional human-centric sensor control. Lens is lightweight and adapts sensor parameters to specific models and scenes in real-time. At its core, Lens utilizes VisiT, a training-free, model-specific quality indicator that evaluates individual unlabeled samples at test time using confidence scores without additional adaptation costs. To validate Lens, we introduce ImageNet-ES Diverse, a new benchmark dataset capturing natural perturbations from varying sensor and lighting conditions. Extensive experiments on both ImageNet-ES and our new ImageNet-ES Diverse show that Lens significantly improves model accuracy across various baseline schemes for sensor control and model modification while maintaining low latency in image captures. Lens effectively compensates for large model size differences and integrates synergistically with model improvement techniques. Our code and dataset are available at github.com/Edw2n/Lens.git.

Figures (7)

• Figure 1: The concept of Lens: Lens mimics the human vision system, where eyesight quality can be improved through visual sensor control, such as glasses. It leverages sensor parameter adjustments to acquire higher-quality images, thereby enhancing model accuracy.
• Figure 2: Workflow of Lens. Lens is a post-hoc, adaptive, and camera-agnostic sensor control system that dynamically responds to scene characteristics while accounting for model- and scene-specific manners based on VisiT scores to provide optimal image quality for neural networks.
• Figure 3: Quality indicators as proxies for image quality assessment: Each score is normalized between 0 to 1.
• Figure 4: Environment and sensor specifics of ImageNet-ES Diverse.
• Figure 5: Representative examples of our ImageNet-ES Diverse dataset.