LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision

TL;DR

This work tackles the computational burden of vision transformers on high-resolution images by introducing LookWhere, a framework that splits computation into a lightweight low-resolution selector and a high-resolution extractor. It learns where to look and what to see through What-Where Distillation from a fixed self-supervised teacher (DINOv2), enabling end-to-end pretraining and task-agnostic transfer. Empirically, LookWhere achieves superior accuracy-efficiency trade-offs across standard benchmarks (ImageNet, ADE20K) and high-resolution, spatially sparse tasks (Traffic Signs, Birds, Billiards), delivering large reductions in FLOPs and latency while maintaining or improving accuracy. The approach promises practical deployment of high-resolution vision systems with broad transferability across diverse visual tasks.

Abstract

Vision transformers are ever larger, more accurate, and more expensive to compute. The expense is even more extreme at high resolution as the number of tokens grows quadratically with the image size. We turn to adaptive computation to cope with this cost by learning to predict where to compute. Our LookWhere method divides the computation between a low-resolution selector and a high-resolution extractor without ever processing the full high-resolution input. We jointly pretrain the selector and extractor without task supervision by distillation from a self-supervised teacher, in effect, learning where and what to compute simultaneously. Unlike prior token reduction methods, which pay to save by pruning already-computed tokens, and prior token selection methods, which require complex and expensive per-task optimization, LookWhere economically and accurately selects and extracts transferrable representations of images. We show that LookWhere excels at sparse recognition on high-resolution inputs (Traffic Signs), maintaining accuracy while reducing FLOPs by up to 34x and time by 6x. It also excels at standard recognition tasks that are global (ImageNet classification) or local (ADE20K segmentation), improving accuracy while reducing time by 1.36x. See https://github.com/antofuller/lookwhere for the code and weights.

Figures (5)

• Figure 1: Attention as Supervision. Self-supervised models (like DINOv2 oquab2023dinov2) learn what is visually interesting without tasks or labels. We use a last step of their deep computation, the final attention map, to start ours; supervising an efficient selector that predicts where to process for adaptive computation.
• Figure 2: LookWhere is trained by distillation from a self-supervised teacher to learn where to compute, from the attention map, and what to compute, from the class and patch token representations. Only the teacher sees the high-res input to make its attention and tokens for the losses. The selector predicts where to look from the low-res input. The extractor predicts what to see (e.g. a boy in a hat and jersey, a coffee cup, $\dots$) from the top $k$ high-res patches (and low-res tokens) from the selector.
• Figure 3: Looking at Adaptive Computation. We visualize the selector’s prediction of where to compute and the extractor's sparse input for what to see. Left: Spatially-sparse traffic sign recognition. Right: Fine-grained bird recognition. The same selector generalizes across these images and tasks.
• Figure 4: High-Resolution Recognition on Traffic Signs, Birds (CUB), and Billiards. We plot the accuracy and throughput of LookWhere, its teacher (DINOv2), and SoTA token reduction and selection methods. LookWhere reaches the Pareto frontier for sparse object recognition (left, right) and for fine-grained classification (center). To more fully measure computation, points are colored by memory usage (darker is higher/less efficient). See Appendices \ref{['appendix:experimental_details']} and \ref{['appendix:additional_finetuning']} for more details.
• Figure :