Towards In-context Scene Understanding

TL;DR

This work tackles the challenge of dense scene understanding without task-specific finetuning by introducing a retrieval-based decoding mechanism that uses a memory bank of annotated features. It couples this with contextual pretraining and a spatial attention pooling strategy to produce robust, transferable representations (Hummingbird) that can be configured on the fly for semantic segmentation and monocular depth estimation via nearest-neighbor retrieval. Empirically, the approach approaches fully finetuned performance on some tasks, significantly improves data efficiency, and enables rapid adaptation suitable for interactive assistant-style vision systems. The results suggest a practical path toward decoder-free, in-context scene understanding that scales across tasks and data regimes.

Abstract

In-context learning$\unicode{x2013}$the ability to configure a model's behavior with different prompts$\unicode{x2013}$has revolutionized the field of natural language processing, alleviating the need for task-specific models and paving the way for generalist models capable of assisting with any query. Computer vision, in contrast, has largely stayed in the former regime: specialized decoders and finetuning protocols are generally required to perform dense tasks such as semantic segmentation and depth estimation. In this work we explore a simple mechanism for in-context learning of such scene understanding tasks: nearest neighbor retrieval from a prompt of annotated features. We propose a new pretraining protocol$\unicode{x2013}$leveraging attention within and across images$\unicode{x2013}$which yields representations particularly useful in this regime. The resulting Hummingbird model, suitably prompted, performs various scene understanding tasks without modification while approaching the performance of specialists that have been finetuned for each task. Moreover, Hummingbird can be configured to perform new tasks much more efficiently than finetuned models, raising the possibility of scene understanding in the interactive assistant regime.

Figures (8)

• Figure 1: In-context scene understanding with nearest neighbor retrieval. On the left, we provide the system with a "prompt" of annotated images. On the right, we ask the system to describe new query images. The network computes dense features for each location and uses them to query features computed from the prompt. The labels associated with the nearest prompt features are then aggregated to make predictions about the query. Note that the system makes no assumptions about the nature of the labels, and as such can be used to solve a variety of different scene understanding tasks in-context. The nearest neighbors and predictions in this example are computed with our Hummingbird model.
• Figure 2: Hummingbird model components.
• Figure 3: Data efficiency of Hummingbird . The model is evaluated with retrieval-based evaluation ("NN retrieval"), linear probing ("Linear + frozen"), or full finetuning ("Linear + E2E FT").
• Figure 4: Adaptation time of Hummingbird . The model is evaluated with retrieval-based evaluation ("NN retrieval"), linear probing ("Linear + frozen"), or full finetuning ("Linear + E2E FT").
• Figure 5: Effect of memory bank length on nearest neighbor lookup at inference time. Inference time is for a single image. Lookups were done on a single Nvidia A100 GPU.