Towards Two-Stream Foveation-based Active Vision Learning

TL;DR

This work addresses weakly-supervised object localization by proposing a bio-inspired two-stream framework that mimics dorsal (where) and ventral (what) visual pathways. It combines a ventral stream trained with image-level labels and a dorsal stream trained via reinforcement learning to actively adjust foveated glimpses, enabling iterative localization from partial observations. The approach is validated on CelebA and CUB-200-2011, showing competitive localization and attribute identification performance, and demonstrating that the dorsal stream can generalize to unseen datasets without re-training. The study advances bio-plausible active vision methods and highlights the potential of combining supervised learning with RL to perform robust WSOL with limited supervision.

Abstract

Deep neural network (DNN) based machine perception frameworks process the entire input in a one-shot manner to provide answers to both "what object is being observed" and "where it is located". In contrast, the "two-stream hypothesis" from neuroscience explains the neural processing in the human visual cortex as an active vision system that utilizes two separate regions of the brain to answer the what and the where questions. In this work, we propose a machine learning framework inspired by the "two-stream hypothesis" and explore the potential benefits that it offers. Specifically, the proposed framework models the following mechanisms: 1) ventral (what) stream focusing on the input regions perceived by the fovea part of an eye (foveation), 2) dorsal (where) stream providing visual guidance, and 3) iterative processing of the two streams to calibrate visual focus and process the sequence of focused image patches. The training of the proposed framework is accomplished by label-based DNN training for the ventral stream model and reinforcement learning for the dorsal stream model. We show that the two-stream foveation-based learning is applicable to the challenging task of weakly-supervised object localization (WSOL), where the training data is limited to the object class or its attributes. The framework is capable of both predicting the properties of an object and successfully localizing it by predicting its bounding box. We also show that, due to the independent nature of the two streams, the dorsal model can be applied on its own to unseen images to localize objects from different datasets.

Figures (14)

• Figure 1: (a) The dorsal (shown in green) and the ventral (shown in purple) stream pathways separate at the primary visual cortex area and stretch to the parietal lobe and the temporal lobe, respectively. (b) Example of foveation in human vision. When looking at the golden Pegasus, the amount of details gradually reduces from the visual fixation point.
• Figure 2: Overview of the proposed framework. Stage 1 (initialization) takes place when the input is observed for the first time and uses the dorsal M1 model to predict the initial fixation point. Stage 2 (foveated feature extraction) processes the foveated glimpses using the dorsal M2A and the ventral M2B models to predict the set of actions to adjust the foveation, and extract features identifying the object of interest, respectively. The example shows foveated glimpse extracted at the 3rd iteration. Stage 3 (foveated glimpse adjustment) adjusts the foveated region (i.e. the blue rectangle shown on the input image) resulting in a new foveated glimpse. Stages 2 and 3 are then iteratively implemented to process and extract the sequence of foveated glimpses with the goal of localizing and identifying the object in the image.
• Figure 3: Overview of how the M2A model training is formulated using reinforcement learning. (a) Standard scheme of reinforcement learning, illustrating how the environment provides state and reward information based on the action that an agent performs on the environment. (b) Example of how M2A model (i.e. the agent) performs the action ($a_{t=10})$) on the environment, which result in a positive improvement in the form of improved localization from the state $S_{t=10}$ to the state $S_{t=11}$. (c) Example of how M2A model (i.e. the agent) performs the action ($a_{t=11})$) on the environment, which result in a negative/undesired change in the form of unnecessarily expanding the foveated glimpse from the state $S_{t=11}$ to the state $S_{t=12}$.
• Figure 4: Neural network architectures used to realize different models needed by the proposed framework for experiments on CelebA dataset.
• Figure 5: Statistics over the foveated glimpse iterations: (top left) average cosine similarity, (top right) average IoU, (bottom) average cumulative reward for RL averaged over test split of CelebA dataset.