Semantic-Based Active Perception for Humanoid Visual Tasks with Foveal Sensors

TL;DR

The paper presents a semantic-based foveal active perception framework that updates a per-cell Dirichlet semantic map from object-detection outputs to guide gaze decisions for scene exploration and targeted visual search. It combines a calibrated foveal observation model, Bayesian fusion via Kaplan’s rule, and both predictive and non-predictive gaze strategies, evaluating them against a VOCUS2 saliency baseline. Across experiments on COCO-derived data, the semantic approach shows superior scene semantic representation and, with predictive search, significant gains in locating targets among distractors, at the cost of higher computation. The work demonstrates the value of top-down semantic information for gaze control and points to fruitful directions for integrating semantic guidance with traditional bottom-up cues in human-like visual tasks.

Abstract

The aim of this work is to establish how accurately a recent semantic-based foveal active perception model is able to complete visual tasks that are regularly performed by humans, namely, scene exploration and visual search. This model exploits the ability of current object detectors to localize and classify a large number of object classes and to update a semantic description of a scene across multiple fixations. It has been used previously in scene exploration tasks. In this paper, we revisit the model and extend its application to visual search tasks. To illustrate the benefits of using semantic information in scene exploration and visual search tasks, we compare its performance against traditional saliency-based models. In the task of scene exploration, the semantic-based method demonstrates superior performance compared to the traditional saliency-based model in accurately representing the semantic information present in the visual scene. In visual search experiments, searching for instances of a target class in a visual field containing multiple distractors shows superior performance compared to the saliency-driven model and a random gaze selection algorithm. Our results demonstrate that semantic information, from the top-down, influences visual exploration and search tasks significantly, suggesting a potential area of research for integrating it with traditional bottom-up cues.

Figures (14)

• Figure 1: Example of a regular image (top) that is artificially foveated (bottom) to emulate a visual field, together with the object predictions, outputted by an object detector, represented by their bounding-boxes and confidence scores.
• Figure 2: Example of a Cartesian image (left) to which the artificial foveal system, proposed by Almeida et al. fovsys, has been applied. There are presented two foveal images, with wider (center) and narrower (right) foveal dimensions.
• Figure 3: Simplified version of the Itti-Koch attention system trad_sal, inspired by both the neuronal architecture and behavior of the early primate visual system.
• Figure 4: Our general methodological approach (based on main) to semantic visual search and scene exploration tasks. An image is foveated on some initial image location, to simulate the human visual sensor. The foveal image is fed to an object detection model that may generate multiple bounding-boxes and the respective categorical scores. This information is used to update a world-fixed semantic map. This can be done in two ways: using the raw scores of the object detector (solid line) or with scores calibrated according to the effects of the foveal image characteristics (dotted line), since increased blur in the periphery will increase uncertainty to the classification scores. Alternatively, we also tested the use of a traditional saliency map approach to predict the next best focal point. The image is then foveated at the new selected focal point, simulating a saccade. A clasical Inhibition of Return method (IOR) trad_sal is applied to prevent returning to a previously visited location. The process is repeated until some terminal condition is met.
• Figure 5: Illustration of a foveal scene, fixed on a focal point, from which an arbitrary detector outputs $L_t$ object predictions. The distance $d_{t,l}$ between the center of a bounding-box $\mathcal{B}_{t,l}$ and the center of the fovea is also represented.