Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fusing Foveal Fixations Using Linear Retinal Transformations and Bayesian Experimental Design

Christopher K. I. Williams

TL;DR

This work tackles trans-saccadic fusion by modeling a high-resolution latent image $oldsymbol{x}$ and rendering each fixation as a linear retinal transformation $ oldsymbol{y}_a = V_{oldsymbol{\ell}(a)} oldsymbol{x}$. It leverages (mixture) factor analysis to relate the latent scene to observed glimpses, enabling exact Gaussian inference and reconstruction from multiple fixations, while formulating next-look decisions as Bayesian experimental design using the Expected Information Gain criterion. The authors derive exact BED results for FA and provide informative bounds for MoFA, demonstrating that optimal fixation sequences reduce uncertainty and improve reconstruction on Frey faces and MNIST 2s, with substantial gains over random gaze plans. The work suggests practical active-vision gains and lays groundwork for extensions to deeper generative models, richer geometric transformations, and multi-object scenes, highlighting the potential for robust, information-driven gaze planning in vision systems.

Abstract

Humans (and many vertebrates) face the problem of fusing together multiple fixations of a scene in order to obtain a representation of the whole, where each fixation uses a high-resolution fovea and decreasing resolution in the periphery. In this paper we explicitly represent the retinal transformation of a fixation as a linear downsampling of a high-resolution latent image of the scene, exploiting the known geometry. This linear transformation allows us to carry out exact inference for the latent variables in factor analysis (FA) and mixtures of FA models of the scene. Further, this allows us to formulate and solve the choice of "where to look next" as a Bayesian experimental design problem using the Expected Information Gain criterion. Experiments on the Frey faces and MNIST datasets demonstrate the effectiveness of our models.

Fusing Foveal Fixations Using Linear Retinal Transformations and Bayesian Experimental Design

TL;DR

This work tackles trans-saccadic fusion by modeling a high-resolution latent image and rendering each fixation as a linear retinal transformation . It leverages (mixture) factor analysis to relate the latent scene to observed glimpses, enabling exact Gaussian inference and reconstruction from multiple fixations, while formulating next-look decisions as Bayesian experimental design using the Expected Information Gain criterion. The authors derive exact BED results for FA and provide informative bounds for MoFA, demonstrating that optimal fixation sequences reduce uncertainty and improve reconstruction on Frey faces and MNIST 2s, with substantial gains over random gaze plans. The work suggests practical active-vision gains and lays groundwork for extensions to deeper generative models, richer geometric transformations, and multi-object scenes, highlighting the potential for robust, information-driven gaze planning in vision systems.

Abstract

Humans (and many vertebrates) face the problem of fusing together multiple fixations of a scene in order to obtain a representation of the whole, where each fixation uses a high-resolution fovea and decreasing resolution in the periphery. In this paper we explicitly represent the retinal transformation of a fixation as a linear downsampling of a high-resolution latent image of the scene, exploiting the known geometry. This linear transformation allows us to carry out exact inference for the latent variables in factor analysis (FA) and mixtures of FA models of the scene. Further, this allows us to formulate and solve the choice of "where to look next" as a Bayesian experimental design problem using the Expected Information Gain criterion. Experiments on the Frey faces and MNIST datasets demonstrate the effectiveness of our models.
Paper Structure (21 sections, 30 equations, 3 figures, 2 tables)

This paper contains 21 sections, 30 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Retinal transformation
  4. Factor analysis and MoFA models
  5. Fusing multiple glimpses
  6. Bayesian experimental design
  7. Learning $W$ and the $\Psi^{y}_i$s
  8. Experiments
  9. Frey faces: Experimental design
  10. Frey faces: Learning the parameters from $Y$ data
  11. MNIST 2s: Experimental design
  12. Related work
  13. 3D reconstruction from multiple views:
  14. Saliency maps:
  15. Classifiers:
  16. ...and 6 more sections

Figures (3)

  • Figure 1: (a) The $20 \times 20$ retinal transformation used in our experiments. The innermost squares are $1 \times 1$ pixels, while the outermost are $4 \times 4$. (b) an image ($28 \times 20$) from the Frey dataset at full resolution. (c) Visualization of the same image after the retinal transformation is applied to the top $20 \times 20$ block of the image. Note: the bottom 8 rows of the original image are not observed, and are shown here as white. (d) Visualization of the input image under a different retinal transformation, with an offset $[8,4]$. Again regions of the input image that are not observed are shown as a white border.
  • Figure 2: (a) Original Frey face image, (b) retinal transformation 1, (c) reconstruction from this RT, (d) retinal transformation 2, (e) reconstruction from both RTs.
  • Figure 3: Top row: An example image (left) undergoes RT1 and is reconstructed as in the third panel. The posterior variance after RT1 (4th panel) is high towards the bottom of the image where there were no observations. Bottom row: the prior variance (averaged over the 10 components) is shown on the left. A second retinal transformation is applied, and the resulting reconstruction using both RTs is shown in the third panel. The posterior variance after RT1 and RT2 is much reduced relative to RT1 only.