Actively Inferring Optimal Measurement Sequences

TL;DR

This work tackles the challenge of reconstructing high-dimensional data from costly measurements by introducing an active sequential inference framework that operates in the low-dimensional latent space of a variational autoencoder (VAE). It extends the VAE with a partial encoder to map partial measurements into latent space and couples this with a hybrid inference strategy that uses simulated measurements and either a partial VAE posterior or stochastic variational inference to guide measurement selection. Three criteria are proposed for choosing the next measurement: likelihood-based (QP), mutual information (MI), and an inference-free Hadamard optimisation (HO); the convolutional Hadamard basis enables efficient sensing for image-like data. Experiments on Fashion MNIST show that useful measurement patterns are identified within about 10 steps, with the partial VAE approach achieving superior speed and competitive accuracy compared with SVI, demonstrating the method’s potential to significantly reduce data acquisition needs in imaging and related tasks.

Abstract

Measurement of a physical quantity such as light intensity is an integral part of many reconstruction and decision scenarios but can be costly in terms of acquisition time, invasion of or damage to the environment and storage. Data minimisation and compliance with data protection laws is also an important consideration. Where there are a range of measurements that can be made, some may be more informative and compliant with the overall measurement objective than others. We develop an active sequential inference algorithm that uses the low dimensional representational latent space from a variational autoencoder (VAE) to choose which measurement to make next. Our aim is to recover high dimensional data by making as few measurements as possible. We adapt the VAE encoder to map partial data measurements on to the latent space of the complete data. The algorithm draws samples from this latent space and uses the VAE decoder to generate data conditional on the partial measurements. Estimated measurements are made on the generated data and fed back through the partial VAE encoder to the latent space where they can be evaluated prior to making a measurement. Starting from no measurements and a normal prior on the latent space, we consider alternative strategies for choosing the next measurement and updating the predictive posterior prior for the next step. The algorithm is illustrated using the Fashion MNIST dataset and a novel convolutional Hadamard pattern measurement basis. We see that useful patterns are chosen within 10 steps, leading to the convergence of the guiding generative images. Compared with using stochastic variational inference to infer the parameters of the posterior distribution for each generated data point individually, the partial VAE framework can efficiently process batches of generated data and obtains superior results with minimal measurements.

Figures (7)

• Figure 1: Active Learning. At step $k=0$, for each of $N$ starting points, sample $\mathbf{z}_i$ from the prior for step $k=0$, push $\mathbf{z}_i$ through the decoder to obtain a generated image $\hat{\mathbf{x}}_i$ and estimate possible measurements $\hat{\mathbf{y}}_i$ for each pattern not yet measured (e.g. $\hat{\mathbf{y}}_1$, $\hat{\mathbf{y}}_2$ and $\hat{\mathbf{y}}_3$ illustrated above). Use the partial encoder to approximate the posterior with probability distributions and estimate probability densities $q^1(\mathbf{z}_1|\hat{\mathbf{y}}_1)$, $q^2(\mathbf{z}_1|\hat{\mathbf{y}}_2)$ and $q^3(\mathbf{z}_1|\hat{\mathbf{y}}_3)$ (illustrated above). Select the pattern which maximises the chosen expression and take the measurement associated with this pattern. Update the measurement set and update the predictive prior for the next step. By repeating these steps, we move towards the target distribution $p(\mathbf{z}|\mathbf{x})$.
• Figure 2: Graphical representation of the VAE model. An image $\mathbf{x}$ is generated by a random variable $\mathbf{z}$ parameterized by a deep neural network with parameters $\theta$, a). A variational distribution parameterized by a deep neural network with parameters $\phi$ is introduced to infer $\mathbf{z}$ given $\mathbf{x}$, b). The encoder and decoder are trained together using $N$ images. To train the partial encoder we simulate $N \times E$ measurements $\mathbf{y}^{[b]}$ where $b$ is a subset of patterns, c). The partial encoder parameterized by $\phi^p$ is trained with the original decoder, d). The SVI method involves inferring the mean $\mu$ and variance $\Sigma$ for each image individually, e).
• Figure 3: Comparison of pVAE and SVI. The performance scores (mse left column and ssim right column) are averaged over 100 samples from ten images (each from a different class) and the algorithm is run for 100 steps. Here lower is better for mse left and higher is better for ssim right. The performance of pVAE (1 iteration) bold line with SVI and a variable number of iterations $\{10,20,30,40,50,60,70\}$mixed light lines. The SVI method improves as the number of iterations increases. At 100 steps the results for pVAE lie between the results for SVI with 60 (SVI60) and 70 (SVI70) iterations. In terms of timings, the pVAE takes $9\times{10}^{-4}$ seconds per step and the SVI method takes $7.2\times{10}^{-3}$ seconds per iteration. With many iterations required for SVI, this makes the SVI method at least an order of magnitude slower than pVAE. Time measurements were taken using a NVIDIA GeForce RTX 3090 GPU.
• Figure 4: Comparison of choice criteria in terms of the structural similarity index (SSIM). The different criteria for choosing the next pattern (QP, MI and HO), equations (\ref{['eqn:critlogq']}), (\ref{['eqn:critlogMI']}) and (\ref{['eqn:critH']}) respectively, were evaluated (mean SSIM) using 10 test images, one from each class, and 1 (top left), 10 (top right), 100 (bottom left), 200 (bottom right) latent vector samples over 100 steps. Our method (QP) outperforms Hadamard optimisation (HO) for the first 25 steps and both show superior performance to MI across all the experiments. In terms of timings, HO took 0.7778 seconds, QP took 1.3583 seconds and MI took 1.4219 seconds to complete 50 steps. Time measurements were taken using a NVIDIA GeForce RTX 3090 GPU.
• Figure 5: UMAP is used to reduce the mean of the latent representations of 10,000 test images to 2 dimensions (left hand column). The classes are colour coded and the UMAP clusters within class and between similar classes (i.e. ankle boot, sneaker and shoe) indicate that the class structure has been retained by the latent representation and further dimension reduction. The mean of 100 samples is similarly projected (black crosses) on to the map at each step of the algorithm. The measurements made, the samples projected back into image space and the image to be recovered are shown in the top left corner, centre and bottom right corner of the (right hand column) respectively. We see the diversity of possible images in the right hand column, and this decreases as uncertainty is reduced by more measurements.