NeuralPrefix: A Zero-shot Sensory Data Imputation Plugin

TL;DR

Data intermittency in sensing degrades downstream tasks and cross-domain transfer. NeuralPrefix employs a continuous-time prefix model with latent dynamics $h(t)$ satisfying $\frac{d h(t)}{dt}=g_\theta(h(t),t)$, solved by an ODE solver to fill missing frames before the task model. It features a ConvGRU-based encoder, an ODE-driven latent state, a backward autoregressive decoder, and modular frame generation with motion flow, optimized by a shrinkage loss. The approach demonstrates strong domain- and modality-OOD generalization, substantial gains in SSIM and MSE over baselines, and a tunable speed–quality trade-off suitable for real-time deployment in hazard perception and pervasive sensing.

Abstract

Real-world sensing challenges such as sensor failures, communication issues, and power constraints lead to data intermittency. An issue that is known to undermine the traditional classification task that assumes a continuous data stream. Previous works addressed this issue by designing bespoke solutions (i.e. task-specific and/or modality-specific imputation). These approaches, while effective for their intended purposes, had limitations in their applicability across different tasks and sensor modalities. This raises an important question: Can we build a task-agnostic imputation pipeline that is transferable to new sensors without requiring additional training? In this work, we formalise the concept of zero-shot imputation and propose a novel approach that enables the adaptation of pre-trained models to handle data intermittency. This framework, named NeuralPrefix, is a generative neural component that precedes a task model during inference, filling in gaps caused by data intermittency. NeuralPrefix is built as a continuous dynamical system, where its internal state can be estimated at any point in time by solving an Ordinary Differential Equation (ODE). This approach allows for a more versatile and adaptable imputation method, overcoming the limitations of task-specific and modality-specific solutions. We conduct a comprehensive evaluation of NeuralPrefix on multiple sensory datasets, demonstrating its effectiveness across various domains. When tested on intermittent data with a high 50% missing data rate, NeuralPreifx accurately recovers all the missing samples, achieving SSIM score between 0.93-0.96. Zero-shot evaluations show that NeuralPrefix generalises well to unseen datasets, even when the measurements come from a different modality.

Figures (11)

• Figure 1: Zero-shot Imputation (ZSI). We frame the new problem of zero-shot imputation and highlight potential applications (Fig. \ref{['fig:application']}). Given sensory data in a specific domain (A), ZSI seeks to learn an imputation model that can be transferred to unseen domains (B) without re-training. The table above contrasts the idea to vanilla imputation. Vanilla imputation handles intermittency in the same domain (e.g. same sensor) . Zero-shot imputation, on the other hand, generalizes imputation capabilities to domains (modalities) unseen during training.
• Figure 2: Example application of Zero-shot Impuation. (Left) A hazard perception suit encompasses several wearable sensors, such as on-chip radars, to sense the ambiance and inform/warn the wearer of important events, including quickly falling objects or approaching vehicles. Intermittency (e.g. transient sensor failure) can result in missing those events and hence dangerous consequences. (Right) NeuralPrefix can preface existing model to reconstruct the complete signal in a zero-shot manner (i.e. generalizing to unseen modalities). It doesn’t require modifying either the sensor's operational cycle (on/off schedule) or the underlying ML model.
• Figure 3: Common dynamics exist in different datasets in the data space despite the modality disparity.
• Figure 4: Continuous vs Discrete Latent States in Neural Networks. (Bottom Left) state estimation using a form of ODE. The state at time $t_i$, denoted as $\mathbf{h}(t_i)$, is updated to next state $\mathbf{h}(t_{i+1})$ through the equation above. Since intermediate steps ($\Delta t$) can be arbitrarily small, the continuous trajectory (curved path) can be obtained.
• Figure 5: ODE data recovery modes include interpolative, predictive and retrospective.