A Foundation Model for Virtual Sensors

TL;DR

The paper addresses the inefficiency and fragmentation of virtual-sensor design by introducing a foundation model that predicts $N$ virtual sensors from $M$ input time series using a causal decoder transformer. It uses prototype tokens to autoregressively forecast selected sensors and learns input relevance vectors to identify the most informative inputs per sensor, enabling cross-task synergies and structured sparsity. Large-scale evaluation on standard benchmarks and an automotive CAN bus dataset shows substantial gains, including up to $\$ up to 0?; refer to numbers in text for exact figures, while maintaining or improving predictive accuracy, and scalability to hundreds of sensors with near-constant parameter count around $7.4$–$7.8$M. The approach enables edge-deployable, explainable virtual sensing in large networks, reducing computation and memory while expanding practical deployment.

Abstract

Virtual sensors use machine learning to predict target signals from available measurements, replacing expensive physical sensors in critical applications. Existing virtual sensor approaches require application-specific models with hand-selected inputs for each sensor, cannot leverage task synergies, and lack consistent benchmarks. At the same time, emerging time series foundation models are computationally expensive and limited to predicting their input signals, making them incompatible with virtual sensors. We introduce the first foundation model for virtual sensors addressing both limitations. Our unified model can simultaneously predict diverse virtual sensors exploiting synergies while maintaining computational efficiency. It learns relevant input signals for each virtual sensor, eliminating expert knowledge requirements while adding explainability. In our large-scale evaluation on a standard benchmark and an application-specific dataset with over 18 billion samples, our architecture achieves 415x reduction in computation time and 951x reduction in memory requirements, while maintaining or even improving predictive quality compared to baselines. Our model scales gracefully to hundreds of virtual sensors with nearly constant parameter count, enabling practical deployment in large-scale sensor networks.

Figures (15)

• Figure 1: ➀ Recent works train, maintain, and deploy isolated application-specific models for every virtual sensor. ➁ Our foundational approach unifies multiple virtual sensors into a single model, exploiting synergies and achieving superior scalability. It learns relevant input signals and selectively predicts virtual sensors, increasing efficiency and explainability without expert knowledge.
• Figure 2: We present a foundation model that can predict a user-selected target virtual sensor ($z^\prime_2$, green) out of multiple possible virtual sensors. ➀ We first define which virtual sensor to forecast (here sensor $z^\prime_2$). ➁ Next, we introduce an initially empty prototype token $[t]{\vec{0}}$ and add the corresponding variate embedding $v^\prime_2 \in \mathcal{V^\prime}$ to guide the neural network to predict the specified virtual sensor, based on available sensor signals ($z_1$, pink; $z_2$, red; $z_3$, orange). We divide input signals $z_1,z_2,z_3$ into patches, embed them into tokens, and add time and variate embedding $\mathcal{V}$ to them. Starting from empty prototypes in cycle $t_1$, ➂ we forecast virtual sensors autoregressively using the model's previous prediction $z^\prime_2$ as current input in step $t_2$. ➃ Within the transformer, our architecture learns relevant input signals for each virtual sensor from attention gradients through trainable signal relevance vectors $\mathcal{R^\prime}$, adding explainability. After selecting the corresponding signal relevance vector $r^\prime_2$, we compute the cross-relevance for signal-to-signal communication, duplicate our mask across time to learn time-independent signal importance, and apply it as attention bias. ➄ By later sparsifying signal relevance vectors $\mathcal{R^\prime}$ based on a threshold $r^\prime_{\mathrm{thres}}$, we structurally prune irrelevant input signals to achieve considerable efficiency gains.
• Figure 3: Varying our signal importance threshold $r^\prime_{\mathrm{thres}}$ during training or inference generates smooth trade-offs between input set sparsity and MSE on the Traffic dataset.
• Figure 4: Sparsity increases as fewer virtual sensors are predicted simultaneously through our sensor selection mechanism. Results are shown for the Traffic dataset.
• Figure 5: Our architecture subsequently identifies 2.0 virtual sensors among 1000.0 signals in our synthetic random dataset during training, reducing the validation loss in a stepwise manner.