packetLSTM: Dynamic LSTM Framework for Streaming Data with Varying Feature Space

TL;DR

A dynamic LSTM-based novel method, called packetLSTM, to model the dimension-varying streams of streaming data and achieves state-of-the-art results on five datasets, and its underlying principle is extended to other RNN types, like GRU and vanilla RNN.

Abstract

We study the online learning problem characterized by the varying input feature space of streaming data. Although LSTMs have been employed to effectively capture the temporal nature of streaming data, they cannot handle the dimension-varying streams in an online learning setting. Therefore, we propose a dynamic LSTM-based novel method, called packetLSTM, to model the dimension-varying streams. The packetLSTM's dynamic framework consists of an evolving packet of LSTMs, each dedicated to processing one input feature. Each LSTM retains the local information of its corresponding feature, while a shared common memory consolidates global information. This configuration facilitates continuous learning and mitigates the issue of forgetting, even when certain features are absent for extended time periods. The idea of utilizing one LSTM per feature coupled with a dimension-invariant operator for information aggregation enhances the dynamic nature of packetLSTM. This dynamic nature is evidenced by the model's ability to activate, deactivate, and add new LSTMs as required, thus seamlessly accommodating varying input dimensions. The packetLSTM achieves state-of-the-art results on five datasets, and its underlying principle is extended to other RNN types, like GRU and vanilla RNN.

Figures (7)

• Figure 1: Performance comparison of packetLSTM with other models. The legend and outer labels are methods and datasets, respectively. Exact balanced accuracy values are provided in Table \ref{['tab:results']}.
• Figure 2: (a) The packetLSTM architecture based on (b) the data snapshot.
• Figure 3: Model performance in the scenario of sudden and obsolete features. The mean balanced accuracy in each data interval (e.g., 0 - 0.2 M) is shown in its middle (0.1 M). The pink-colored y-axis represents the Features. For e.g., '1-4' means features 1 to 4 are available in instances shaded with pink color. Both y-axes are shared by the two graphs.
• Figure 4: Models performance in reappearing features scenario and its capability of learning without forgetting. The mean balanced accuracy is depicted in the middle of each data interval.
• Figure 5: (a) The packetRNN architecture based on (b) the data snapshot. In this framework, the Vanilla RNN may be substituted with a GRU to establish the packetGRU framework.