Dreaming is All You Need

TL;DR

SleepNet and DreamNet address the exploration-exploitation trade-off in deep learning by introducing sleep- and dream-inspired training cycles that integrate unsupervised and supervised learning for both computer vision and natural language processing. SleepNet leverages a pre-trained encoder during sleep-like connections to enrich representations, while DreamNet adds a full autoencoder-based dream cycle to reconstruct and further augment hidden states. Across diverse CV and NLP datasets, DreamNet and SleepNet outperform state-of-the-art baselines, with DreamNet delivering the strongest gains thanks to its dream-based feature consolidation and robust pre-training. The work presents a general, cognitively grounded framework that can be applied to multiple domains and tasks, offering improved performance and potential for broader impact.

Abstract

In classification tasks, achieving a harmonious balance between exploration and precision is of paramount importance. To this end, this research introduces two novel deep learning models, SleepNet and DreamNet, to strike this balance. SleepNet seamlessly integrates supervised learning with unsupervised ``sleep" stages using pre-trained encoder models. Dedicated neurons within SleepNet are embedded in these unsupervised features, forming intermittent ``sleep" blocks that facilitate exploratory learning. Building upon the foundation of SleepNet, DreamNet employs full encoder-decoder frameworks to reconstruct the hidden states, mimicking the human "dreaming" process. This reconstruction process enables further exploration and refinement of the learned representations. Moreover, the principle ideas of our SleepNet and DreamNet are generic and can be applied to both computer vision and natural language processing downstream tasks. Through extensive empirical evaluations on diverse image and text datasets, SleepNet and DreanNet have demonstrated superior performance compared to state-of-the-art models, showcasing the strengths of unsupervised exploration and supervised precision afforded by our innovative approaches.

Figures (9)

• Figure 1: This diagram illustrates how sleep and dreams enhance memory formation and performance. During sleep, the body undergoes three phases: encoding, where information is initially sensed; consolidation, where information is transferred to long-term storage; and recall, where stored memories are retrieved sleep4. Sleep supports this process by creating an optimal environment for neuronal interactions, resulting in stronger memory retention and recall abilities sleep1.
• Figure 2: Overview of the Visual SleepNet Architecture, featuring M "Sleep Blocks" that are constructed by chain-like blocks processing data through convolutional layers ("Cov"), normalization ("Norm"), and sleep connection in block ①. Sleep connection includes an encoder and a deconvolution layer for feature extraction and dimension adjustment, mimicking cognitive sleep cycles. The workflow culminates in a dense layer followed by a softmax classifier for output classification.
• Figure 3: Overview of the Textual SleepNet Architecture, featuring M "Sleep Blocks" that are constructed by chain-like blocks processing data through LSTM, normalization ("Norm"), and sleep connection in block ①. Sleep connection includes softmax and argmax functions to make a legitimate sequence for the upcoming encoder for feature extraction, mimicking cognitive sleep cycles. The workflow culminates in a dense layer followed by a softmax classifier for output classification.
• Figure 4: Overview of the Visual DreamNet Architecture, integrating "Dream Blocks" that include chain-like blocks processing data through convolutional layers ("Cov"), normalization ("Norm"), and the dream connection, enhanced with a full encoder-decoder setup for advanced feature consolidation and reconstruction, simulating "dream" states where the network reinterprets input data. The simulated "dreams" will be processed in block ③ with convolutional layers and pass to the dense layer for the final classification.
• Figure 5: Overview of the Textual DreamNet Architecture, integrating "Dream Blocks" that include by chain-like blocks processing data through LSTMs, normalization ("Norm"), and the dream connection, enhanced with a full encoder-decoder setup for advanced feature consolidation and reconstruction, simulating 'dream' states where the network reinterprets input data. The simulated "dreams" will be processed in block ③ with convolutional layers and pass to the dense layer for the final classification.