RP-CATE: Recurrent Perceptron-based Channel Attention Transformer Encoder for Industrial Hybrid Modeling

TL;DR

This work addresses the gap in industrial hybrid modeling where single ML methods and unexploited data associations limit performance. It introduces RP-CATE, a Transformer-based architecture that replaces self-attention with channel attention and integrates PSD, RP, and PID-based attention to learn underlying associations in non-strictly-sequential industrial data. The approach yields superior predictive accuracy and robustness against baselines in acentric-factor modeling, with a substantial Model Improvement Rate compared to purely mechanistic models. The proposed framework has broad potential for application across engineering domains requiring accurate, interpretable hybrid predictions.

Abstract

Nowadays, industrial hybrid modeling which integrates both mechanistic modeling and machine learning-based modeling techniques has attracted increasing interest from scholars due to its high accuracy, low computational cost, and satisfactory interpretability. Nevertheless, the existing industrial hybrid modeling methods still face two main limitations. First, current research has mainly focused on applying a single machine learning method to one specific task, failing to develop a comprehensive machine learning architecture suitable for modeling tasks, which limits their ability to effectively represent complex industrial scenarios. Second, industrial datasets often contain underlying associations (e.g., monotonicity or periodicity) that are not adequately exploited by current research, which can degrade model's predictive performance. To address these limitations, this paper proposes the Recurrent Perceptron-based Channel Attention Transformer Encoder (RP-CATE), with three distinctive characteristics: 1: We developed a novel architecture by replacing the self-attention mechanism with channel attention and incorporating our proposed Recurrent Perceptron (RP) Module into Transformer, achieving enhanced effectiveness for industrial modeling tasks compared to the original Transformer. 2: We proposed a new data type called Pseudo-Image Data (PID) tailored for channel attention requirements and developed a cyclic sliding window method for generating PID. 3: We introduced the concept of Pseudo-Sequential Data (PSD) and a method for converting industrial datasets into PSD, which enables the RP Module to capture the underlying associations within industrial dataset more effectively. An experiment aimed at hybrid modeling in chemical engineering was conducted by using RP-CATE and the experimental results demonstrate that RP-CATE achieves the best performance compared to other baseline models.

Figures (4)

• Figure 1: Overview of the RP-CATE framework.
• Figure 2: The process of data transformation. The red dashed box in the figure represents the Cyclic Sliding Window (CSW), with a window size $w$ set to $k^2$ and a sliding step size of 1. For each sliding step, the CSW produces a data set $z_i$. After reaching the $m - w + 1$-th step, there will be gaps within the window. These gaps will cyclically wrap around to the beginning of the input, filling the gaps in turn until the window has traversed all the data from left to right. Ultimately, $CSW(.)$ will generate the data set $z_i$ with the same number of elements as the original data, where $i=\{1,2,...,m\}$.
• Figure 3: Diagram of Channel Attention Block. The figure demonstrates the process of generating attention using a sample ${PID}_i$ from $PID$. In this process, the red italicized text represents intermediate variables. Specifically, $h_{Max} \in H_{Max}, h_{Avg} \in H_{Avg}, h_1 \in H_1, h_2 \in H_2$ and $attention \in Attentions$
• Figure 4: Attention maps. The y-axis is the feature indices of the dataset, and the x-axis is the sample indices. On the color bar to the right, the green dashed lines indicates the average attention weights for different features under N:1, while the red represents the average attention weights under N:2.