EAPCR: A Universal Feature Extractor for Scientific Data without Explicit Feature Relation Patterns

TL;DR

EAPCR, a universal feature extractor designed for data without explicit Feature Relation Patterns (FRPs), is introduced, demonstrating its robustness and superior performance in scientific tasks without FRPs.

Abstract

Conventional methods, including Decision Tree (DT)-based methods, have been effective in scientific tasks, such as non-image medical diagnostics, system anomaly detection, and inorganic catalysis efficiency prediction. However, most deep-learning techniques have struggled to surpass or even match this level of success as traditional machine-learning methods. The primary reason is that these applications involve multi-source, heterogeneous data where features lack explicit relationships. This contrasts with image data, where pixels exhibit spatial relationships; textual data, where words have sequential dependencies; and graph data, where nodes are connected through established associations. The absence of explicit Feature Relation Patterns (FRPs) presents a significant challenge for deep learning techniques in scientific applications that are not image, text, and graph-based. In this paper, we introduce EAPCR, a universal feature extractor designed for data without explicit FRPs. Tested across various scientific tasks, EAPCR consistently outperforms traditional methods and bridges the gap where deep learning models fall short. To further demonstrate its robustness, we synthesize a dataset without explicit FRPs. While Kolmogorov-Arnold Network (KAN) and feature extractors like Convolutional Neural Networks (CNNs), Graph Convolutional Networks (GCNs), and Transformers struggle, EAPCR excels, demonstrating its robustness and superior performance in scientific tasks without FRPs.

Figures (7)

• Figure 1: Motivation for designing feature extractors for data without FRPs.
• Figure 2: The illustration of the method. (a) an overview of EAPCR. (b) an illustration of how Embedding and bilinear Attention can expose all possible FRPs. (c) an illustration of how the permuted cnns considers combinations of originally close matrix elements as well as combinations of originally distant elements.
• Figure 3: Results on synthesized data without explicit FRPs. (a) Illustration of synthesized data: In the raw image, pixel correlations align with spatial positions (a-1), but in the synthesized data, the spatial correlations breaks (a-2). (b) Experimental results confirm that EAPCR's strong performance is not due to the use of a similarly designed permutation when synthesizing the data. (c) Comparison of various methods on data with FRPs and synthesized data without explicit FRPs.
• Figure 4: The comparison and ablation experiments conducted on the more synthesized data. Asterisks indicate that only subsets of gray data were used for experimental efficiency. This involves employing limited random sampling to create combined training and testing datasets. For example, from ImageNet, 10 categories, each comprising 400 randomly selected images, were selected for training and 50 for testing, optimizing both time and computational resources.
• Figure 5: Analysis of inter-pixel distance and statistical correlations. (a) Relationship between the Pearson correlation coefficient and inter-pixel distance. (b) Relationship between mutual information and inter-pixel distance.

Theorems & Definitions (6)

• Proposition F.1
• proof
• Proposition F.2
• proof
• Proposition F.3
• proof