Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Infinite-Dimensional Feature Interaction

Chenhui Xu, Fuxun Yu, Maoliang Li, Zihao Zheng, Zirui Xu, Jinjun Xiong, Xiang Chen

TL;DR

This work introduces InfiNet, a model architecture that enables feature interaction within an infinite-dimensional space created by RBF kernel and achieves new state-of-the-art status, owing to its capability to leverage infinite-dimensional interactions, significantly enhancing model performance.

Abstract

The past neural network design has largely focused on feature representation space dimension and its capacity scaling (e.g., width, depth), but overlooked the feature interaction space scaling. Recent advancements have shown shifted focus towards element-wise multiplication to facilitate higher-dimensional feature interaction space for better information transformation. Despite this progress, multiplications predominantly capture low-order interactions, thus remaining confined to a finite-dimensional interaction space. To transcend this limitation, classic kernel methods emerge as a promising solution to engage features in an infinite-dimensional space. We introduce InfiNet, a model architecture that enables feature interaction within an infinite-dimensional space created by RBF kernel. Our experiments reveal that InfiNet achieves new state-of-the-art, owing to its capability to leverage infinite-dimensional interactions, significantly enhancing model performance.

Infinite-Dimensional Feature Interaction

TL;DR

This work introduces InfiNet, a model architecture that enables feature interaction within an infinite-dimensional space created by RBF kernel and achieves new state-of-the-art status, owing to its capability to leverage infinite-dimensional interactions, significantly enhancing model performance.

Abstract

The past neural network design has largely focused on feature representation space dimension and its capacity scaling (e.g., width, depth), but overlooked the feature interaction space scaling. Recent advancements have shown shifted focus towards element-wise multiplication to facilitate higher-dimensional feature interaction space for better information transformation. Despite this progress, multiplications predominantly capture low-order interactions, thus remaining confined to a finite-dimensional interaction space. To transcend this limitation, classic kernel methods emerge as a promising solution to engage features in an infinite-dimensional space. We introduce InfiNet, a model architecture that enables feature interaction within an infinite-dimensional space created by RBF kernel. Our experiments reveal that InfiNet achieves new state-of-the-art, owing to its capability to leverage infinite-dimensional interactions, significantly enhancing model performance.
Paper Structure (21 sections, 14 equations, 4 figures, 5 tables)

This paper contains 21 sections, 14 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. From Feature Representation Space to Interaction Space
  4. Feature Interaction Space
  5. Dimension of Feature Interaction Space
  6. Expanding Interaction Space to Infinite Dimension
  7. Expanding Interaction Space with Reproducing Kernel
  8. Infinite-Dimensional Feature Interaction with RBF Kernel
  9. Demo Case Performance of Models on Different Feature Space
  10. Method
  11. InfiBlock: Infinite-Dimensional Spatial Feature Interaction
  12. Model Architectures
  13. Experiments
  14. ImageNet Classification
  15. MS COCO Detection
  16. ...and 6 more sections

Figures (4)

  • Figure 1: (a) Traditional feature representation without interaction woo2023convnexthe2016deep. (b) Recent work with finite feature interaction rao2022hornetxu2022quadralib. (c) Our method: Kernel-enabled infinite feature interaction.
  • Figure 2: Comparison of simple representation, finite interaction, and infinite-dimensional interaction. The ? circle in DemoBlock is chosen from element-wise Add, element-wise Mul. or RBF kernel.
  • Figure 3: Overview of InfiNet. (a) Four-stage hierarchical InfiNet design. (b) InfiBlock Design
  • Figure 4: Visualization Comparison of (1) Feature Representation Space model, (2) Finite Feature Interaction Space model, (3) Infinite-Dimensional Feature Interaction model