Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feature Network Methods in Machine Learning and Applications

Xinying Mu, Mark Kon

TL;DR

The paper reframes ML feature vectors as functions on a feature network $G=(V,W)$ and leverages graph signal processing and graph Laplacian regularization to uncover structure in high-dimensional feature spaces. It introduces deep hierarchical feature networks built by recursive clustering, enabling graph-convolution-like pooling and multiple learning architectures, including average pooling and SVM-based feature propagation. A graph-Sobolev framework via the graph Laplacian, including the smoothness discriminant and multi-layer smoothness features, provides principled regularization and discriminative power, demonstrated on gene expression and cancer datasets. Overall, the work offers a versatile, geometry-aware approach to feature engineering and learning that generalizes CNN concepts to arbitrary graphs and exploits prior relational structures for improved predictive performance.

Abstract

A machine learning (ML) feature network is a graph that connects ML features in learning tasks based on their similarity. This network representation allows us to view feature vectors as functions on the network. By leveraging function operations from Fourier analysis and from functional analysis, one can easily generate new and novel features, making use of the graph structure imposed on the feature vectors. Such network structures have previously been studied implicitly in image processing and computational biology. We thus describe feature networks as graph structures imposed on feature vectors, and provide applications in machine learning. One application involves graph-based generalizations of convolutional neural networks, involving structured deep learning with hierarchical representations of features that have varying depth or complexity. This extends also to learning algorithms that are able to generate useful new multilevel features. Additionally, we discuss the use of feature networks to engineer new features, which can enhance the expressiveness of the model. We give a specific example of a deep tree-structured feature network, where hierarchical connections are formed through feature clustering and feed-forward learning. This results in low learning complexity and computational efficiency. Unlike "standard" neural features which are limited to modulated (thresholded) linear combinations of adjacent ones, feature networks offer more general feedforward dependencies among features. For example, radial basis functions or graph structure-based dependencies between features can be utilized.

Feature Network Methods in Machine Learning and Applications

TL;DR

The paper reframes ML feature vectors as functions on a feature network and leverages graph signal processing and graph Laplacian regularization to uncover structure in high-dimensional feature spaces. It introduces deep hierarchical feature networks built by recursive clustering, enabling graph-convolution-like pooling and multiple learning architectures, including average pooling and SVM-based feature propagation. A graph-Sobolev framework via the graph Laplacian, including the smoothness discriminant and multi-layer smoothness features, provides principled regularization and discriminative power, demonstrated on gene expression and cancer datasets. Overall, the work offers a versatile, geometry-aware approach to feature engineering and learning that generalizes CNN concepts to arbitrary graphs and exploits prior relational structures for improved predictive performance.

Abstract

A machine learning (ML) feature network is a graph that connects ML features in learning tasks based on their similarity. This network representation allows us to view feature vectors as functions on the network. By leveraging function operations from Fourier analysis and from functional analysis, one can easily generate new and novel features, making use of the graph structure imposed on the feature vectors. Such network structures have previously been studied implicitly in image processing and computational biology. We thus describe feature networks as graph structures imposed on feature vectors, and provide applications in machine learning. One application involves graph-based generalizations of convolutional neural networks, involving structured deep learning with hierarchical representations of features that have varying depth or complexity. This extends also to learning algorithms that are able to generate useful new multilevel features. Additionally, we discuss the use of feature networks to engineer new features, which can enhance the expressiveness of the model. We give a specific example of a deep tree-structured feature network, where hierarchical connections are formed through feature clustering and feed-forward learning. This results in low learning complexity and computational efficiency. Unlike "standard" neural features which are limited to modulated (thresholded) linear combinations of adjacent ones, feature networks offer more general feedforward dependencies among features. For example, radial basis functions or graph structure-based dependencies between features can be utilized.
Paper Structure (16 sections, 9 equations, 2 figures, 7 tables)

This paper contains 16 sections, 9 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Feature networks and functions on feature networks
  3. Deep learning with feature networks - an example in image processing
  4. Deep features: Generalizations of deep and convolutional neural network structures
  5. Construction of a feature network
  6. Network components
  7. Deep hierarchical feature networks
  8. Tree Structured Networks
  9. Average pooling
  10. Deep learning with SVM bagging
  11. Graph Laplacian Methods
  12. Introduction to the smoothness discriminant
  13. Multiple smoothness features via network clustering
  14. Feature vector regularization
  15. Discussion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Deep hierarchical structure for undirected Graph $G$ with two levels of clustering, i.e., two layers in the network learning.
  • Figure 2: Classification performance with one layer average pooling for two cancer datasets. The red line in the graph indicates the benchmark performance by SVM with linear kernel.