Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Integrating Meta-Features with Knowledge Graph Embeddings for Meta-Learning

Antonis Klironomos, Ioannis Dasoulas, Francesco Periti, Mohamed Gad-Elrab, Heiko Paulheim, Anastasia Dimou, Evgeny Kharlamov

Abstract

The vast collection of machine learning records available on the web presents a significant opportunity for meta-learning, where past experiments are leveraged to improve performance. Two crucial meta-learning tasks are pipeline performance estimation (PPE), which predicts pipeline performance on target datasets, and dataset performance-based similarity estimation (DPSE), which identifies datasets with similar performance patterns. Existing approaches primarily rely on dataset meta-features (e.g., number of instances, class entropy, etc.) to represent datasets numerically and approximate these meta-learning tasks. However, these approaches often overlook the wealth of past experimental results and pipeline metadata available. This limits their ability to capture dataset - pipeline interactions that reveal performance similarity patterns. In this work, we propose KGmetaSP, a knowledge-graph-embeddings approach that leverages existing experiment data to capture these interactions and improve both PPE and DPSE. We represent datasets and pipelines within a unified knowledge graph (KG) and derive embeddings that support pipeline-agnostic meta-models for PPE and distance-based retrieval for DPSE. To validate our approach, we construct a large-scale benchmark comprising 144,177 OpenML experiments, enabling a rich cross-dataset evaluation. KGmetaSP enables accurate PPE using a single pipeline-agnostic meta-model and improves DPSE over baselines. The proposed KGmetaSP, KG, and benchmark are released, establishing a new reference point for meta-learning and demonstrating how consolidating open experiment data into a unified KG advances the field.

Integrating Meta-Features with Knowledge Graph Embeddings for Meta-Learning

Abstract

The vast collection of machine learning records available on the web presents a significant opportunity for meta-learning, where past experiments are leveraged to improve performance. Two crucial meta-learning tasks are pipeline performance estimation (PPE), which predicts pipeline performance on target datasets, and dataset performance-based similarity estimation (DPSE), which identifies datasets with similar performance patterns. Existing approaches primarily rely on dataset meta-features (e.g., number of instances, class entropy, etc.) to represent datasets numerically and approximate these meta-learning tasks. However, these approaches often overlook the wealth of past experimental results and pipeline metadata available. This limits their ability to capture dataset - pipeline interactions that reveal performance similarity patterns. In this work, we propose KGmetaSP, a knowledge-graph-embeddings approach that leverages existing experiment data to capture these interactions and improve both PPE and DPSE. We represent datasets and pipelines within a unified knowledge graph (KG) and derive embeddings that support pipeline-agnostic meta-models for PPE and distance-based retrieval for DPSE. To validate our approach, we construct a large-scale benchmark comprising 144,177 OpenML experiments, enabling a rich cross-dataset evaluation. KGmetaSP enables accurate PPE using a single pipeline-agnostic meta-model and improves DPSE over baselines. The proposed KGmetaSP, KG, and benchmark are released, establishing a new reference point for meta-learning and demonstrating how consolidating open experiment data into a unified KG advances the field.
Paper Structure (15 sections, 1 equation, 7 figures, 10 tables)

This paper contains 15 sections, 1 equation, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Problem Description
  4. Knowledge Graph Embeddings for Meta-Learning
  5. MetaExe-KG: A Knowledge Graph of Datasets And Pipelines
  6. KGmetaSP: Knowledge Graph Embeddings for Meta-Learning
  7. Experiments
  8. Experimental Setup
  9. Evaluation Results
  10. Conclusion
  12. Implementation Details and Additional Experiments
  13. Experimental Configuration
  14. Analysis of Computational Cost
  15. Additional Experiments

Figures (7)

  • Figure 1: Comparison of the information leveraged by our proposed approach (green) versus existing approaches (orange) for representing datasets in meta-learning. For a pair of datasets, A and B, existing approaches represent datasets using only intrinsic meta-feature vectors (top-right). Our proposed approach also includes past pipeline configurations metadata (top left) as additional context into a KG, to represent datasets.
  • Figure 2: KGmetaSP overview.
  • Figure 3: ML pipeline configuration example represented as an Exe-KG.
  • Figure 4: High-level view of MetaExe-KG.
  • Figure 5: Embedding aggregation process.
  • ...and 2 more figures