Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Statistical Validation of Column Matching in the Database Schema Evolution of the Brazilian Public School Census

Muriki G. Yamanaka, Diogo H. de Almeida, Paulo R. Lisboa de Almeida, Simone Dominico, Leticia M. Peres, Marcos S. Sunye, Eduardo C. de Almeida

TL;DR

The paper tackles integrating evolving open data releases of Brazil's School Census into the LDE database, where annual schema changes pose consistency challenges. It introduces a data-driven, goodness-of-fit–based methodology using Kolmogorov–Smirnov, Anderson–Darling, Welch's $t$-test, and the $F$-test to compare column distributions across years, paired with a base-schema–driven Schema Matching Algorithm that classifies mappings as identical, new, or data-missing. Empirical results on 12 years of LDE data show high matching performance, with Top-1 accuracy around $0.85$ (KS) and Top-3 around $0.90$ (KS), though accumulated-year comparisons exhibit some drops due to policy-driven and historical data shifts. The approach reduces manual validation workload for domain experts and improves the reliability of schema evolution in public educational data, enabling more robust longitudinal analyses and policy insights.

Abstract

Publicly available datasets are subject to new versions, with each new version potentially reflecting changes to the data. These changes may involve adding or removing attributes, changing data types, and modifying values or their semantics. Integrating these datasets into a database poses a significant challenge: how to keep track of the evolving database schema while incorporating different versions of the data sources? This paper presents a statistical methodology to validate the integration of 12 years of open-access datasets from Brazil's School Census, with a new version of the datasets released annually by the Brazilian Ministry of Education (MEC). We employ various statistical tests to find matching attributes between datasets from a specific year and their potential equivalents in datasets from later years. The results show that by using the Kolmogorov-Smirnov test we can successfully match columns from different dataset versions in about 90% of cases.

Statistical Validation of Column Matching in the Database Schema Evolution of the Brazilian Public School Census

TL;DR

The paper tackles integrating evolving open data releases of Brazil's School Census into the LDE database, where annual schema changes pose consistency challenges. It introduces a data-driven, goodness-of-fit–based methodology using Kolmogorov–Smirnov, Anderson–Darling, Welch's -test, and the -test to compare column distributions across years, paired with a base-schema–driven Schema Matching Algorithm that classifies mappings as identical, new, or data-missing. Empirical results on 12 years of LDE data show high matching performance, with Top-1 accuracy around (KS) and Top-3 around (KS), though accumulated-year comparisons exhibit some drops due to policy-driven and historical data shifts. The approach reduces manual validation workload for domain experts and improves the reliability of schema evolution in public educational data, enabling more robust longitudinal analyses and policy insights.

Abstract

Publicly available datasets are subject to new versions, with each new version potentially reflecting changes to the data. These changes may involve adding or removing attributes, changing data types, and modifying values or their semantics. Integrating these datasets into a database poses a significant challenge: how to keep track of the evolving database schema while incorporating different versions of the data sources? This paper presents a statistical methodology to validate the integration of 12 years of open-access datasets from Brazil's School Census, with a new version of the datasets released annually by the Brazilian Ministry of Education (MEC). We employ various statistical tests to find matching attributes between datasets from a specific year and their potential equivalents in datasets from later years. The results show that by using the Kolmogorov-Smirnov test we can successfully match columns from different dataset versions in about 90% of cases.
Paper Structure (11 sections, 6 equations, 1 figure, 3 tables, 1 algorithm)

This paper contains 11 sections, 6 equations, 1 figure, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Goodness-of-fit Schema Evolution Methodology
  4. Goodness-of-Fit
  5. The Schema Matching Algorithm
  6. Experimental Results
  7. Experimental Protocol
  8. Results -- Matches Considering the Previous Year
  9. Results -- Matches Considering the Accumulated Years
  10. Related Work
  11. Conclusion and Future Work

Figures (1)

  • Figure 1: Illustration of schema evolution showing the data file headers from 2018 and 2020, as well as the impact of header changes on the integrated schema. Arrows indicate the mappings. Columns are presented in their original Portuguese names.