Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Principles for Open Data Curation: A Case Study with the New York City 311 Service Request Data

David Tussey, Jun Yan

TL;DR

This paper tackles the challenge of making open government data reliable and usable by diagnosing data curation failures in a high-volume municipal dataset. Using the NYC 311 Service Request data from 2022–2023 as a case study, it identifies structural issues, missing data, and domain-value problems that hinder cross-agency analyses and downstream analytics. It then proposes a concrete set of data curation principles—across consistency, accuracy, storage efficiency, dictionary maintenance, automated QA, and transparency—and actionable steps to implement them. The work demonstrates that systematic curation improves data reliability and utility for policy, research, and civic applications, with broad relevance to other municipal datasets and cross-domain datasets used in public sector analytics.

Abstract

In the early 21st century, the open data movement began to transform societies and governments by promoting transparency, innovation, and public engagement. The City of New York (NYC) has been at the forefront of this movement since the enactment of the Open Data Law in 2012, creating the NYC Open Data portal. The portal currently hosts 2,700 datasets, serving as a crucial resource for research across various domains, including health, urban development, and transportation. However, the effective use of open data relies heavily on data quality and usability, challenges that remain insufficiently addressed in the literature. This paper examines these challenges via a case study of the NYC 311 Service Request dataset, identifying key issues in data validity, consistency, and curation efficiency. We propose a set of data curation principles, tailored for government-released open data, to address these challenges. Our findings highlight the importance of harmonized field definitions, streamlined storage, and automated quality checks, offering practical guidelines for improving the reliability and utility of open datasets.

Principles for Open Data Curation: A Case Study with the New York City 311 Service Request Data

TL;DR

This paper tackles the challenge of making open government data reliable and usable by diagnosing data curation failures in a high-volume municipal dataset. Using the NYC 311 Service Request data from 2022–2023 as a case study, it identifies structural issues, missing data, and domain-value problems that hinder cross-agency analyses and downstream analytics. It then proposes a concrete set of data curation principles—across consistency, accuracy, storage efficiency, dictionary maintenance, automated QA, and transparency—and actionable steps to implement them. The work demonstrates that systematic curation improves data reliability and utility for policy, research, and civic applications, with broad relevance to other municipal datasets and cross-domain datasets used in public sector analytics.

Abstract

In the early 21st century, the open data movement began to transform societies and governments by promoting transparency, innovation, and public engagement. The City of New York (NYC) has been at the forefront of this movement since the enactment of the Open Data Law in 2012, creating the NYC Open Data portal. The portal currently hosts 2,700 datasets, serving as a crucial resource for research across various domains, including health, urban development, and transportation. However, the effective use of open data relies heavily on data quality and usability, challenges that remain insufficiently addressed in the literature. This paper examines these challenges via a case study of the NYC 311 Service Request dataset, identifying key issues in data validity, consistency, and curation efficiency. We propose a set of data curation principles, tailored for government-released open data, to address these challenges. Our findings highlight the importance of harmonized field definitions, streamlined storage, and automated quality checks, offering practical guidelines for improving the reliability and utility of open datasets.

Paper Structure

This paper contains 39 sections, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. NYC 311 Service Request (SR) Data
  3. Data Cleansing Issues
  4. Structural Issues
  5. Undocumented Fields:
  6. Representation of Missing data:
  7. Challenges with Daylight Saving Time (DST):
  8. Missing Data by Field
  9. Validating Data for Acceptable Values
  10. Zip Codes:
  11. Created and Closed Dates:
  12. Closed before Created (Negative Duration):
  13. Identical Created & Closed Dates (Zero Durations):
  14. Created or Closed at Midnight:
  15. Resolution Action Update Date
  16. ...and 24 more sections

Figures (5)

  • Figure 1: 311 SR counts by agency for the time period 2022--2023.
  • Figure 2: Top 20 complaint types for the 311 SR during 2022--2023.
  • Figure 3: Number and percentage of empty/blank entries for the 311 SR data during 2022--2023.
  • Figure 4: Violin plots for negative SR durations and post-closed resolution updates (2022--2023.
  • Figure 5: 311 SRs Created & Closed exactly on the hour (hh:00:00) for 2022--2023.