Tag: Open Data @ NYC Planning

In this session we will discuss the problems that Data Engineering at the Department of City Planning encountered managing datasets and introduce the open source tooling that we’ve built to manage metadata, generate documentation, enforce data quality, and automate distribution of data to platforms, with a focus specifically on NYC Open Data.

This talk is aimed primarily at those who have an interest in automating some or all of the above. We will walk through how we, at City Planning, catalog our dataset metadata; how that metadata is used to generate READMES, data dictionaries, and other metadata files; how to leverage metadata for automated QA; how to automate distribution of data to destinations like the Tyler/Socrata open data platform, databases, FTP servers, and data lakes; and finally how interested developers can make use of our framework and potentially contribute code of their own.

This presentation is part of the Open Data @ NYC Planning event series.

Click here to RSVP for virtual attendance.

Click the blue “Going” button below to RSVP for in-person attendance at the Department of City Planning’s offices (120 Broadway, New York, NY 10271).

As data analysts and engineers know, quality source data is crucial to sound analyses and healthy pipelines. It can also take a lot of time, effort, and resources to wrangle. Data Engineering at the Department of City Planning has written a new tool (python module/CLI) to manage data extraction and archival.

In this session, we will show potential users how they might simplify and automate extracting data from external sources like NYC Open Data and ArcGIS Online. We will touch on some of the built-in features of the tool as well as where we’re going: simple data cleaning, automatic geocoding, data validation, and more.

This presentation is part of the Open Data @ NYC Planning event series.

Click here to RSVP for virtual attendance.

Click the blue “Going” button below to RSVP for in-person attendance at the Department of City Planning’s offices (120 Broadway, New York, NY 10271).

NYC Planning has developed the Fast Tracker app to allow users to determine whether planned housing projects are eligible for the new citywide Green Fast Track rule. The rule streamlines housing production by allowing projects of a specific typology to simplify the environmental review process, while satisfying state and city environmental standards. The app integrates ESRI Experience Builder with Survey123 and Microsoft Power Automate so users can enter project criteria and determine eligibility under the rule. We will discuss developing the app in tandem with the rule, under a strict timeline and scope. This presentation will focus on how the app was developed, how data was collected, reviewed, and processed for inclusion in the app. We will also talk about how the app and data pipelines are maintained.

This presentation is part of the Open Data @ NYC Planning event series.

Click here to RSVP for virtual attendance.

Click the blue “Going” button below to RSVP for in-person attendance at the Department of City Planning’s offices (120 Broadway, New York, NY 10271).

Comparing data from different census years can be a challenge due to changes in geographies. When those data are Public Use Microdata Sample (PUMS), it may be difficult to know where to start. In this presentation, Donnise Hurley from the NYC Department of City Planning will demonstrate step-by-step how to access PUMS data using the Census API, prepare data for analysis, harmonize older data into the 2020 PUMAs (approximate NYC’s Community Districts boundaries), and make an interactive map using a few lines of code. Attendees will gain a basic understanding of PUMS data, learn how to calculate margins of error and use them to create statistically reliable map categories, and learn data wrangling techniques.  All analyses will be conducted in R statistical software, but the techniques presented are transferrable to other programs.

This presentation is part of the Open Data @ NYC Planning event series.

Click here to RSVP for virtual attendance.

Click the blue “Going” button below to RSVP for in-person attendance at the Department of City Planning’s offices (120 Broadway, New York, NY 10271).