Behind the Scenes: HPD's Housing New York Open Data
What/Who is HPD?
The New York City Department of Housing Preservation and Development (HPD) publishes nearly 40 open datasets on the city's Open Data Portal. These datasets highlight the work we do every day to meet our agency mission: to promote the quality and affordability of the city's housing and the strength and diversity of its many neighborhoods. HPD accomplishes its objectives through four key services:
• Preserving affordable housing and protecting tenants
• Developing new affordable housing
• Enforcing the Housing Maintenance Code
• Engaging neighborhoods in planning
While we publish Open Data related to a variety of our programs and services, this blog post will focus on the datasets related to the Mayor's Housing New York plan.
What is Housing New York?
Chart based on Affordability Levels Table in Housing New York by the Numbers https://www1.nyc.gov/site/hpd/about/housing-new-york-by-the-numbers.page
HPD is the agency responsible for leading Housing New York (HNY), Mayor de Blasio's plan to create or preserve 300,000 affordable homes by 2026. HNY is the most ambitious housing plan in the country; 300,000 units are enough homes to house the entire population of Seattle or Boston.
Over the course of this 12-year plan, the goal is to preserve 60% and create (or build) 40% of the… [more]
Guest Post by DEP - Measuring Harbor Water Quality: Using Data to Show Success
This blogpost originally appeared here.
Since the passage of the Clean Water Act of 1972, we've been hard at work building and upgrading wastewater treatment plants to ensure the city's daily 1.3 billion gallons of wastewater is treated to the highest standards and that the cleanest, most biologically similar water is released back into New York Harbor. We've also taken that a step further by constructing miles of new sewers to increase capacity, planting acres of wetlands to serve as a natural filter and wildlife habitat, and installing thousands of green infrastructure assets across the city to curb the release of pollution into our waterways.
So, we know why New York Harbor is cleaner than it has been in more than a century, but, how do we know it's so clean?
Whales!
Well, yes, whales are definitely an indication. Whales, and a number of aquatic species, are being drawn back to New York in numbers we haven't seen in decades, thanks to a newly burgeoning food supply that is flourishing in cleaner waters. But we can also prove that our water is cleaner thanks to data gathered by DEP scientists who conduct the New York Harbor Survey, an ambient water quality monitoring program that… [more]
Apply: NYC Open Data Queens Borough Ambassadors
Many of you have kindly asked us in the past “How can I help with NYC Open Data efforts?” and we have an opportunity for you!
The NYC Open Data Team is looking for 10-15 volunteers to become NYC Open Data Queens Borough Ambassadors from August - November 2019 as a part of a pilot programming initiative called Data Counts in partnership with the Queens Public Library and civic technology nonprofit BetaNYC. Apply by next Tuesday July 16th via this form.
The goals of this pilot as a part of the Data Counts Program are to: (1) pilot a train-the-trainer approach to scaling up efforts at teaching communities about NYC Open Data and (2) to raise awareness about Census 2020. Specifically, Borough Ambassadors will:
Attend a training led by BetaNYC on 07/22 from 5:30-7:30pm to learn how to lead an Open Data 101 community training in Long Island City (Queens)Lead a minimum of three 90-minute community trainings in Queens Public Library branches between August - November 2019
To be a successful Borough Ambassador, you will need to be able to attend the training on 07/22 mentioned above, pass BetaNYC’s certification criteria for providing Open Data 101 community trainings and lead three 90-minute community trainings… [more]
311 Open Data Changes
UPDATE as of 8:30 AM EST 07/09/19: The 311 Service Request and 311 Call Inquiry datasets are both being refreshed daily as of Sunday July 7th. The blog post below was updated at 8:30 AM EST 07/09/19 to reflect the most accurate updates to the 311 datasets on NYC Open Data.
Many of you are familiar with NYC311, a New York City service that provides access to non-emergency City services and information about City government programs. NYC311 is available online and by phone (dial 311).
The NYC311 system underwent a system upgrade over the June 29-30 weekend, which has resulted in changes to the following three 311 open data datasets:
311 Service Requests from 2010 to Present
311 Call Center Inquiry
311 Web Content Services
These datasets paused all updates from June 27-July 4 2019, and were refreshed on Friday July 5th with another refresh on Sunday July 7th. These datasets will have a few changes as of the system upgrade which are outlined below.
We realize these changes could have downstream effects on any applications or analyses connected to these datasets. If you have any questions or concerns, please contact us via the NYC Open Data Help Desk.
We will be exploring potential updates to these… [more]
NYC Open Data Week 2019 Recap & Summer Announcements
Engaging all five boroughs, thousands of New Yorkers
The NYC Open Data team, in collaboration with local civic tech nonprofit BetaNYC, held the third annual NYC Open Data Week from March 1-9, 2019. This was our largest NYC Open Data Week yet: more than 65 New Yorkers and organizations helped plan 48 events across all five boroughs, bringing out over 3,000 New Yorkers to learn about NYC Open Data and how they can get involved. Online, web traffic to nyc.gov/opendata surged 20% over an average week. According to a post-event survey, participants ranged from ages 18 to 65+ and came from every corner of New York City, as well as Long Island, upstate New York, New Jersey, Pittsburgh, and even London.
For the first time, NYC Open Data Week included events in every borough, allowing New Yorkers from all walks of life to see open data in action and inspiring them to create civic change in their own neighborhoods. Event highlights include School of Data, an annual day-long community conference produced by BetaNYC and attended by over 400 people; an event featuring the Speculation Watch List, developed by the NYC Department of Housing Preservation and Development to predict where tenants may be… [more]
Accessing Data from the NYC Administration for Children's Services
NYC Open Data is an opportunity to engage New Yorkers in the information that is produced and used by City government. In an effort to provide even more transparency to New Yorkers, the NYC Administration for Children's Services (ACS) website now contains a hyperlink to the Open Data site so that one can access our most recently published statistics and reports with just the click of a mouse.
Data is an extremely important and powerful tool; at ACS, data allows us to make continuous improvements so that we deliver the highest quality services to our city's children and families. From finding innovative ways to build outreach to the communities and families we serve, to examining and working towards ways to strengthen current processes, using data is critical. Data can even reveal stories and help us gain insight into what's happening.
We are proud that the Open Data site provides critical information about ACS and the children and families we serve throughout the City. For instance, through Open Data, New Yorkers can easily find the number of NYC children in each community district and borough served in prevention services. New Yorkers can also access the list of prevention services providers ACS works with throughout… [more]
2018 Examination and Verification - Request for Public Comment on SBS, DOT, & BIC Dataset Inventories
Local Law 8 of 2016 requires the Mayor's Office of Data Analytics to examine three agencies annually to verify their compliance with the NYC Open Data Law. The law also requires MODA to "accept suggestions from the public as to possible public data sets within mayoral agencies that have not yet been disclosed."
The Open Data Examination and Verification Survey is based on Local Law 8 of 2016 and is intended to help City agencies uncover datasets that may not already be on the Open Data Portal (ODP) or scheduled for future release in their Open Data Plans. In 2018, the Department of Small Business Services (SBS), Department of Transportation (DOT), and Business Integrity Commission (BIC) were required to complete this survey, which includes:
-- Agency Overview: High-level description of the agency, the types of data it collects, and technical systems
-- Current Inventory: Inventory and data quality questionnaire for datasets already on the Open Data Portal
-- MMR: Inventory of Mayor's Management Report (MMR) indicators and underlying datasets
-- Reports: Inventory of legally mandated reports and underlying datasets
-- Shared: Inventory of datasets shared between agencies for operational purposes
-- FOIL: Inventory of datasets used to respond to freedom of information law (FOIL) requests
-- Websites: Inventory… [more]
NYC Open Data Announces Winners of 1st Annual Citywide Competition!
In case you missed it, the winners of the Open Data citywide competition were announced! Talented New Yorkers from all over the city submitted projects based on one or more of the City's 2,000+ Open Data sets.
This exciting Competition gives New Yorkers the opportunity to be featured on the new Open Data Project Gallery receive a Certificate of Recognition from the Mayor. First place winners will also be invited to present at an upcoming NYC Open Data Event.
The NYC Open Data team today announced four first place winners of the NYC Open Data Project Gallery Contest:
Data Science Award
For the most compelling data analysis using at least one dataset from NYC Open Data
WINNER: Plan(t)wise: planning a green canopy over the Big Apple
Submitted by Niki Athanasiadou.
This project shows which trees can grow the best in NYC based on growth history and survival.
Most Creative Award
For the most compelling visual analysis using at least one dataset from NYC Open Data
WINNER: An Interactive Visualization of Street Trees
Submitted by Allen Yee of cloudred.
This creative project shows the variety and quantity of street trees in all five boroughs. Ever wonder what the most/least common tree in NYC is? Well this project… [more]
Rescheduled Open Data Week Events From March 7th
Rescheduled Open Data Week Events from March 7th
We just made it past the first day of Spring (March 21st!) just to land in another snowstorm...?!?
Speaking of snowstorms, we wanted to make sure you were all aware of events rescheduled from our Open Data Week's snowstorm on March 7th. We hope you'll be able to join us!
Wednesday March 28th
The Missing Product Manager: Panel discussion of different digital City products
12:30-2pm
Free event
Hosts: Department of City Planning & Civic Hall
Register here
Open Data Showcase: Open Data Workshop using python (for beginners!)
4:30-6pm
Free event
Hosts: Hunter College
Register here
Wednesday April 25th
Be in the Know: NYC Open Data for Youth: Open Data Workshop (for beginners!)
5-7pm
Free event
Hosts: NYC Service | Mayor's Office
Register here
Thursday April 5th
Talk Data To Me: Data Science Project Showcase
6:30-8:30pm
Free event
Hosts: General Assembly
Register here
Wednesday April 25th
Using and Improving NYC Open Data Dictionaries: Workshop for librarians engaging with the NYC Open Data Team
1-5pm
Free event
Hosts: METRO, Tiny Panther
Register here
New York City Celebrates Open Data Week 2018
March 3 through March 10, more than 30 events for new yorkers to explore the use & power of NYC Open Data
NYC Open Data Week 2018 is a week-long celebration to raise awareness of the City's public data. Through March 10, New Yorkers are invited to experience more than 30 events, exhibits, panels, and workshops across the city that explore how NYC Open Data is being leveraged by New Yorkers. Events include a data art exhibition, a demo of a new platform to identify risk to affordable housing in Brooklyn, a tour of a data exhibit at the Museum of the City of New York, workshops for student entrepreneurs on how to use open data to build their business, and more. The full schedule is available here. The City's Open Data Portal, visited 75,000 times each month on average, allows New Yorkers to access nearly 2,000 free municipal datasets, ranging from 311 complaints to crime incidents by neighborhood to the location of every street tree in the city.
"A fair city is an open city. NYC Open Data puts the data we use to make decisions in government back in the hands of all New Yorkers. Every day, New Yorkers in all… [more]
Data Collections
The NYC Open Data portal recently added a new "asset type" called Data Collection. Most City data is stored in relational databases and up until now we did not have a good way of surfacing these to the portal. A relational database is a set of tables related to each other through primary and foreign keys. A primary key uniquely identifies every record in that table. A foreign key refers back to the primary key in another table. In this way the tables are linked and make sense when displayed as a group or Collection rather than individually.
Housing Development Projects Receiving City Funding (Local Law 44) is the first Data Collection on the NYC Open Data portal.
The Department of Housing Preservation and Development (HPD) is required by law to publish data relating to housing projects receiving funding from the city. HPD's Performance Management and Analytics team initially published the required information by putting the tables into a downloadable zipped folder on the NYC Open Data portal, but wanted a more accessible and user-friendly format to display this data. From that, the idea of a Data Collection was initiated.
This Collection contains 11 data tables, all prefixed with Local Law 44.… [more]
January 2018 Policy Updates from the NYC Open Data team
nycdoitt:
In 2017, the Open Data program enjoyed the spotlight at three City Council hearings as lawmakers, advocates, and the de Blasio administration worked together to craft new legislation to sustain the open data program into perpetuity. In addition, the Open Data program implemented new data quality and documentation policies to comply with previous amendments to the Open Data Law. Below is a summary of these updates.
Improvements to Data Quality
2017 marked the beginning of a holistic "clean-up" of the Open Data Portal's dataset inventory. To create a better user experience, we have begun to remove certain datasets, improve the search function, standardize geospatial fields across datasets, and document each dataset's metadata in data dictionaries.
Dataset Removal and Improvements to Search
Our new dataset removal policy applies to data that does not qualify as a "public dataset" according to the Open Data Law.
A dataset will be removed from public access when the agency owner and the NYC Open Data team agree that it does not legally qualify under the law. The dataset will then be listed in the public "Dataset Removals" dataset, which contains its name, agency, hyperlink, and reason for removal. A copy of the dataset will… [more]
NYC Open Data Week is back: March 3-10, 2018
nycdoitt:
Enjoy chatting with the NYC Open Data Team? Or...perhaps you're merely Open-Data-Curious?
To raise awareness about NYC Open Data--a free data resource!--last year the NYC Open Data Team partnered with the civic technology community to produce Open Data Week 2017, which engaged over 900 New Yorkers! We're now asking for submissions of ideas for Open Data Week 2018 and hope you'll share an idea. The deadline for submissions is December 15th. Email: aschmoeker@analytics.nyc.gov with any questions, you can also learn more via our coverage in StateScoop a few weeks ago!
For inspiration, here are some great sessions and concepts from last year:
Data Jam / Hackathon
DataKind, 92Y and Bill & Melinda Gates Foundation: Giving Tuesday DataDive.
Two day event engaging data scientists to help brainstorm ways to increase the impact of #GivingTuesday
Panel Discussion
General Assembly Panel Discussion: Data and...Health.
General Assembly brought together a panel of experts and influencers from the health and wellness spaces to discuss how big data is impacting their organizations.
Workshop
NYC Parks Computer Resource Centers Open Data for All: TreesCount! Workshop.
Free workshop presented by NYC Parks and the NYC Open Data team offered a broad introduction to the NYC Open Data Portal… [more]
Open Data for All 2017 Progress Report and Compliance Plan
Since the passage of Local Law 11 of 2012, the Open Data team has issued an annual report each July to inform New Yorkers about the progress we're making - both in complying with the law and in engaging New Yorkers to use Open Data in new, creative ways.
On July 14, we released our Open Data for All 2017 Progress Report and Compliance Plan. For the first time, we were able to publish it as an interactive website. In the report, we highlight our accomplishments over the past year, feature vignettes about everyday users, and update our compliance plan. We want to hear from our users. To provide comment on the report itself, or to tell us your own Open Data story, please comment directly through the interactive website here and scroll to the bottom of the page. We're accepting feedback through the website through September 1. Users may contact us at any time here.
In addition to requesting comment online, the Open Data team periodically hosts events to connect with our users. On Wednesday, July 19, the Open Data team hosted its Summer Open Data Updates event at Civic Hall, where we presented a deeper look into the report, showcased… [more]
Isn't celebrating fun?
It's been five years since New York City signed the strongest open data law in the country. We've been busy ever since, and celebrated a LOT in honor of our 5-year anniversary…
We launched a pretty new beta website: www.nyc.gov/opendata (send us any feedback here) There's a new way for you to contact us if you have questions, comments, concerns: www.nyc.gov/opendata/engage/ ; some folks have already noticed the difference...!Our partners at the Department of City Planning launched their new Facilitates Explorer tool - check it out: https://capitalplanning.nyc.gov900+ people participated in NYC's First NYC Open Data Week in 12 events across three boroughsWe taught 15 New Yorkers were taught how to use NYC Open Data via our first NYC Open Data for All: TreesCount! Workshop; the first 1-day data-literacy program developed by the City of New York with NYC Open Data as a core teaching toolOur NYC Chief Analytics Officer, Dr. Amen Ra Mashariki, announced at the Socrata Connect Conference that he will no longer give speeches on NYC Open Data. Instead, our team will find New Yorkers who have been impacted by open data to speak to its promise.A few folks said nice things about our work (thanks!) and the future of…
[more]New Website!
You may have noticed that NYC Open Data has a new look! We're excited about our new website and hope you'll explore and let us know what you think.
The site includes a new "Contact Us" page, which will serve as a one-stop shop for any feedback you have for the NYC Open Data Team. The page includes a Screendoor-enabled tool that will allow us to respond to your questions, data requests, and any other inquiries in a more timely manner.
In order to streamline the collection of inquiries, we have worked closely with Socrata to make the following changes:
Dataset nominations: We disabled the old nominations page and are routing all new dataset requests to the Contact Us page. You can still track the status of previously made requests here. We have removed the response to these dataset nominations from this dataset as the responses were always customer-service related in nature and did not provide insight into the data itself.Comments: We disabled the ability to post new comments on specific datasets in the Socrata platform. We realized that users were posting questions here, which made it difficult for us to quickly respond to. As a result, we are funneling inquiries to the…
[more]Breathing City
Joey Cherdarchuk of Dark Horse Analytics first posted this creative use of Open Data in 2014. Several datasets, including this building footprints shapefile, the Department of City Planning's Bytes of the Big Apple, and census data (here and here), were used to create this mesmerizing graphic representing the ebb and flow of working New Yorkers in Manhattan.
Seen any similar analyses using this type of data? Let us know on Twitter or Facebook!
Open Data for All at the International Open Data Conference
Back in October, the Government of Spain, World Bank, and Open Data for Development Network hosted the 4th International Open Data Conference (IODC) in Madrid. Last week, they published the International Open Data Roadmap, which represents the collective vision of the best and brightest Open Data leaders across the globe.
Dr. Amen Ra Mashariki, New York City Chief Analytics Officer and head of the NYC Open Data initiative, presented at the IODC on how the City works toward Open Data for All - the idea that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker.
This strategy, Dr. Mashariki said, centers around six core values:
Start with Users: The first step in opening data is focusing on demand. We support creative analytic thinking to enable users to answer the questions most important to them and also make efforts to spur demand in communities less familiar to Open Data.Treat the publication of the dataset as its debut: Though a lot of work goes into making a dataset ready to go live, our work isn't done once that dataset is published. Opening data requires that it is updated, errors are identified, and users are able to engage…
[more]One Dataset, Three Ways
UPDATE: A third use has been identified.
Last April, DoITT released a three-dimensional (3-D) Building Massing Model of New York City, available for download on DoITT's website and the Open Data Portal. The 3-D model was captured from the 2014 aerial survey. Aerial surveys are conducted every two years to produce digital orthophotography, or aerial photography that can be accurately scaled and used for maps. The data contains every NYC building present when that survey was taken.
Providing this data to the public allows users to creatively visualize New York City's iconic skyline, conduct analyses and build applications. We are familiar with three separate uses of this data:
NYC Building Shadow Mapping: To commemorate the Winter Solstice in December, New York Times contributors analyzed the building shadows of the entire City. Interactive maps at different times of the year - Winter, Spring/Fall, and Summer - allow readers to explore the amount of time each NYC building spends in shadow.Cesium Map: Cesium is an open-source library for 3D globes and maps. One user created an interactive map of New York City, visualizing New York City's over 1 million buildings as 3D models. Map users may explore this map by address or landmark, and buildings…
[more]For-Hire Vehicle Data
Today, the Taxi and Limousine Commission (TLC) held a public hearing on a new rule to address the risks of fatigued driving and add trip reporting requirements for For-Hire Vehicle (FHV) bases.
This rule would require FHV bases, including app-based services such as Uber and Lyft, to report pick-up and drop-off times and locations for all trips. This would allow TLC to verify that drivers are limiting the number of hours they spend on the road to avoid fatigued driving. If this rule passes, the data will be published to NYC Open Data, similarly to the data for green and yellow taxi cab trips that is already on Open Data.
Amen Ra Mashariki, Chief Analytics Officer and Director of the Mayor's Office of Data Analytics, testified before the TLC Board of Commissioners today on how Open Data in general - and TLC data in particular - benefits all New Yorkers. Read Dr. Mashariki's testimony and follow updates on the proposed new rules at nyc.gov/taxi.
End of Year Updates from the Open Data Team
2016 has been a productive year in the world of Open Data! Our Open Data team, comprised of the Department of Information Technology and Telecommunications and the Mayor's Office of Data Analytics, want to share our incredible progress over the past year.
Progress on Open Data for All
In July 2015, we committed to Open Data For All, an inclusive vision in which all New Yorkers, not just the tech-savvy, can find value in New York City's data. We believe that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker.
Open Data for All means ALL New Yorkers: Analytics underlies Open Data: we use analytics to assess who Open Data is reaching - and who it's not. We recently partnered with students from New York University's Center for Urban Science and Progress to publish "Reducing Data Poverty in NYC: Achieving Open Data for All." The researchers used Structural Equation Modeling (SEM) to measure which factors contribute to lack of Open Data use in certain communities. This study suggests that "data representation" - when data captures some aspect of the data user or their community - is associated with higher use of the Open Data…
[more]Curious about the allocation of tree species around New York
City, Brooklyn-based web developer and designer Jill Hubley took a deep dive
into the results of the 2005 Street Tree Census to see what arboreal
patterns took shape in neighborhoods across the five boroughs
Curious about the allocation of tree species around New York City, Brooklyn-based web developer and designer Jill Hubley took a deep dive into the results of the 2005 Street Tree Census to  see what arboreal patterns took shape in neighborhoods across the five boroughs. The map Jill created – both kaleidoscopic and visually intriguing – depicts the distribution and biodiversity of the city’s street trees. Users can filter by species and zoom in for a detailed view of streets.
The map also reflects the results of careful planting considerations by the New York City Parks Department. Site selection takes into account several factors to determine what trees are a good fit for a particular area including site condition, overhead clearance, tree bed width, and biological diversity. Diseases and pests that target particular types of trees make varied plantings around the City a necessity.
Click through and get a detailed perspective on the botany of your neighborhood!
NYC Wi-Fi Hotspot Locations | NYC Open Data
Visit New York City's Open Data portal to see a map of public wifi hotspots across the five boroughs, including City parks, subway stations, payphones, and facilities of the three public library systems. The data also informs the NYCityMap, an interactive visualization of City services, cultural institutions, programs, projects, and other features.
New York City Is Home To More Than 65 Different
New York City is home to more than 65 different types of street trees ranging from the Alder to Zelkova. With over 600,000 street trees across the five boroughs, one might wonder where they are and which types are most common. Using NYC Open Data of street trees, a few local designers decided to find out… According to their analysis, Queens is home to a third of the city's street trees and the Maple and Plane Tree (Sycamore) are the most common.
To learn more, check out their Interactive Visualization of NYC Street Trees.
Interested in NYC tree planting? Visit MillionTreesNYC
New York City Becomes Even More Beautiful When The
New York City becomes even more beautiful when the sun begins to set. Every day, there is a moment when the sun fits perfectly between buildings, so that rays of light reflect among the skyscrapers.
Twice a year, when the sun aligns with the east-west midtown street grid it creates a phenomenon known as "Manhattanhenge." Yet it's possible to experience henge events during every sunset all over the city. Last year, using data from NYC Open Street Maps (OSM) and various technologies, folks at CartoDB created a map that locates every "NYChenge" that occurs in New York City every single day.
Explore NYCHenge
