Rescheduled Open Data Week Events from March 7th
We just made it past the first day of Spring (March 21st!) just to land in another snowstorm…?!?
Speaking of snowstorms, we wanted to make sure you were all aware of events rescheduled from our Open Data Week’s snowstorm on March 7th. We hope you’ll be able to join us!
Wednesday March 28th
The Missing Product Manager: Panel discussion of different digital City products
Hosts: Department of City Planning & Civic Hall
Open Data Showcase: Open Data Workshop using python (for beginners!)
Hosts: Hunter College
Wednesday April 25th
Be in the Know: NYC Open Data for Youth: Open Data Workshop (for beginners!)
Hosts: NYC Service | Mayor’s Office
Thursday April 5th
Talk Data To Me: Data Science Project Showcase
Hosts: General Assembly
Wednesday April 25th
Using and Improving NYC Open Data Dictionaries: Workshop for librarians engaging with the NYC Open Data Team
Hosts: METRO, Tiny Panther
New York City Celebrates Open Data Week 2018
March 3 through March 10, more than 30 events for new yorkers to explore the use & power of NYC Open Data
NYC Open Data Week 2018 is a week-long celebration to raise awareness of the City’s public data. Through March 10, New Yorkers are invited to experience more than 30 events, exhibits, panels, and workshops across the city that explore how NYC Open Data is being leveraged by New Yorkers. Events include a data art exhibition, a demo of a new platform to identify risk to affordable housing in Brooklyn, a tour of a data exhibit at the Museum of the City of New York, workshops for student entrepreneurs on how to use open data to build their business, and more. The full schedule is available here. The City’s Open Data Portal, visited 75,000 times each month on average, allows New Yorkers to access nearly 2,000 free municipal datasets, ranging from 311 complaints to crime incidents by neighborhood to the location of every street tree in the city.
“A fair city is an open city. NYC Open Data puts the data we use to make decisions in government back in the hands of all New Yorkers. Every day, New Yorkers in all five boroughs use open data to improve their communities,” said Mayor Bill de Blasio. “Open Data Week is about highlighting those stories and giving all New Yorkers inspiration to make a difference.”
“The NYC Open Data Portal is a powerful tool that ensures transparency and fosters civic innovation within our City to help improve the quality of life for all New Yorkers,” said Samir Saini, Commissioner of the Department of Information Technology and Telecommunications.
“While New York City has impressive open data stats to boast: nearly 2,000 published data assets and 20,000 visitors to the site per week on average - much of its value happens behind the scenes in making our government more data-driven,” said Emily W. Newman, Acting Director of the Mayor’s Office of Operations. “Congratulations to the Mayor’s Office of Data Analytics and DoITT–they make it possible for NYC Open Data to engage dozens of City agencies and City-affiliated organizations. Open Data is transforming the way our City thinks about data-sharing and the power of analytics to drive change for New Yorkers.”
“Since committing to Open Data for All in 2015, we have dismissed the idea ‘If you build it they will come’ and taken efforts to engage more New Yorkers than ever in the data created by their City,” said Adrienne Schmoeker, Director of Civic Engagement & Strategy at the Mayor’s Office of Data Analytics. “We are thrilled to celebrate Open Data Week with all our partners who make transparent government possible.”
Highlights of Open Data Week include:
- Staten Island (Saturday, March 10): Our Stories: The Soul of Data is organized by First Star College of Staten Island (CSI) Academy with the Administration of Children’s Services; the free four-hour workshop will empower 20+ foster youth with the tools to find and use NYC Open Data.
- Manhattan (Saturday March 10): Open Data L-Train Innovation Challenge is a day-long event organized by Forum for the Future, Collectively and Grand Central Tech calling designers and civic technologists for a day of problem-solving.
- Brooklyn (Thursday, March 8): Identify Affordable Housing Risks with Data at Brooklyn Borough Hall will unveil a new web portal to host housing data from multiple sources, allowing organizations to share, validate, and bolster their findings and research to show trends and threats to affordable housing in Brooklyn. Hosted by Brooklyn Borough President Eric Adams and OpenGov.
In celebration of International Open Data Day on March 3, the City is launching the Open Data Project Gallery which shows 5 examples of how data has been used to address urban problems. This new feature was inspired by Open Data Week 2017, designed and prototyped by the NYC firm Fahrenheit 212 and developed by the Department of Information Technology and Telecommunications Gov Lab & Studio. The City is also offering a chance for New Yorkers and international users to enter their own projects into the running for the following awards, which will be judged by a panel of experts from the City, WNYC, SAVI at the Pratt Institute, NYC Tech Alliance, and Technical.ly Brooklyn:
- Mayor’s Civics Award
- Data Science Award
- Open Data Award
- Most Creative Award
- NYC Innovator Award
The contest officially launches on March 3 and runs through May 1, see the Open Data Website for more details: nyc.gov/opendata.
“Data was born to be free, and NYC’s Open Data program ensures it lives that way” said Miguel Gamiño, Jr., New York City Chief Technology Officer. “Democratization of data makes it possible for any entrepreneur and startup to access one of the most valuable ingredients for building technologies that serve the public and make technology work for all people.”
“TLC is proud to celebrate Open Data Week and the sixth anniversary of the City’s Open Data Law!” said TLC Commissioner Meera Joshi. “Since the law passed in 2012, there has been massive growth in the for-hire industries in NYC, and TLC has expanded data reporting requirements to understand the effects of this growth and to better regulate services, making much of this granular trip data available to the public. TLC goes beyond the law’s requirements and has identified new metrics that are most useful for the public in understanding how taxis, app-based services, and traditional for-hire services operate in NYC, allowing the public to see firsthand the growth and its effects, no matter a person’s technical skills with data.”
“With NYC Open Data Week, the City has created a unique opportunity for New Yorkers to step into the world of data,” said Gregg Bishop, Commissioner of the New York City Department of Small Business Services. “Not only is the City opening up data for all but, given the upward mobility that data careers offer, we’re providing free skills training and job connection to open up opportunity for New Yorkers in an increasingly tech-driven economy.”
“NYC Service is thrilled to engage our Youth Leadership Councils in Open Data Week,” said NYC Chief Service Officer Paula Gavin. “Community data is a foundation for civic engagement and the Open Data platform is an important tool that encourages our City’s youth to better understand their neighborhoods in new ways, allowing them access to information that can be used to improve policy and practice in all five boroughs.”
“Our Open Data Law has given rise to a dazzling constellation of new apps, research projects, and even businesses aimed at improving New Yorkers’ lives through the creative use of public datasets,” said Manhattan Borough President Gale A. Brewer. “Happy Open Data Week to all the civic hackers out there building things with public data, and thank you to the Mayor’s Office of Data Analytics, DoITT, DoRIS, and the public servants at every city agency working to put government data to work for the people we serve.”
“I believe nearly every sector of our municipal government would be enhanced by a better utilization and activation of civic data, through real-time monitoring, enhanced public accountability, and dynamic agency response. My administration will continue to champion technological advances like these that improve the City’s ability to dig into the crevices of persistent challenges facing New Yorkers. I’m pleased that Brooklyn Borough Hall is part of Open Data Week, and I thank the Mayor’s Office of Data Analytics for organizing this important initiative,” said Brooklyn Borough President Eric L. Adams.
Council Member Peter Koo, Chair of the Committee on Technology said, “Open Data Week demonstrates NYC’s commitment to a transparent, accountable and less-cynical government that takes pride in giving people access to information. Today, accessible data is being used by everyone from students to professionals in an effort to influence policy and change in their communities, so it is with great excitement that we ask for this year’s participants to step up their game, and take full advantage of this tremendous opportunity to create new innovations with NYC’s data, and to create a better city for everyone.”
“New York City’s Open Data policy puts us at the forefront of giving residents the information they need to make a difference,” said Council Member Ben Kallos. “Since its implementation, the Open Data Portal has bred countless new laws, apps, and other solutions to improve access to city services for all New Yorkers. Thank you to Mayor de Blasio for leading the way on Open Data for All and using Open Data Week to highlight new innovative ways of presenting public information to empower all New Yorkers.”
The City of New York passed the Open Data Law in 2012. Since then, every City agency has contributed datasets to the portal, with more being added every year. In 2015, Mayor de Blasio introduced Open Data for All, a vision to maximize New Yorkers’ engagement with City data. The Portal was subsequently relaunched in 2017 with a more user-friendly design allowing novices and experienced data researchers alike to find and use the valuable information it offers. Since relaunch, the Open Data Portal has seen record use, recording more than 75,000 average visitors per month. The Mayor’s Office of Data Analytics (MODA) and the City’s Department of Information Technology and Telecommunications (DoITT) partner to form the Open Data team.
Open Data Week was inspired, and is supported by, BetaNYC, a civic tech organization based in New York City. They are the organizers of two Open Data Week events, School of Data and Unlocking Open Data for Community Boards with the Manhattan Borough President’s Office.
“After 6 years and 7 additional laws, New York City has nearly 2,000 public data assets from more than 50 different municipal publishers - truly the world’s greatest open data program. All worthwhile work takes time, energy and determination - while the road hasn’t been easy it has been worth it! It is a pleasure to be a part of the NYC Open Data community and an honor to kick off the second NYC Open Data Week,” said Noel Hidalgo, Executive Director of BetaNYC. “The BetaNYC community is looking forward to an amazing week of storytelling and workshops. We are fortunate to have an Administration, Council, community groups, and individuals who are will to dive beyond the data points and collaborate to make this City work for all of us. The BetaNYC community looks forward to a fantastic Open Data Week. We hope to see all of you at NYC’s community data conference.”
“Once again New York is leading the tech industry by demonstrating that open data serves the public interest and spurs innovation in both government and private endeavors,” said Andrew Rasiej, Chairman of the NY Tech Alliance and CEO of Civic Hall. “Now Six Years after New York City’s landmark open data law was passed Open Data Week confirms that both government and citizens are using information that makes government more effective and accountable to the citizens it serves.”
“Allowing New York’s innovators access to the wealth of data collected by NYC agencies creates opportunities for startups, large companies, and technologists everywhere. It gives our entrepreneurs the chance to not only solve public-facing problems, but also the tools to build businesses that matter,” said Julie Samuels, Executive Director, Tech:NYC.
“Six years ago, New York City put a plan in motion to leverage its data as a strategic asset. We at Socrata have been fortunate to be part of that journey, helping to make that data not just open but also accessible and usable to all. New York City has created an ecosystem that fosters startups and economic activity around the city, makes the city government more accountable, supports a vibrant civic community, and improves the daily digital experience of New Yorkers. The success of their program is a blueprint that so many other cities are emulating,” said Kevin Merritt, CEO, Socrata.
“The Museum of the City of New York is thrilled to celebrate Open Data Week by offering curator led tours of our data driven interactive gallery exploring the future of New York City. The Future City Lab, part of our signature New York at Its Core exhibition and home to the world’s largest visual display of data about the city, is made possible only by New York City’s commitment to open data and embodies the idea that meeting the challenges of the future requires access to information in the present. We are thrilled to open our doors to all those who are as excited as we are about Open Data Week.” said Whitney Donhauser, Ronay Menschel Director of the Museum of the City of New York.
“New York City has set a gold standard for public access to data through initiatives like Open Data for All, the Open Data Portal, and Open Data Week. At Rentlogic, we support the administration’s commitment to making data easy to understand, using it to improve the quality of life for all New Yorkers, and leveraging it to bring fairness and equity to our City’s housing market,” said Rentlogic CEO, Yale Fox. “Choosing where to live is one of the biggest decisions a person makes, and we’re proud that Rentlogic has brought transparency to a difficult process in a simple, user-friendly platform for the public good.”
“At Forum for the Future, we design open and collaborative strategies for a more sustainable world, and being part of New York City Open Data Week amplifies our ability to do this. Together with the Department of Transportation, Dell, Grand Central Tech, and other partners, we are organizing a hackathon that is bringing the public and private sectors together to collaborate around the L-train shutdown sustainability implications for the city. We are thrilled to be working with the Mayor’s Office of Data Analytics and other organizations that are committed to using and sharing open data to create positive impact. We are grateful for all of the work and support of the NYC Open Data Team to make our innovation challenge and NYC Open Data Week possible,” said Rodrigo Bautista, Principal Change Designer at Forum for the Future.
“Fahrenheit 212 is proud to have partnered with the City of New York and the Mayor’s Office of Data Analytics on NYC Open Data. As part of our commitment to using innovation for civic and social benefit, we consider NYC Open Data an invaluable resource and fuel for innovation to improve the lives of our neighbors, communities and local businesses. We share in the Mayor’s Office of Data Analytics’ goal to accelerate New York City’s innovation engine. Partnering with NYC Open Data has been a great way for us to contribute to that mission,” said Todd Rovak, CEO of Fahrenheit 212.
data and open government are not a point-in-time event, but an ongoing
process. In 2012 Local law 11 marked a
huge milestone for open data in New York and served as a model for cities
across the globe. Six years later, at a time when public trust in our democracy
is as shaken as it’s ever been, today’s events offer a powerful renewal of New
York’s ongoing commitment to transparent and accountable government. From
efforts to ensure more robust oversight and compliance to initiatives to make
sure data is not just available to some but useful
to all, New York has demonstrated a willingness to work toward that future.
There is more work to be done to show not just New Yorkers, but the entire
globe a positive, more open vision of the future of American democracy. Let’s
all hope the City of New York remains up to the task, and can serve as a model
once more in the coming year,” said
Stephen Larrick, Director of Open Cities, Sunlight Foundation.
“Data isn’t just
data. It’s people’s lives, businesses, and homes; it’s the stories of who we
are and how we live. Reboot is honored to partner with New York City, and
committed to sharing the lessons from working with this pioneering
administration with other cities in the US and around the globe. As we seek to
research and design new ways to bring data to life for New Yorkers, we are
thrilled to see NYC taking the bold next step of moving beyond opening data to
solving problems and changing lives with data,” said Zack Brisson, Principal at Reboot.
Assembly is a proud supporter of New York City’s Open Data Week. Our
instructors leverage NYC Open Data datasets in our data analytics and data
science programs, and they have been an invaluable resource for our students
who are hungry for real-life data they can use in their projects and
portfolios,“ said Tom Ogletree,
Director of Social Impact and External Affairs at General Assembly.
"This is our second year participating in Open Data Week, and we are proud
of the many initiatives General Assembly is working on with the City to promote
access to technology training and tools for all New Yorkers.”
excited to participate in the OpenData Week and showcase our use of OpenData
across the Hunter College curriculum,” said
Katherine St. John, professor of computer science at Hunter College. “We look forward to having attendees at
our event explore and visualize NYC OpenData and all that it offers.”
Innovation Center at City College is thrilled to be part of Open Data Week. Our
March 6th Summit
in Harlem will feature both hands-on instruction on how to use NYC Open Data to
tackle local problems, and will celebrate the diversity of tech founders here
in NYC. We support the many ways in which the Mayor’s Office is making
technology innovation accessible through initiatives like Open Data!” said Lindsay Siegel, Executive Director of
the Zahn Innovation Center at The City College of New York, City University of
New York (CUNY).
represents opportunity. We are thankful for the ongoing dedication and effort
from municipalities like New York for expanding the footprint of available open
data. There is much more work to be done- and thanks to the efforts of
visionary leaders who see beyond what is in place today to continue to expand
the scope, breadth and ease of access to open data to serve all constituents.
For companies like Vizalytics, who serve public and private sector clients in
the US, Canada, Europe and Australia, open data is critical to our business
model. When we first began our work in 2012, it was NYC Open Data that was
foundational to our success, and we are grateful for being past NYC Big Apps
winners, as well as working with NYC Mayor’s Office of Tech and Innovation on neighborhoods.nyc,” said
Aileen Gemma Smith, CEO Vizalytics Technology.
innovation consultancy Luminary Labs is hosting a panel discussion, “Correlate & Innovate: using
non-traditional data sets for innovation,” at its office during Open Data
Week. “From healthcare and finance to transportation and government,
organizations are increasingly embracing open data in the pursuit of
innovation,” said CEO Sara Holoubek of
Luminary Labs. “Open Data Week creates a space and time for these important
“We applaud the City of New York’s continued
efforts to increase not only access to data but also to educate users about
data through enhancements such as the NYC Open Data Portal metadata screens and
data dictionaries. This, along with endeavors like the NYC Planning Lab’s
human-centered design approach, significantly increases the understanding and
impact we and our community partners are able to achieve,” said Jessie Braden, Director of Pratt
Institute’s Spatial Analysis and Visualization Initiative.
“New York City’s
commitment to open data has promoted and fostered operational efficiency within
and across agencies, but also more informed and engaged citizens,” said Jim Barry, Esri developer network
program manager. “Much of this city’s open data has some location component
to it, and on Monday, March 5, during NYC Open Data Week, Esri will be hosting
a hands-on workshop, showing how to use “Insights for ArcGIS” to
combine, analyze, and visualize NYC open data on maps and charts as actionable
“We’re proud of our partnership with the
NYC Mayor’s Office of Data Analytics and New York City nonprofits where
together we enable New Yorkers to put Open Data to good use in their
communities. CARTO’s community and grants programs empower open data users to analyze,
visualize, and act on civic data in new ways,” said Tyler Bird, Community Lead at CARTO.
“We’re excited to have the Open
Contracting Partnership engage as a part of Open Data Week and to begin
exploring public contracting in New York City. We look forward to the
opportunity to dig into NYC’s open data infrastructure and explore ways to
implement an open data contracting standard to give everyone a way to track
spending from planning through to public contracts with companies and onto the
delivery of services. Continuing to drive the values of NYC Open Data through
to the contracting process will help businesses to access new opportunities and
create a level playing field. It will help concerned New Yorkers follow public
money and strengthen NYC’s open government efforts,” said Open Contracting Partnership’s Executive Director, Gavin Hayman.
Most Open Data Week events are free of charge and open to the public. Visit
open-data.nyc for details and event schedules.
The NYC Open Data portal recently added a new
“asset type” called Data Collection. Most City data is stored in relational
databases and up until now we did not have a good way of surfacing these to the
portal. A relational database is a set of tables related to each other through
primary and foreign keys. A primary key uniquely identifies every record in
that table. A foreign key refers back to the primary key in another table. In
this way the tables are linked and make sense when displayed as a group or
Collection rather than individually.
Housing Development Projects Receiving City
Funding (Local Law 44) is the first Data Collection on the NYC Open Data portal.
The Department of Housing Preservation and
Development (HPD) is required by law to publish data relating to housing
projects receiving funding from the city. HPD’s Performance Management and
Analytics team initially published the required information by putting the
tables into a downloadable zipped folder on the NYC Open Data portal, but
wanted a more accessible and user-friendly format to display this data. From
that, the idea of a Data Collection was initiated.
This Collection contains 11 data tables, all
prefixed with Local Law 44. The main table is the Project table, with ProjectID
as its primary key. Each project can have one or more buildings, these are
captured in the Building data table. The ProjectID field in the Building table
refers back to the ProjectID from the Project table. Likewise, each of the
other tables also have a ProjectID field that refers back to the Project table
(see image below). In addition to the way the tables are displayed online, the
Data Dictionary is combined into a single document,
making it easier to see how the tables are connected.
For instance, if you were interested in
buildings receiving financial assistance that were completed in the past year,
you would find the ProjectIDs in the Project table that fall within the last
year according to the ActualCompletionDate field and then go to the Buildings
table to pick out the buildings with those ProjectIDs. The tables can be
combined in various ways depending on the question you have or analysis you’re
looking to do.
Other datasets on the Open Data portal that
could benefit from being surfaced as a Data Collection include ACRIS and OATH
Hearings Division Case Status (formerly ECB violations).
ACRIS is a database of property records and
documents. It currently lives on the Open Data portal in 15 separate datasets.
This is an example of a relational database on NYC Open Data that could benefit
from more robust documentation and presentation on how these datasets relate to
each other. Someone may by chance find one of the datasets and not realize that
the information they are looking for exists within a different ACRIS dataset.
OATH Hearing Division Case Status dataset
contains information on the outcome of hearings relating to ECB
violations. It is currently presented on
NYC Open Data as one merged dataset originating from two data tables in the
source system: notice of violations and charges. Each violation can have one or
more charges. It is joined on the portal as a single dataset, where each row is
a violation and the charges are tacked on as fields. While this format is
useful for some purposes, it is a challenge to deconstruct it into a list of
The Open Data Team is working with Socrata to
make Data Collection a Socrata asset type, which will make it searchable in the
catalog as an entity (currently only the individual tables are searchable) and
give each Collection a combined primer page as opposed to separate primer pages
for each data table. If you have additional ideas on how to improve the
usability of relational data on NYC Open Data get in touch with us via “Contact
Us” and send your idea via a General Inquiry, we’d love to hear from you.
January 2018 Policy Updates from the NYC Open Data team
In 2017, the Open Data program enjoyed the spotlight at three City Council hearings as lawmakers, advocates, and the de Blasio administration worked together to craft new legislation to sustain the open data program into perpetuity. In addition, the Open Data program implemented new data quality and documentation policies to comply with previous amendments to the Open Data Law. Below is a summary of these updates.
Improvements to Data Quality
2017 marked the beginning of a holistic “clean-up” of the Open Data Portal’s dataset inventory. To create a better user experience, we have begun to remove certain datasets, improve the search function, standardize geospatial fields across datasets, and document each dataset’s metadata in data dictionaries.
Dataset Removal and Improvements to Search
Our new dataset removal policy applies to data that does not qualify as a “public dataset” according to the Open Data Law.
A dataset will be removed from public access when the agency owner and the NYC Open Data team agree that it does not legally qualify under the law. The dataset will then be listed in the public “Dataset Removals” dataset, which contains its name, agency, hyperlink, and reason for removal. A copy of the dataset will be retained for three months after it is removed from public view, after which point the data will be permanently archived. In this three month “grace” period, a user may object to the removal of a dataset by contacting the Open Data team at opendata.cityofnewyork.com/engage. We will consult with data owners and public records officers at the relevant agency before making a final determination on removing the dataset.
In general, datasets considered for removal are infrequently accessed by public users, not regularly updated, and not actively maintained by the agency data owner. Users have complained that these datasets “clutter” the catalog. Their removal will make it easier to search for and find relevant, high-quality datasets.
In addition, beginning this spring, user-created “Community” views will be removed from the search function on the Open Data catalog. Community views are created when a user filters or visualizes an “Official” dataset from a City agency and saves it to their NYC Open Data account. Note that your existing community filters will still be accessible at the same links – they will just not appear when using the platform’s search function.
Geocoding Street Addresses
To make it easier for datasets with geospatial fields to be compared or combined, we standardized geospatial attributes associated with street addresses. Every dataset containing a street address is now required to also include fields with its latitude and longitude, neighborhood details, political districts, and other fields. Additional details on this standard can be found in section 126.96.36.199 of the Technical Standards Manual.
DoITT leveraged the Department of City Planning’s GeoSupport geocoding tool, which pairs columns containing street address data with data attributes required by the new standard. Users may also use the GeoSupport tool to geocode datasets themselves through the Geoclient API. Geoclient is a RESTful web service interface to the Geosupport system developed by DoITT’s GIS/Mapping unit.
A record of datasets eligible for the geocoding standard, along with datasets that have already been geocoded, is maintained in the 2017 NYC Open Data Plan - Address Standardization dataset. Most eligible datasets have already been standardized. Currently, DoITT is adding geocoding to the “automation” workflows for datasets that are automatically updated.
After a thorough effort to document definitions for data fields for all datasets last year, most datasets now have data dictionaries. The data dictionary not only provides definitions on data attributes but also gives context on how and why the data is collected. You can track which data dictionaries are still in progress on the Data Dictionary Compliance Public Assets dataset. If you find a data dictionary that could use further clarification, tell us about it. We will follow up with the agency and let them know.
In December, Local Law 244 of 2017 and Local Law 251 of 2017 became law, extending the duration of the Open Data Law and creating new annual reporting requirements. We would like to thank the Committee on Technology and Chair Vacca for their unwavering support of the Open Data initiative. The Open Data policy that was borne from the Law is unparalleled among American municipalities, and the amendments the Committee has passed over the last three years will ensure that the program thrives into the future.
Extension of Open Data Mandate
The Open Data Law requires City agencies to publish all public datasets by December 31, 2018. New legislation requires datasets created after this deadline to be published, extending the Open Data mandate into perpetuity. Agencies that have already identified datasets in their Open Data plans are still required to publish them by the end of this year.
Technical Standards Manual
The Technical Standards Manual is the foundational document for the Open Data program, containing information on technical specifications and policy. New legislation requires that every two years, we conduct a thorough review and update the document. The update will begin late this year and include community feedback.
Agency open data coordinator
The head of each City agency is now required to officially name one employee as its Open Data Coordinator, the agency’s main liaison for data publishing and responding to public inquiries about that agency’s datasets. While almost every City agency already has an existing open data coordinator, the addition of ODCs to the administrative code will help ensure the role has sufficient visibility and resources from agency leadership.
Web portal site analytics
Later this year, we will publish information on user traffic, including numbers on pageviews and users who access the portal. This information is already included at data.cityofnewyork.us/analytics, and the Open Data team will make usage data more easily understandable this year.
Annual Open Data Plan
For the past four years, the annual Open Data plan has been published each July 15th. Going forward, the plan will be published every September 15th to align Open Data compliance reporting requirements with other reporting requirements centered around the City’s fiscal year (July 1 - June 30).
Starting this year, the Open Data plan will include comprehensive information on each dataset on the open data portal, including the dataset’s:
1. Scheduled publication date
2. Actual publication date
3. Most recent update date
5. Whether it complies data retention standard (which mandates that row-level data be maintained on the dataset)
6. Whether it has a data dictionary
7. Whether it meets the geocoding standard, does not meet the geocoding, or is ineligible for the geospatial standard
8. Whether updates to the dataset are automated;
9. Whether updates to the dataset “feasibly can be automated,” and if not, a reason why
In addition, agencies will now list the names of datasets that agency records officers use to respond to public records requests. This builds on previous legislation that required agencies to report metrics on datasets used to respond to FOIL requests in their annual compliance plans and helps ensure that Open Data Coordinators work closely with their legal affairs and public records personnel.
MODA Examination and Verification
Each year, the Mayor’s Office of Data Analytics is required by Local Law 8 of 2016 to conduct an Open Data Examination and Verification (E&V) of three city agencies. The purpose of the process is twofold: it allows MODA to critically examine three specific City agencies’ data inventories and also holds up a mirror to the NYC Open Data program at large.
In 2016, MODA examined Department of Sanitation (DSNY), Department of Correction (DOC), and Department of Housing Preservation and Development (HPD). On December 1, 2017, MODA released the results of the 2017 E&V cycle, which examined the Department of Environmental Protection (DEP), the Fire Department (FDNY), and the Department of Buildings (DOB). The results of the examination and verification, as well as MODA’s recommendations on how to improve citywide compliance with the Open Data Law, can be found in the “Reports” section of the Open Data site.
The 2018 Examination and Verification report will cover the Business Integrity Commission (BIC), Department of Small Business Services (SBS), and Department of Transportation, and will be released on December 1, 2018.
NYC Open Data Week is back: March 3-10, 2018
with the NYC Open Data Team? Or…perhaps you’re merely Open-Data-Curious?
awareness about NYC Open Data—a free data
resource!—last year the NYC Open Data Team
partnered with the civic technology community to produce Open
Data Week 2017, which
engaged over 900 New Yorkers! We’re now asking for submissions of ideas for Open
Data Week 2018 and hope
you’ll share an idea. The deadline for submissions is December 15th. Email: firstname.lastname@example.org with any questions, you can also learn more via our coverage
in StateScoop a few weeks ago!
For inspiration, here are some great sessions and concepts
from last year:
Data Jam / Hackathon
DataKind, 92Y and Bill & Melinda Gates Foundation: Giving Tuesday DataDive.
Two day event engaging data scientists to help brainstorm ways to increase the
impact of #GivingTuesday
General Assembly Panel Discussion: Data and…Health.
General Assembly brought together a panel of experts
and influencers from the health and wellness spaces to discuss how big data is
impacting their organizations.
NYC Parks Computer Resource Centers Open Data
for All: TreesCount! Workshop.
Free workshop presented by NYC Parks and the NYC Open
Data team offered a broad introduction to the NYC Open Data Portal along with
the concept of data literacy and analysis using NYC TressCount! Data, which is
the most accurate map of NYC’s street trees ever created.
NYC Big Apps: NYC Open Data Portal & Department of City Planning Facilities
Free workshop for NYC Big
Apps participants to learn about these tools as they work to build out solutions
as a part of the annual Big Apps competition.
Prime Produce & Startup M/IG: Open Data Dinner
Prime Produce and Startup M/IG teamed up to host a
dinner discussion introducing their communities in a process of prototyping
what a more aesthetically influenced public policy would look like for NYC.
Reaktor Open Data Studio.
Reaktor hosted a happy hour to share ideas about how
Open Data could be utilized in new ways.
Made in NY Media Center + Fabernovel Data &
Media: Open Data Breakfast.
Made in NY Media Center teamed up with FaberNovel to
host an interactive breakfast for developers, agency and civil service
non-profits to explore using open data to build products and conduct research
& analysis to create new applications.
Department of Small Business Services: 2017 Smart
Inaugural NYC Smart Districts Summit, where community
and technology leaders collaboratively explored how emerging technologies are
being leveraged to address the most pressing district-level challenges.
BetaNYC School of Data
Community conference showcasing NYC’s civic design, civic/government technology
and open data ecosystem.
Civic Hall Presents: Open Data, Mapping Global
Security & the Department of Defense
Civic Hall teamed up with the National
Geospatial-Intelligence Agency (NGA) to present a demo of its geospatial data
portals to address how we can get national security data into the open.
Civic Hall Presents: Demo
of CALC Tool with 18F
18F is a digital innovation team in the federal US
government; CALC is a tool developed by the team to assist contracting officers
and contracting specialists in making informed decisions through market
research and price analysis for labor categories on federal government contracts.
of Staten Island (CSI) Tech Incubator + Vizalytics: Data – A Driving Force of
Vizalytics and the NYC Open Data Team demo’d their platforms then took
questions from the audience.
Open Data for All 2017 Progress Report and Compliance Plan
Since the passage of Local Law 11
of 2012, the Open Data team has issued an annual report each July to inform New
Yorkers about the progress we’re making – both in complying with the law and in
engaging New Yorkers to use Open Data in new, creative ways.
On July 14, we released our Open
Data for All 2017 Progress Report and Compliance Plan. For the first time,
we were able to publish it as an interactive website. In the report, we
highlight our accomplishments over the past year, feature vignettes about
everyday users, and update our compliance plan. We want to hear from our users.
To provide comment on the report itself, or to tell us your own Open Data
story, please comment directly through the interactive website here
and scroll to the bottom of the page. We’re accepting feedback through the
website through September 1. Users may contact us at any time here.
In addition to requesting comment
online, the Open Data team periodically hosts events to connect with our users.
On Wednesday, July 19, the Open Data team hosted its Summer Open Data Updates
event at Civic Hall, where we presented a deeper look into the report,
showcased newly released datasets, and highlighted the types of dataset
requests the team receives. Shane Leese from RentHop and Mary Tobin from the
Brownsville Partnership, both of whom were featured in the report as users of
NYC Open Data, spoke about their Open Data experiences. David Rimshnick from
FactorPrism also presented, demonstrating how to make the best of 311 request
data using his analytics tool. Thanks to the hundreds who attended or
livestreamed the event, and special thanks to both the speakers and Civic Hall
for their help in making the event a great success! If you missed the event,
you can view the recorded livestream in full at https://www.facebook.com/CivicHallNYC/videos/1822368181123039/.
Isn’t celebrating fun?
It’s been five years since New York City signed the strongest open data
law in the country. We’ve been busy ever since, and celebrated a LOT in honor
of our 5-year anniversary…
- We launched a pretty new beta website: www.nyc.gov/opendata (send us any
- There’s a new way for you to contact us if you
have questions, comments, concerns: www.nyc.gov/opendata/engage/ ; some
folks have already noticed the difference…!
- Our partners at the Department of City Planning
launched their new Facilitates Explorer tool – check it out: https://capitalplanning.nyc.gov
- 900+ people participated in NYC’s First NYC Open Data Week in 12 events across three boroughs
- We taught 15 New Yorkers were taught how to use
NYC Open Data via our first NYC Open Data for All:
TreesCount! Workshop; the first 1-day data-literacy program developed by
the City of New York with NYC Open Data as a core teaching tool
- Our NYC Chief Analytics Officer, Dr. Amen Ra
at the Socrata Connect Conference that he will no longer give speeches on
NYC Open Data. Instead, our team will find New Yorkers who have been impacted
by open data to speak to its promise.
- A few folks said nice things about our work
(thanks!) and the future of NYC Open Data: State Scoop, AM
New York, GovTech,
Ash Center, Progrss,
Big Apps NYC 2017, Chris
Chavez via Medium, Carto
All of these wonderful things would
not have been possible without the great support, effort and cheerleading from:
BetaNYC, BureauBlank, Reinvent Albany, DoITT NYC Gov Lab & Studio, NYC Parks
Analytics, NYC Parks Computer Resource Centers, Capital Planning Team at the
Department of City Planning, Office of Digital Strategy, The Department of
Small Business Services, Carto, Microsoft, Socrata, Screendoor, Vizalytics,
General Assembly, Grand Central Tech, Civic Hall, Civic Hall Labs, The College
of Staten Island’s Tech Incubator, The Manhattan Borough President’s Office,
the National Geo-Spatial Intelligence Agency (NGA), 18F, DataKind, NYC Big
Apps, The Made in New York Media Center, Fabernovel Data & Media, Reaktor,
Prime Produce, Startup M/IG, GovLab, Luminary Labs and many others!
A final thank you to all 8.5
million New Yorkers - you inspire us every day to do this work and it
wouldn’t be where it is without you all and our ongoing dialogue.
Stay tuned for more great things to come from NYC Open Data! Also – if you have a
suggestion for something to be featured on our blog, get in touch with us here.
You may have noticed that NYC Open
Data has a new look! We’re excited about our new website and hope
you’ll explore and let us
know what you think.
The site includes a new “Contact
Us” page, which will serve as a one-stop shop for any feedback you have for the
NYC Open Data Team. The page includes a Screendoor-enabled tool that will allow
us to respond to your questions, data requests, and any other inquiries in a
more timely manner.
In order to streamline the
collection of inquiries, we have worked closely with Socrata to make the
- Dataset nominations: We
disabled the old nominations page and are routing all new dataset requests to
Us page. You can still track the status of previously made requests here. We have removed the response to these
dataset nominations from this dataset as the responses were always
customer-service related in nature and did not provide insight into the data
- Comments: We disabled the ability to post
new comments on specific datasets in the Socrata platform. We realized that
users were posting questions here, which made it difficult for us to quickly
respond to. As a result, we are funneling inquiries to the Contact
Us page while we research commenting alternatives to implement in
the future. No current comments will be removed.
If you have any
questions about these changes, please get in touch through the new Contact Us page!
Joey Cherdarchuk of Dark Horse Analytics first posted this creative use of Open
Data in 2014. Several datasets, including this building
footprints shapefile, the Department of City Planning’s Bytes of the
Big Apple, and census data (here and here), were used to create
this mesmerizing graphic representing the ebb and flow of working New Yorkers
Seen any similar analyses using this type of data? Let us
know on Twitter or Facebook!
Open Data for All at the International Open Data Conference
Back in October, the Government of Spain, World Bank, and Open Data for Development Network hosted the 4th International Open Data Conference (IODC) in Madrid. Last week, they published the International Open Data Roadmap, which represents the collective vision of the best and brightest Open Data leaders across the globe.
Dr. Amen Ra Mashariki, New York City Chief Analytics Officer and head of the NYC Open Data initiative, presented at the IODC on how the City works toward Open Data for All – the idea that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker.
This strategy, Dr. Mashariki said, centers around six core values:
- Start with Users: The first step in opening data is focusing on demand. We support creative analytic thinking to enable users to answer the questions most important to them and also make efforts to spur demand in communities less familiar to Open Data.
- Treat the publication of the dataset as its debut: Though a lot of work goes into making a dataset ready to go live, our work isn’t done once that dataset is published. Opening data requires that it is updated, errors are identified, and users are able to engage and provide feedback.
- Encourage purposeful and easy engagement: Open Data is a platform that should be used as a tool for feedback. Substantive engagement is more than an obligation; it must be default for Open Data.
- Empower agencies: By empowering Open Data Coordinators and champions in every agency and office, we can narrow the gap between agency experts and Open Data users.
- Integrate Open Data into citywide processes: As Open Data becomes routine across NYC government, we will look for opportunities to make small changes in existing processes.
- Learn, test, standardize – and learn again: We must test our assumptions and try new ideas, collect information on our efforts, and analyze our performance for points of improvement.
Watch Dr. Mashariki’s full presentation (37 minutes) at IODC2016.
One Dataset, Three Ways
UPDATE: A third use has been identified.
Last April, DoITT released a three-dimensional (3-D)
Building Massing Model of New York City, available for download on DoITT’s
website and the Open
Data Portal. The 3-D model was captured from the 2014 aerial survey. Aerial
surveys are conducted every two years to produce digital orthophotography, or
aerial photography that can be accurately scaled and used for maps. The data
contains every NYC building present when that survey was taken.
Providing this data to the public allows users to creatively
visualize New York City’s iconic skyline, conduct analyses and build
applications. We are familiar with three separate uses of this data:
Building Shadow Mapping: To commemorate the Winter Solstice in December,
New York Times contributors analyzed the building shadows of the entire City.
Interactive maps at different times of the year – Winter, Spring/Fall, and
Summer – allow readers to explore the amount of time each NYC building spends
Map: Cesium is an open-source
library for 3D globes and maps. One user created an interactive map of New York
City, visualizing New York City’s over 1 million buildings as 3D models. Map
users may explore this map by address or landmark, and buildings may be
presented as colors visually representing their heights.
- Esri Map: CoolMaps is a collection of creative maps that use the Esri Mapping Platform. This Vision Zero map uses the 3D model of New York City to represent live traffic, collisions, and traffic injuries and deaths, based on NYC Department of Transportation’s datasets on the Open Data Portal.
Seen any other examples? Let us know! DM us on Twitter or Facebook!
For-Hire Vehicle Data
Today, the Taxi and Limousine Commission (TLC) held a public hearing on a new rule to address the risks of fatigued driving and add trip reporting requirements for For-Hire Vehicle (FHV) bases.
This rule would require FHV bases, including app-based services such as Uber and Lyft, to report pick-up and drop-off times and locations for all trips. This would allow TLC to verify that drivers are limiting the number of hours they spend on the road to avoid fatigued driving. If this rule passes, the data will be published to NYC Open Data, similarly to the data for green and yellow taxi cab trips that is already on Open Data.
Amen Ra Mashariki, Chief Analytics Officer and Director of the Mayor’s Office of Data Analytics, testified before the TLC Board of Commissioners today on how Open Data in general – and TLC data in particular – benefits all New Yorkers. Read Dr. Mashariki’s testimony and follow updates on the proposed new rules at nyc.gov/taxi.
End of Year Updates from the Open Data Team
2016 has been a productive year in the world of Open Data! Our Open Data team, comprised of the Department of Information Technology and Telecommunications and the Mayor’s Office of Data Analytics, want to share our incredible progress over the past year.
Progress on Open Data for All
In July 2015, we committed to Open Data For All, an inclusive vision in which all New Yorkers, not just the tech-savvy, can find value in New York City’s data. We believe that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker.
- Open Data for All means ALL New Yorkers: Analytics underlies Open Data: we use analytics to assess who Open Data is reaching – and who it’s not. We recently partnered with students from New York University’s Center for Urban Science and Progress to publish “Reducing Data Poverty in NYC: Achieving Open Data for All.” The researchers used Structural Equation Modeling (SEM) to measure which factors contribute to lack of Open Data use in certain communities. This study suggests that “data representation” – when data captures some aspect of the data user or their community – is associated with higher use of the Open Data Portal. This insight has pushed us to redouble our efforts on making more communities aware of the benefits of Open Data.
- Open Data for All means ALL City datasets: When city leaders passed the Open Data Law in 2012, New York City committed to opening each and every one of its 1600+ public datasets. The Department of Sanitation, Department of Correction, and Department of Housing Preservation and Development were the first agencies we collaborated with as part of a coordinated effort to locate new datasets to publish on the Portal. Just this week, we released a report of our findings.
- New York Police Department (NYPD) Complaint Data: Two new datasets provide information on 10 years’ worth of felony, misdemeanor, and violation crimes reported to the NYPD from 2006 through 2016. Historical data through 2015 can be found here, and 2016 data can be found here.
- New York Fire Department (FDNY) Incident Dispatch Data: FDNY recently released Fire Incident Dispatch Data and EMS Incident Dispatch Data datasets. Each dataset contains a wealth of information that spans from the time that an incident is created to the time it is closed in the dispatch system.
Improvements to the Open Data Portal
We are always looking to improve the Portal to meet users where they’re at. New features have been built with mobile devices and seamless data discovery in mind.
- New catalog: Open Data for All means making datasets more accessible to those not already in the know. A new search experience allows datasets to be quickly filtered by category, data format, and other information, making it easier than ever to find datasets meaningful to the user.
- Primer: Many datasets released on the Open Data Portal are raw data that may never have surfaced outside of an agency. “Primer” is a unique landing page for each dataset that will serve as a guide for users before they dive into the data. Along with new data dictionaries, this new introduction page will help narrow the gap between agency experts and Open Data users.
Technical Standards Manual Updated
We updated the Technical Standards Manual, the document of record on all things Open Data. Data standards are “rules” on how agencies should manage or present their data. The new standards were designed to make Open Data more usable to the maximum number of users. Here are the latest we finalized:
- Address Data (188.8.131.52): These new standards ensure that every agency is writing addresses in the same way so that it’s easier for users to make maps with the City’s data. In September, we invited the public to contribute ideas for new address standards. Based on diverse feedback, the new address standards reflect the geospatial fields most frequently captured by City agencies, information that is in highest demand from public users, and attributes that will have the biggest impact on citywide operations once they are standardized.
- Retention and Archiving (4.4.3): We’re committed to maintaining both up-to-date records and historical data. The new retention and archiving standard ensures that no useful and accurate records are being removed from the Open Data portal in order to preserve the New York City historical record as represented by the City’s official data.
Expect more updates to come as we roll into 2017. We are committed to engaging the public throughout the year to help us keep improving, and we plan to hold more public events in the future. We’re also working on a new Open Data website, so keep an eye on our Twitter, Facebook, and Tumblr for the most up-to-date information!
Curious about the allocation of tree species around New York
City, Brooklyn-based web developer and designer Jill Hubley took a deep dive
into the results of the 2005 Street Tree Census to see what arboreal
patterns took shape in neighborhoods across the five boroughs. The
map Jill created – both kaleidoscopic and visually intriguing – depicts the
distribution and biodiversity of the city’s street trees. Users can filter by
species and zoom in for a detailed view of streets.
The map also reflects the results of careful planting
considerations by the New York City Parks Department. Site selection takes into
account several factors to determine what trees are a good fit for a particular
area including site condition, overhead clearance, tree bed width, and
biological diversity. Diseases and pests that target particular types of trees
make varied plantings around the City a necessity.
Click through and get a detailed perspective on the botany of your
NYC Wi-Fi Hotspot Locations | NYC Open Data
Visit New York City’s Open Data portal to see a map of public wifi hotspots across the five boroughs, including City parks, subway stations, payphones, and facilities of the three public library systems. The data also informs the NYCityMap, an interactive visualization of City services, cultural institutions, programs, projects, and other features.
New York City is home to more than 65 different types of street trees ranging from the Alder to Zelkova. With over 600,000 street trees across the five boroughs, one might wonder where they are and which types are most common. Using NYC Open Data of street trees, a few local designers decided to find out… According to their analysis, Queens is home to a third of the city’s street trees and the Maple and Plane Tree (Sycamore) are the most common.
To learn more, check out their Interactive Visualization of NYC Street Trees.
Interested in NYC tree planting? Visit MillionTreesNYC
New York City becomes even more beautiful when the sun begins to set. Every day, there is a moment when the sun fits perfectly between buildings, so that rays of light reflect among the skyscrapers.
Twice a year, when the sun aligns with the east-west midtown street grid it creates a phenomenon known as “Manhattanhenge.” Yet it’s possible to experience henge events during every sunset all over the city. Last year, using data from NYC Open Street Maps (OSM) and various technologies, folks at CartoDB created a map that locates every “NYChenge” that occurs in New York City every single day.
Exploring urban data through New York City subway maps
Tunnel Vision NYC, a new app created by Bill Lindmeier as a thesis project at ITP / NYU, layers data from the MTA and U.S. Census Bureau on MTA subway maps. Simply point your phone at the map to see data visualizations of turnstile activity, rent prices, income and more.
Get the Tunnel Vision NYC app
Register for MTA datafeeds
Download NYC population by census tract data on NYC OpenData
This data visualization by Andrew Hill displays vehicle collisions aggregated by time of day using recently released NYPD motor vehicle collision data.
Mayor de Blasio has set the goal of eliminating traffic fatalities in New York City. We’re looking to the civic tech community to help us achieve that vision.
Sarah Kaufman of New York University’s Rudin Center for Transportation recently analyzed six months of Citi Bike subscriber data. Her results, shown on this map, illustrate a striking difference in ridership. Kaufman writes: “of the top ten stations for each gender, women preferred the Brooklyn residential neighborhoods of Fort Greene and Clinton Hill, while men were overwhelmingly represented in bustling Manhattan." Currently 32% of Citi Bike riders are women and 68% are men.
Read Sarah Kaufman’s post about Citi Bike and gender
View Citi Bike data here
View the Citi Bike challenge for NYC BigApps and affiliated data on the BigApps page of the NYC OpenData portal
NYC OpenData powers the annual NYC BigApps Competition. From May to September, hundreds of top developers, entrepreneurs, thought leaders, designers, makers, and marketers will convene at events across the City to address civic challenges through technology. The NYC OpenData portal is home to a number of helpful public and private datasets from nonprofits, City agencies, and other organizations, all selected to support the BigApps development process. To access these datasets, visit the NYC BigApps data catalog. To learn more about the BigApps competition, visit the official NYC BigApps website.
Explore NYC’s 1,053,713 buildings by year of construction.
Access the data used to build this visualization via NYC OpenData.
Map built by Brandon Liu. Data via NYC OpenData (PLUTO and building footprints) and OpenStreetMap. Made using TileMill by MapBox. Inspired by Justin Palmer’s Portland map and BKLYNR’s Brooklyn map.
Check out the latest mashup of NYC parking ticket data with parking fines by Alihan Polat/Studio M+.
Get parking violation data via NYC OpenData.
NYC Collaboration with OpenStreetMap
The team at Mapbox created this impressive animation highlighting two NYC OpenData sets being imported into OpenStreetMap: building footprints and address points in New York City.
The OpenStreetMap (OSM) community is adding vital NYC OpenData to the OSM database.
Curious to learn more about the noise complaints mentioned in the Health Department’s post? Visit New York City’s Open Data portal to view the 311 Service Request dataset, which includes 311 noise complaints from 2010 to present. 311 Service Request data from 2009 is also available on the Open Data portal.
Reblogged from nychealth:
Noise in NYC
Big cities like NYC are full of great sights, sounds … and noises.
Ambient noise is the noise from traffic, construction, industrial or recreation activities, animals, or people’s voices, that someone doesn’t want to hear. Too much ambient noise can cause stress, higher blood pressure, and interference with sleep.
To gain a better understanding of ambient noise disturbance among all New Yorkers, a recent Community Health Survey asked adults about how often they were disrupted by noise within the previous three months and why. Here’s what we learned:
- 4 in 10 New Yorkers reported having activities disrupted by noise from outside their homes at least once in the previous 3 months.
- 3 in 4 of New Yorkers experiencing frequent noise disruptions —about 828,000 New Yorkers—reported noise disruption 7 or more times per week.
- More than half of all those reporting any noise disruption said they were disturbed by noise coming from traffic – noise from cars, trucks, or other vehicles, excluding emergency sirens – and about half said neighbors and emergency sirens caused their noise disruption.
NYC also tracks noise complaints through its 311 calling system. Of the 1,783,133 complaints to the 311 call system in 2009:
- 111,730 (6%) of 311 calls were noise-related.
- More than half of 311 noise complaints were related to noise from loud music and parties (34%) or other social environment causes (24%) such as noise from neighbors, loud talking, loud TV, alarms going off, ice cream trucks, or noise from ventilation units.
- 1 out of 5 noise calls to 311 were to complain about traffic or transportation noise.
- 311 complaint data show that residents of Manhattan disproportionally called about noise-related complaints in 2009.
- Central Harlem-Morningside Heights, Chelsea-Village, and Union Square-Lower Manhattan were among the top five communities with the highest 311 noise-related calls rates as well as the highest prevalence of noise disruption, as reported to the Community Health Survey.
Want to learn more? Check out our new report for more NYC noise facts.