Open Data for All 2017 Progress Report and Compliance Plan
Since the passage of Local Law 11
of 2012, the Open Data team has issued an annual report each July to inform New
Yorkers about the progress we’re making – both in complying with the law and in
engaging New Yorkers to use Open Data in new, creative ways.
On July 14, we released our Open
Data for All 2017 Progress Report and Compliance Plan. For the first time,
we were able to publish it as an interactive website. In the report, we
highlight our accomplishments over the past year, feature vignettes about
everyday users, and update our compliance plan. We want to hear from our users.
To provide comment on the report itself, or to tell us your own Open Data
story, please comment directly through the interactive website here
and scroll to the bottom of the page. We’re accepting feedback through the
website through September 1. Users may contact us at any time here.
In addition to requesting comment
online, the Open Data team periodically hosts events to connect with our users.
On Wednesday, July 19, the Open Data team hosted its Summer Open Data Updates
event at Civic Hall, where we presented a deeper look into the report,
showcased newly released datasets, and highlighted the types of dataset
requests the team receives. Shane Leese from RentHop and Mary Tobin from the
Brownsville Partnership, both of whom were featured in the report as users of
NYC Open Data, spoke about their Open Data experiences. David Rimshnick from
FactorPrism also presented, demonstrating how to make the best of 311 request
data using his analytics tool. Thanks to the hundreds who attended or
livestreamed the event, and special thanks to both the speakers and Civic Hall
for their help in making the event a great success! If you missed the event,
you can view the recorded livestream in full at https://www.facebook.com/CivicHallNYC/videos/1822368181123039/.
Isn’t celebrating fun?
It’s been five years since New York City signed the strongest open data
law in the country. We’ve been busy ever since, and celebrated a LOT in honor
of our 5-year anniversary…
- We launched a pretty new beta website: www.nyc.gov/opendata (send us any
- There’s a new way for you to contact us if you
have questions, comments, concerns: www.nyc.gov/opendata/engage/ ; some
folks have already noticed the difference…!
- Our partners at the Department of City Planning
launched their new Facilitates Explorer tool – check it out: https://capitalplanning.nyc.gov
- 900+ people participated in NYC’s First NYC Open Data Week in 12 events across three boroughs
- We taught 15 New Yorkers were taught how to use
NYC Open Data via our first NYC Open Data for All:
TreesCount! Workshop; the first 1-day data-literacy program developed by
the City of New York with NYC Open Data as a core teaching tool
- Our NYC Chief Analytics Officer, Dr. Amen Ra
at the Socrata Connect Conference that he will no longer give speeches on
NYC Open Data. Instead, our team will find New Yorkers who have been impacted
by open data to speak to its promise.
- A few folks said nice things about our work
(thanks!) and the future of NYC Open Data: State Scoop, AM
New York, GovTech,
Ash Center, Progrss,
Big Apps NYC 2017, Chris
Chavez via Medium, Carto
All of these wonderful things would
not have been possible without the great support, effort and cheerleading from:
BetaNYC, BureauBlank, Reinvent Albany, DoITT NYC Gov Lab & Studio, NYC Parks
Analytics, NYC Parks Computer Resource Centers, Capital Planning Team at the
Department of City Planning, Office of Digital Strategy, The Department of
Small Business Services, Carto, Microsoft, Socrata, Screendoor, Vizalytics,
General Assembly, Grand Central Tech, Civic Hall, Civic Hall Labs, The College
of Staten Island’s Tech Incubator, The Manhattan Borough President’s Office,
the National Geo-Spatial Intelligence Agency (NGA), 18F, DataKind, NYC Big
Apps, The Made in New York Media Center, Fabernovel Data & Media, Reaktor,
Prime Produce, Startup M/IG, GovLab, Luminary Labs and many others!
A final thank you to all 8.5
million New Yorkers - you inspire us every day to do this work and it
wouldn’t be where it is without you all and our ongoing dialogue.
Stay tuned for more great things to come from NYC Open Data! Also – if you have a
suggestion for something to be featured on our blog, get in touch with us here.
You may have noticed that NYC Open
Data has a new look! We’re excited about our new website and hope
you’ll explore and let us
know what you think.
The site includes a new “Contact
Us” page, which will serve as a one-stop shop for any feedback you have for the
NYC Open Data Team. The page includes a Screendoor-enabled tool that will allow
us to respond to your questions, data requests, and any other inquiries in a
more timely manner.
In order to streamline the
collection of inquiries, we have worked closely with Socrata to make the
- Dataset nominations: We
disabled the old nominations page and are routing all new dataset requests to
Us page. You can still track the status of previously made requests here. We have removed the response to these
dataset nominations from this dataset as the responses were always
customer-service related in nature and did not provide insight into the data
- Comments: We disabled the ability to post
new comments on specific datasets in the Socrata platform. We realized that
users were posting questions here, which made it difficult for us to quickly
respond to. As a result, we are funneling inquiries to the Contact
Us page while we research commenting alternatives to implement in
the future. No current comments will be removed.
If you have any
questions about these changes, please get in touch through the new Contact Us page!
Joey Cherdarchuk of Dark Horse Analytics first posted this creative use of Open
Data in 2014. Several datasets, including this building
footprints shapefile, the Department of City Planning’s Bytes of the
Big Apple, and census data (here and here), were used to create
this mesmerizing graphic representing the ebb and flow of working New Yorkers
Seen any similar analyses using this type of data? Let us
know on Twitter or Facebook!
Open Data for All at the International Open Data Conference
Back in October, the Government of Spain, World Bank, and Open Data for Development Network hosted the 4th International Open Data Conference (IODC) in Madrid. Last week, they published the International Open Data Roadmap, which represents the collective vision of the best and brightest Open Data leaders across the globe.
Dr. Amen Ra Mashariki, New York City Chief Analytics Officer and head of the NYC Open Data initiative, presented at the IODC on how the City works toward Open Data for All – the idea that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker.
This strategy, Dr. Mashariki said, centers around six core values:
- Start with Users: The first step in opening data is focusing on demand. We support creative analytic thinking to enable users to answer the questions most important to them and also make efforts to spur demand in communities less familiar to Open Data.
- Treat the publication of the dataset as its debut: Though a lot of work goes into making a dataset ready to go live, our work isn’t done once that dataset is published. Opening data requires that it is updated, errors are identified, and users are able to engage and provide feedback.
- Encourage purposeful and easy engagement: Open Data is a platform that should be used as a tool for feedback. Substantive engagement is more than an obligation; it must be default for Open Data.
- Empower agencies: By empowering Open Data Coordinators and champions in every agency and office, we can narrow the gap between agency experts and Open Data users.
- Integrate Open Data into citywide processes: As Open Data becomes routine across NYC government, we will look for opportunities to make small changes in existing processes.
- Learn, test, standardize – and learn again: We must test our assumptions and try new ideas, collect information on our efforts, and analyze our performance for points of improvement.
Watch Dr. Mashariki’s full presentation (37 minutes) at IODC2016.
One Dataset, Three Ways
UPDATE: A third use has been identified.
Last April, DoITT released a three-dimensional (3-D)
Building Massing Model of New York City, available for download on DoITT’s
website and the Open
Data Portal. The 3-D model was captured from the 2014 aerial survey. Aerial
surveys are conducted every two years to produce digital orthophotography, or
aerial photography that can be accurately scaled and used for maps. The data
contains every NYC building present when that survey was taken.
Providing this data to the public allows users to creatively
visualize New York City’s iconic skyline, conduct analyses and build
applications. We are familiar with three separate uses of this data:
Building Shadow Mapping: To commemorate the Winter Solstice in December,
New York Times contributors analyzed the building shadows of the entire City.
Interactive maps at different times of the year – Winter, Spring/Fall, and
Summer – allow readers to explore the amount of time each NYC building spends
Map: Cesium is an open-source
library for 3D globes and maps. One user created an interactive map of New York
City, visualizing New York City’s over 1 million buildings as 3D models. Map
users may explore this map by address or landmark, and buildings may be
presented as colors visually representing their heights.
- Esri Map: CoolMaps is a collection of creative maps that use the Esri Mapping Platform. This Vision Zero map uses the 3D model of New York City to represent live traffic, collisions, and traffic injuries and deaths, based on NYC Department of Transportation’s datasets on the Open Data Portal.
Seen any other examples? Let us know! DM us on Twitter or Facebook!
For-Hire Vehicle Data
Today, the Taxi and Limousine Commission (TLC) held a public hearing on a new rule to address the risks of fatigued driving and add trip reporting requirements for For-Hire Vehicle (FHV) bases.
This rule would require FHV bases, including app-based services such as Uber and Lyft, to report pick-up and drop-off times and locations for all trips. This would allow TLC to verify that drivers are limiting the number of hours they spend on the road to avoid fatigued driving. If this rule passes, the data will be published to NYC Open Data, similarly to the data for green and yellow taxi cab trips that is already on Open Data.
Amen Ra Mashariki, Chief Analytics Officer and Director of the Mayor’s Office of Data Analytics, testified before the TLC Board of Commissioners today on how Open Data in general – and TLC data in particular – benefits all New Yorkers. Read Dr. Mashariki’s testimony and follow updates on the proposed new rules at nyc.gov/taxi.
End of Year Updates from the Open Data Team
2016 has been a productive year in the world of Open Data! Our Open Data team, comprised of the Department of Information Technology and Telecommunications and the Mayor’s Office of Data Analytics, want to share our incredible progress over the past year.
Progress on Open Data for All
In July 2015, we committed to Open Data For All, an inclusive vision in which all New Yorkers, not just the tech-savvy, can find value in New York City’s data. We believe that every New Yorker can benefit from Open Data, and Open Data can benefit from every New Yorker.
- Open Data for All means ALL New Yorkers: Analytics underlies Open Data: we use analytics to assess who Open Data is reaching – and who it’s not. We recently partnered with students from New York University’s Center for Urban Science and Progress to publish “Reducing Data Poverty in NYC: Achieving Open Data for All.” The researchers used Structural Equation Modeling (SEM) to measure which factors contribute to lack of Open Data use in certain communities. This study suggests that “data representation” – when data captures some aspect of the data user or their community – is associated with higher use of the Open Data Portal. This insight has pushed us to redouble our efforts on making more communities aware of the benefits of Open Data.
- Open Data for All means ALL City datasets: When city leaders passed the Open Data Law in 2012, New York City committed to opening each and every one of its 1600+ public datasets. The Department of Sanitation, Department of Correction, and Department of Housing Preservation and Development were the first agencies we collaborated with as part of a coordinated effort to locate new datasets to publish on the Portal. Just this week, we released a report of our findings.
- New York Police Department (NYPD) Complaint Data: Two new datasets provide information on 10 years’ worth of felony, misdemeanor, and violation crimes reported to the NYPD from 2006 through 2016. Historical data through 2015 can be found here, and 2016 data can be found here.
- New York Fire Department (FDNY) Incident Dispatch Data: FDNY recently released Fire Incident Dispatch Data and EMS Incident Dispatch Data datasets. Each dataset contains a wealth of information that spans from the time that an incident is created to the time it is closed in the dispatch system.
Improvements to the Open Data Portal
We are always looking to improve the Portal to meet users where they’re at. New features have been built with mobile devices and seamless data discovery in mind.
- New catalog: Open Data for All means making datasets more accessible to those not already in the know. A new search experience allows datasets to be quickly filtered by category, data format, and other information, making it easier than ever to find datasets meaningful to the user.
- Primer: Many datasets released on the Open Data Portal are raw data that may never have surfaced outside of an agency. “Primer” is a unique landing page for each dataset that will serve as a guide for users before they dive into the data. Along with new data dictionaries, this new introduction page will help narrow the gap between agency experts and Open Data users.
Technical Standards Manual Updated
We updated the Technical Standards Manual, the document of record on all things Open Data. Data standards are “rules” on how agencies should manage or present their data. The new standards were designed to make Open Data more usable to the maximum number of users. Here are the latest we finalized:
- Address Data (18.104.22.168): These new standards ensure that every agency is writing addresses in the same way so that it’s easier for users to make maps with the City’s data. In September, we invited the public to contribute ideas for new address standards. Based on diverse feedback, the new address standards reflect the geospatial fields most frequently captured by City agencies, information that is in highest demand from public users, and attributes that will have the biggest impact on citywide operations once they are standardized.
- Retention and Archiving (4.4.3): We’re committed to maintaining both up-to-date records and historical data. The new retention and archiving standard ensures that no useful and accurate records are being removed from the Open Data portal in order to preserve the New York City historical record as represented by the City’s official data.
Expect more updates to come as we roll into 2017. We are committed to engaging the public throughout the year to help us keep improving, and we plan to hold more public events in the future. We’re also working on a new Open Data website, so keep an eye on our Twitter, Facebook, and Tumblr for the most up-to-date information!
Curious about the allocation of tree species around New York
City, Brooklyn-based web developer and designer Jill Hubley took a deep dive
into the results of the 2005 Street Tree Census to see what arboreal
patterns took shape in neighborhoods across the five boroughs. The
map Jill created – both kaleidoscopic and visually intriguing – depicts the
distribution and biodiversity of the city’s street trees. Users can filter by
species and zoom in for a detailed view of streets.
The map also reflects the results of careful planting
considerations by the New York City Parks Department. Site selection takes into
account several factors to determine what trees are a good fit for a particular
area including site condition, overhead clearance, tree bed width, and
biological diversity. Diseases and pests that target particular types of trees
make varied plantings around the City a necessity.
Click through and get a detailed perspective on the botany of your
NYC Wi-Fi Hotspot Locations | NYC Open Data
Visit New York City’s Open Data portal to see a map of public wifi hotspots across the five boroughs, including City parks, subway stations, payphones, and facilities of the three public library systems. The data also informs the NYCityMap, an interactive visualization of City services, cultural institutions, programs, projects, and other features.
New York City is home to more than 65 different types of street trees ranging from the Alder to Zelkova. With over 600,000 street trees across the five boroughs, one might wonder where they are and which types are most common. Using NYC Open Data of street trees, a few local designers decided to find out… According to their analysis, Queens is home to a third of the city’s street trees and the Maple and Plane Tree (Sycamore) are the most common.
To learn more, check out their Interactive Visualization of NYC Street Trees.
Interested in NYC tree planting? Visit MillionTreesNYC
New York City becomes even more beautiful when the sun begins to set. Every day, there is a moment when the sun fits perfectly between buildings, so that rays of light reflect among the skyscrapers.
Twice a year, when the sun aligns with the east-west midtown street grid it creates a phenomenon known as “Manhattanhenge.” Yet it’s possible to experience henge events during every sunset all over the city. Last year, using data from NYC Open Street Maps (OSM) and various technologies, folks at CartoDB created a map that locates every “NYChenge” that occurs in New York City every single day.
Exploring urban data through New York City subway maps
Tunnel Vision NYC, a new app created by Bill Lindmeier as a thesis project at ITP / NYU, layers data from the MTA and U.S. Census Bureau on MTA subway maps. Simply point your phone at the map to see data visualizations of turnstile activity, rent prices, income and more.
Get the Tunnel Vision NYC app
Register for MTA datafeeds
Download NYC population by census tract data on NYC OpenData
This data visualization by Andrew Hill displays vehicle collisions aggregated by time of day using recently released NYPD motor vehicle collision data.
Mayor de Blasio has set the goal of eliminating traffic fatalities in New York City. We’re looking to the civic tech community to help us achieve that vision.
Sarah Kaufman of New York University’s Rudin Center for Transportation recently analyzed six months of Citi Bike subscriber data. Her results, shown on this map, illustrate a striking difference in ridership. Kaufman writes: “of the top ten stations for each gender, women preferred the Brooklyn residential neighborhoods of Fort Greene and Clinton Hill, while men were overwhelmingly represented in bustling Manhattan." Currently 32% of Citi Bike riders are women and 68% are men.
Read Sarah Kaufman’s post about Citi Bike and gender
View Citi Bike data here
View the Citi Bike challenge for NYC BigApps and affiliated data on the BigApps page of the NYC OpenData portal
NYC OpenData powers the annual NYC BigApps Competition. From May to September, hundreds of top developers, entrepreneurs, thought leaders, designers, makers, and marketers will convene at events across the City to address civic challenges through technology. The NYC OpenData portal is home to a number of helpful public and private datasets from nonprofits, City agencies, and other organizations, all selected to support the BigApps development process. To access these datasets, visit the NYC BigApps data catalog. To learn more about the BigApps competition, visit the official NYC BigApps website.
Explore NYC’s 1,053,713 buildings by year of construction.
Access the data used to build this visualization via NYC OpenData.
Map built by Brandon Liu. Data via NYC OpenData (PLUTO and building footprints) and OpenStreetMap. Made using TileMill by MapBox. Inspired by Justin Palmer’s Portland map and BKLYNR’s Brooklyn map.
Check out the latest mashup of NYC parking ticket data with parking fines by Alihan Polat/Studio M+.
Get parking violation data via NYC OpenData.
NYC Collaboration with OpenStreetMap
The team at Mapbox created this impressive animation highlighting two NYC OpenData sets being imported into OpenStreetMap: building footprints and address points in New York City.
The OpenStreetMap (OSM) community is adding vital NYC OpenData to the OSM database.
Curious to learn more about the noise complaints mentioned in the Health Department’s post? Visit New York City’s Open Data portal to view the 311 Service Request dataset, which includes 311 noise complaints from 2010 to present. 311 Service Request data from 2009 is also available on the Open Data portal.
Reblogged from nychealth:
Noise in NYC
Big cities like NYC are full of great sights, sounds … and noises.
Ambient noise is the noise from traffic, construction, industrial or recreation activities, animals, or people’s voices, that someone doesn’t want to hear. Too much ambient noise can cause stress, higher blood pressure, and interference with sleep.
To gain a better understanding of ambient noise disturbance among all New Yorkers, a recent Community Health Survey asked adults about how often they were disrupted by noise within the previous three months and why. Here’s what we learned:
- 4 in 10 New Yorkers reported having activities disrupted by noise from outside their homes at least once in the previous 3 months.
- 3 in 4 of New Yorkers experiencing frequent noise disruptions —about 828,000 New Yorkers—reported noise disruption 7 or more times per week.
- More than half of all those reporting any noise disruption said they were disturbed by noise coming from traffic – noise from cars, trucks, or other vehicles, excluding emergency sirens – and about half said neighbors and emergency sirens caused their noise disruption.
NYC also tracks noise complaints through its 311 calling system. Of the 1,783,133 complaints to the 311 call system in 2009:
- 111,730 (6%) of 311 calls were noise-related.
- More than half of 311 noise complaints were related to noise from loud music and parties (34%) or other social environment causes (24%) such as noise from neighbors, loud talking, loud TV, alarms going off, ice cream trucks, or noise from ventilation units.
- 1 out of 5 noise calls to 311 were to complain about traffic or transportation noise.
- 311 complaint data show that residents of Manhattan disproportionally called about noise-related complaints in 2009.
- Central Harlem-Morningside Heights, Chelsea-Village, and Union Square-Lower Manhattan were among the top five communities with the highest 311 noise-related calls rates as well as the highest prevalence of noise disruption, as reported to the Community Health Survey.
Want to learn more? Check out our new report for more NYC noise facts.
While New York City is among the nation’s most dense cities, the Department of City Planning’s Projected Population 2000 - 2030 dataset, available on the NYC Open Data portal, provides borough population projections that demonstrate how density differs across the five boroughs.
This graphic shows the approximate population per square mile in each borough in 2000, as well as 2030 projected population per square mile.
View DCP’s Population Projections.
Learn more about how the Department of City Planning estimates population.
Visit the NYC Open Data portal.
Update April 17, 2014: DCP has released revised population predictions through 2040 on the NYC Open Data portal.
Ben Wellington, who teaches a statistics course in the City & Regional Planning program at Pratt in Brooklyn recently released two visualizations using data from the NYC OpenData portal. In the image above, he used the Department of Sanitation’s Monthly Tonnages dataset to explore recycling rates in the five boroughs.
In the image below, he used the street name dictionary and GIS line street base map datasets to visualize street suffixes across the city, showing patterns in planning and street naming. He writes, “Manhattan is made up of mostly streets…the Bronx has the most avenues proportionally, and Queens has the most roads. Staten Island has the largest percentages of lanes and courts, which might go along with its suburban layout.”
In addition to using NYC OpenData in his own work (see more visualizations on his blog, I Quant NY), Ben employs public data from the NYC OpenData portal in his statistics classes at Pratt. Bringing open data into the classroom allows his students to explore their city and analyze information that’s relevant to their interests as urban planners, whether transportation, health inspection, education or other data.
Check out Ben’s work on Citywide recycling patterns here.
Check out Ben’s work on Street Suffixes here.
View more open data on the NYC OpenData Portal.
The Taxi and Limousine Commission recently launched a Car Service “Basefinder”! Use the interactive map to locate the nearest car service/livery base. The map also indicates which bases have wheelchair accessible vehicles. Access the map of taxi bases via NYC OpenData: http://bit.ly/1mfYJS5
Independently designed and launched by Microsoft Research’s Future Social Experiences Labs’ Kati London, HereHere playfully integrates NYC Open Data offering a new perspective on NYC 311 Service Request data. HereHere aggregates non-emergency 311 requests received by phone, web, text and mobile app, into neighborhoods and departs from purely data-based visualizations by assigning a human-reaction to the top service requests. Users can click on a neighborhood to see the current status based on recent complaints as well as the top complaint from the prior year.
For example, the High Bridge neighborhood summed up recent 311 requests saying, “I feel frustrated. Ack! It’s been a while since this has come up, a few reports of wildlife sightings and a few dog off leash reports. More than I’ve seen of this in a while, and a few concerns about a restaurant.”
Each neighborhood also has a Twitter feed and a daily email summary for those who would like the information delivered. Next, Kati and her team plan to add a location-finder, so users can identify which of the 42 neighborhoods they belong to by zip code.
Follow a neighborhood by email or Twitter via the links below:
New York City is going global (and local!); in 2011, 15 New Yorkers were named Brooklyn, 15 were named Dakota, 10 were named Kenya, and 129 were named London. Though none of these names made the top twenty for either boys or girls, this graphic shows the baby names that dominated both the City and New York State.
Data for the most popular New York City names comes from the Department of Health and Mental Hygiene’s “Most Popular Baby Names by Sex and Mother’s Ethnicity” dataset, available on the NYC Open Data Portal, while data on New York State names is available on the New York State Open NY website.