NY State “Open NY” Open Data Initiative Publishes July Update, Director Retires

Screen Shot 2016-01-15 at 2.03.36 PMIn July, Barb Cohn, the State’s Chief Data Officer and head of the Open NY open data program retired from her job at the New York State Office of Information Technology Services. Cohn has not been replaced but she left behind an admirable track record of meeting public reporting deadlines, publishing hundreds of new agency data sets and emphasizing high quality data. (According to ITS, the agency is actively recruiting for the position of NYS Chief Data Officer.)

Before she left, Cohn published Open NY’s July quarterly update, which highlights new public health and transportation data sets.

Highlights from the July Open NY Quarterly report:

  • Data is available across ten categories, and includes 1,454 data sets. (data.ny.gov launched on March 11, 2013 with 244 data sets.)
  • Open NY has implemented a Local Data Index, which allows users to search every data set at once for information related to a city or county, depending on the data set.
  • Data.ny.gov has been accessed in over 200 countries and territories, all 50 States and Washington D.C., and over 12,000 cities worldwide – including over 8,000 U.S. cities.

The quarterly update also spotlights the Open NY Dataset Submission Guideline, which tells agencies how to format data for public use. We previously complimented the Guidelines here. As we said when it was released in May, the Guideline may well be the best of its kind in the United States.

Civic Groups Call On Public Service Commission to Extend Public Comment Period for $7B in Nuclear Plant Subsidies

Four leading good government groups joined the City of New York and numerous environmental and consumer watchdog groups in asking the Public Service Commission to extend the public comment period before voting on awarding up to $7B in public subsidies to keep four Upstate nuclear plants operating. The subsidy would be in the form of a monthly $2 to $3 a month surcharge on household electric bills. Almost all of the subsidy would go to one company, Exelon Corporation. Our groups have no joint position on nuclear power subsidies, but sent the letter because of the magnitude of the public cost, and the desire to see the public get the maximum possible opportunity to be heard. (UPDATE: on July 22, 2016, PSC rejected our request in an emailed letter and noted they had already extended the comment period from six to ten business days.)

Grading the Progress of NYC’s Open Data Law

Screen Shot 2016-07-18 at 12.17.36 PM Last Friday, July 15th, right on schedule as required by the NYC Open Data Law, New York City published Open Data For All 2016 its annual progress report on Open Data. The short four page report rightly touted some big successes publishing new data sets of great interest to the public and automating data sets so they update without human intervention. But the report did not mention or at all explain how the City is dealing with persistent problems with data quality, or how mayoral agencies will meet the new mandates imposed by seven new amendments to the Open Data Law.

The NYC Transparency Working Group and Reinvent Albany are huge boosters of the City’s Open Data initiative, but as advocates, our role is to keep pushing the City to obey the letter and spirit of the law, and that can entail some criticism. Overall, we are very happy to see that the de Blasio administration has made a real commitment to improving open data, and that the mayor signed the seven open data bills passed by City Council in the Fall of 2015 and early 2016 – he did not have to do that, and we do not take his action for granted. We also appreciate that NYC DoITT and the Mayor’s Office published the open data progress report on time.

NYC Open Data Law Implementation 2016


An A plus grade for NYC DoITT’s ongoing work to program agency data sets to automatically update. DoITT set up an impressive 100 auto-updating data sets in the last twelve months for a total of 200 automatically updating data sets out of roughly 1,500 data sets total. This is excellent work and greatly improves the value and reliability of the published data. Datasets that do not automatically update on the open data portal have to be manually re-uploaded whenever a new version is published; this is expensive, often delayed, and creates opportunities for big errors as data sets are updated in all kinds of different ways, including cutting and pasting from spreadsheets.

An A grade for publishing major new data sets including the City Budget, City Record Online, and Seven Major Felonies among the 150 new data sets published. The City also published a huge data set of Taxi and Limousine taxi trips. The TLC was getting about 75 FOIL requests a year for this data set, which is so big that requestors were required to give the TLC a blank hard drive that could hold the roughly 50 DVD’s worth of trip data. Open Data is supposed to help government save on the cost and hassle of fulfilling FOI requests, so we are particularly glad to see this happen.

We are also encouraged to see that NYC is explicitly connecting Freedom of Information Law and open data. For the first time the Progress Report includes Local Law 7 of 2016’s mandate that each agency must publish:

  • The number of FOIL responses that included the release of data.
  • The number of FOIL responses which included a data set not yet on the open data portal.
  • The number of FOIL responses that resulted in data sets being voluntarily published (i.e. created and uploaded) on the open data portal.

Thirty agencies out of roughly eighty complied with this new law, which is a good start, but but by next July, all agencies should be complying.

Lesser Lights

Though it was a very big year for automation and data publishing, the Progress Report includes a few gaping holes and does not say anything about the data quality issues that have dominated City Council hearings and NYC’s public discussion about open data for the last year or more. How is the Mayor’s Office of Data Analytics and DoITT going to get agencies to fix serious data errors? Will the City create a workable process for the public to report data quality problems and get them fixed? We do not know. How is the City going to meet the fast approaching new reporting mandates created by the seven amendments to the Open Data Law? Again we do not know.

What we do know is that every type of open data stakeholder—from community based organizations to academia to watchdog groups to journalists and businesses—has vociferously complained about data errors, the lack of explanatory metadata, and poorly structured data for years and nothing has changed. Many open data users have told us that the data they use is so riddled with obvious errors that it causes them to wonder if any of the data is correct.  This serious crisis of confidence in the quality of agency data needs to be addressed or the great dream of open data as a fundamentally new form of open government is going to fail. Given the commitment of the mayor and City Council to making open data work, we are optimistic that the hard work needed to fix this problem will be done. But, it is still disappointing that the 2016 progress report says nothing about the issue that is most on the mind of the open data community.

Open Data and New York State Subsidies to Business: Start-Up NY Edition

Reinvent Albany champions “open data.” We believe  the government data our taxes paid for should be published online and in formats that the average person can use on their home computer, unless that data endangers public safety or personal privacy. We believe it is especially important for government to put data about spending online, particularly business subsidies — which are taxpayer handouts to businesses. Business subsidies are at high risk for pay to play, which is the nicer way of saying “legal bribery.”

New York State has published some subsidy data in the state’s open data portal and on agency websites, including project specific information about Excelsior, Brownfields, and Recharge NY.  Unfortunately, the state’s largest economic development entities either do not reveal important details of subsidies, like who is getting them in the case of Film and TV tax credits, or they put large tables of numbers online in formats that cannot be read by basic spreadsheet software. In the section of Empire State Development’s website called ESD Reports, the authority publishes tables of data on a variety of subsidy programs in a non-open, .pdf format, which cannot be viewed in a spreadsheet. If the ESD wants to be more transparent, it should post data in both .pdf and and a CSV format which can be used in spreadsheets or a database.  Many agencies already do something like this, for example, the NYPD provides their datasets in both PDF and .xlsx spreadsheet formats. (XLSX can be easily translated into CSV.)  We have asked ESD (and all state agencies) to publish tabular data in a CSV format, and we will ask again here. But for the moment, we scraped the PDF report and converted it into a CSV file that watchdog groups or journalists might find more useful.


Federal Freedom of Information Act Celebrates 50th Birthday with Big Makeover

President Obama marked the Fourth of July and 50th Birthday of the Freedom of Information Act by signing the  FOIA Improvement Act of 2016 into law. While the Federal FOIA was already the model public records law in the country, today’s amended version is even better for people making FOIA requests. Highlights of the new law include:

  • Codifying the requirement that records must be available in electronic formats.
  • Codifying the “three strikes and you’re in” rule: records which are FOIAed three times must be published online in agencies’ reading rooms.
  • Codifying the presumption of access to government records.
  • Limits the applicability of the “draft materials” FOIA exemption to records less than 25 years old.
  • Expands the role and responsibility of the Office of Government Information Services and creates a Chief FOIA Officers Council to improve FOIA oversight.
  • Prohibits an agency from charging a fee for providing records if the agency misses a deadline for meeting a FOIA request unless unusual circumstances apply, and more than 5,000 pages are necessary to respond to the request.

Also of note, the FOIA Improvement Act requires the Office of Management and Budget to create a “consolidated” federal FOIA portal. It appears that this makes FOIA Online, the existing portal, permanent. Unfortunately, the new law does not make it easier for the public to win attorney’s fees when they prevail over an agency in court.

Compared to NYS Freedom of Information Law

FOIL in New York has some catching up to do with the federal FOIA. There’s no statewide FOIL portal, and the state law does not create FOIL reading rooms or require the publication of frequently-requested records in them.

However, New York’s FOIL has featured a presumption of public access to government records by default since 1977. New York is nearly four decades ahead of the curve on this, and the drafters deserve credit for their foresight.