What the NYC Open Data Law Requires (2012’s Local Law plus 7 amendments from 2015, 2016 )

Happy fourth birthday to Local Law 11 of 2012, New York City’s Open Data Law. Last November and January, the first amendments to Local Law 11 were enacted (and take effect at various points in the next ten months). Below are the major mandates in the Open Data Law, as amended by 2015’s Local Law 106107, 108109110, as well as 2016’s Local Law 7 and Local Law 8. Mandates provided by amendments are italicized.

§23-502: Public data set availability

a. All data sets which were on NYC agency websites before March 2012, or subsequently published online, were required to be put on the Open Data Portal by March 2013. Agencies were supposed to report to city council if they could not put these data sets in the Open Data Portal, by the deadline.

b. Data sets on the portal must be machine-readable and allow the public to subscribe to updates.

c. Data sets must be updated as often as necessary to preserve their usefulness, as long as the agency still uses those data sets. DoITT must archive the data sets on the portal (LL 106-2015) by preserving older versions of the data sets which are updated. If a data set is available on the Open Data Portal as well as a city agency’s web site (LL 110 2015), when the data is updated on the web site, it must be updated on the Open Data Portal within ten days.

d. Data sets must be available without any registration requirement, license requirement, or restrictions on their use. (Agencies can require attribution.)

e. Data sets must be available to external searches.

f. Agencies must review their responses to Freedom of Information Law requests (LL 7 – 2016) for data sets which have not yet been included on the Open Data Portal, and describe in their compliance plan: (1) how many data sets not on the Open Data Portal were provided to the public via FOIL, (2) how many of those data sets were not on the portal, and (3) how many data sets were then put on the portal do to FOIL. (The idea is to get agencies to put a data set in the Open Data Portal whenever it is provided under FOIL.)

§23-503: Web portal administration

c. DoITT must create a forum for public feedback and encourage discussion on open data policies and data set availability.

d. Requests for inclusion of a particular data set must be considered by agencies which are prioritizing the publication of data. DoITT must provide a response to each request within two weeks (LL 109 – 2015), and the agency-owner of the requested data set must respond within two months.

§23-504: Open data legal policy

c. The Open Data Law shall not create a private right of action for enforcement of its provisions.

§23-505: Internet data set policy and technical standards

a. DoITT must publish a technical standards manual. Each data set must include a data dictionary (LL 107-2015 ), no later than December 31, 2017 for existing data sets, or within 30 days for new data sets. The technical standards manual must require every agency to geocode every address (LL 108 -2015) in their data sets.

b. DoITT must consult consensus standards bodies to develop the open standards above.

§23-506: Agency compliance plan

a. Each agency has to publish a description of all public data sets under its control, and create a timeline for the publication of each by December 31, 2018.

c. Agencies must update compliance plans every year on July 15 until every data set is published online.

Open data law agency compliance examination (LL 8 – 2016)

Open Data Law agency compliance examination: an agency designated by the mayor will present a plan for investigating agency compliance with the Open Data Law to the Department of Investigation.

  • On December 1 2016, the agency selected by the mayor will report on the compliance of the Departments of Sanitation, Correction, and Housing Preservation and Development.
  • On December 1 2017, agency will report on the Fire Department, the Department of Buildings, and the Department of Environmental Protection.
  • On December 1 2018, agency will report on the Business Integrity Commission, the Department of Transportation, and the Department of Small Business Services.

On December 1, 2019, the agency selected by the mayor will report on all agencies’ compliance with the Open Data Law, and provide a complete list of every public data set absent from the portal which was discovered during the examination.


Reformers Call On Governor, Legislative Leaders to Hold Public Leaders’ Meetings to Openly Address State Ethics Reform

Negotiations around Laws and Rules Governing Ethical Conduct of Officials Must Be Conducted in Public View

(Albany, NY) In a letter delivered today, a coalition of government reform organizations called on Governor Andrew Cuomo, Senate Majority Leader John Flanagan and Assembly Speaker Carl Heastie to hold a series of open public leaders’ meetings to discuss and negotiate – in public – ethics reforms necessary to restore New Yorkers’ battered confidence in Albany.

With a recent poll showing that nearly 90% of New Yorkers believe that unethical behavior is a serious problem in state government a month before former legislative leaders Sheldon Silver and Dean Skelos are sentenced for public corruption, the governor and legislative leaders have an obligation to New Yorkers to reach a significant agreement on ethics reform.

We believe that in order for the agreement to be sufficiently consequential as to change the culture of how business is done in Albany and lessen the occurrence of corruption, it must be discussed and negotiated in the full view of the public.

We call for public leaders’ meetings because we know that the typical Albany pattern will be to discuss ideas over the next few weeks during private negotiations and then secretly hammer out an ethics deal. That secret deal will then be heralded as “historic” with “unprecedented new reforms” that will be a silver bullet to resolve the state’s ethics shortcomings. The deal will be folded into the budget or end-of-session crush and the details and practical effects of the new “reforms” will not come to light until they are in operation.

Over time, it will then become clear that loopholes in those “reforms” will undermine many of their asserted benefits. Thus, what was heralded as “historic reforms” will have little positive impact and Albany will return to its status quo – until the next scandal. As a result, New Yorkers’ confidence in the integrity of our state government will further erode. This is why it is critical that the leaders’ meetings must not be private, but public.
Our organizations have recommended clear, practical reforms that directly address New York’s ethics crisis. There also is no shortage of good ethics reform proposals from the governor and some of the legislative conferences. But there has been little public dialogue about how and whether those proposals respond to what some describe as a political crime wave stemming from the outsized role of money in state politics.

Dick Dadey, Executive Director of Citizens Union, said: “Public leaders’ meetings on ethics reform are so necessary to ensure that big effective solutions are considered, negotiated, and agreed to, as New Yorkers wait to see how our state’s political leaders propose preventing the growing problem of corruption.”

“Two months into the start of session and there’s still a cloud of corruption hanging over the Legislature,“ said Susan Lerner, Executive Director of Common Cause/NY. “It’s up to the Governor and Legislative leaders to clear the air by making a plan in the light of day. Skelos and Silver will be sentenced next month; let’s see if Albany can beat the clock. New Yorkers will not accept any more excuses.”

“New Yorkers have seen what happens when ethics bills are hammered out in the dark,” said Blair Horner, NYPIRG Executive Director.  “Laws with too many loopholes and which do not solve Albany’s mounting ethics problem.  The public doesn’t want a rerun, we urge the governor and state lawmakers to take a new approach – one that discusses the people’s business in the open, where voters can judge the work.”

“Patchwork fixes to Albany’s damaged ethics system won’t get the job done,” said DeNora M. Getachew, Campaign Manager & Legislative Counsel at the Brennan Center for Justice.  “The way to restore the public’s trust in state government is to debate and agree upon concrete solutions in the light of day.  At the top of the list should be closing ethics loopholes, lowering contribution limits and allowing candidates to run competitive campaigns by matching small donations from everyday citizens.”

Barbara Bartoletti, Legislative Director of the League of Women Voters NYS, said: “Ethics reform is long overdue and overwhelmingly supported by the public. It is time for our leaders to discuss reform proposals in an open and transparent leaders’ meeting. The League has advocated for ethics reform for over 30 years and it is time for the leaders to openly negotiate effective reforms.”

“Albany corruption is an old disease rooted in secrecy. Sunlight is the best disinfectant. If the governor and legislative leaders are at all serious about real reforms, they need to stand up and be accountable in public. No more hiding in the shadows,” said John Kaehny, Executive Director of Reinvent Albany.

New Yorkers deserve better. It’s time for meaningful actions on ethics where the changes are hammered out in public, not behind closed doors.

Download a copy of this letter as a PDF

Open Data’s Green, Yellow and Red Buckets

When think of government data as falling into green, yellow, and red “buckets.”  We start by encouraging government to publish all of their green bucket data. Green bucket data is obviously public and can be published with few, if any, redactions. Data in the yellow bucket might be publishable, but raise potential questions.  The red bucket is data that clearly endangers privacy, public safety and security, trade and contract secrets and law enforcement investigations. Most fights over data release are over yellow bucket data, and it often hinges on the definition of privacy or safety, and a rough feel for what the public is OK with.  There are yellow bucket fights over just about everything you could imagine, including government agencies refusing to publish digital maps of a waterfront which show sewer discharge outlets or subway exits.  Probably the most complex area of the yellow bucket are the many inconsistencies in the release of personally identifiable information. Under NY Law, court records are public, as are property ownership records and professional licenses for doctors, lawyers and others. Yet, state and city legislators have proposed bills which ban release of personally identifiable information for any open data. Below is a rough spectrum originally created by open data and search guru, Jim Hendler of RPI and semantic search fame. It is a few years old and needs to be updated, but is a good place to start the conversation.

Screen Shot 2016-03-07 at 12.44.19 PM


NYPD Launches Compstat 2.0 Portal With Rough Edges, No Bulk Download

Screen Shot 2016-02-25 at 2.35.42 PMThe NYPD’s CompStat 2.0 portal launched last week, with Police Commissioner Bill Bratton noting that “CompStat 2.0 has the ability to take … all the information, with few exceptions, that’s shared with our police officers, and now share it with both you, the media, and to the public, more importantly.”

However, that’s not the case. The new portal is basically a new layer of paint over the same limited subset of CompStat data that the NYPD has shared with the public for years. The most notable improvement in CompStat 2.0 is an interactive map of each incident in the CompStat reports, mapped to the nearest intersection. This is a modest step forward for open police data, and rescues a web site that is otherwise a web-based spreadsheet viewer.

The CompStat incident map is fatally flawed, because the actual location data of all incident is locked away from the public. The NYPD has shown us that there is a latitude and longitude associated with each incident, but won’t make it available to download at all. The only way to see the location of each CompStat incident is to click on each dot on the map in turn. Unfortunately, each incident type is the same color, and the map frequently zooms out during routine browsing, making it tedious to examine different incidents in the same precinct. Compare this data release to the Vision Zero map, which provides downloads of more than twenty data sets.

Screen Shot 2016-02-25 at 2.22.44 PMIn addition to the map, CompStat 2.0 has the same problems that CompStat 1.0 had. There is no historical CompStat data made available, only the availability to see data for the last 7 days, the last 28 days, or the year to date. Every week, the NYPD publishes a new snapshot of CompStat data, and discards the previous week’s data. It is unclear why CompStat cannot provide last week’s snapshot as well as this week’s snapshot, to perform a week-by-week analysis of CompStat data. Again, CompStat 2.0 offers a glimpse of this data: the portal can chart the number of incidents on a weekly basis, so this information is clearly available somewhere. The fact that there’s no way to download it in bulk is baffling.

CompStat 2.0 also does not allow users to compare to one precinct’s incidents to another’s, or to compare one month of incidents to another, and so on. Compared to a web site like NY Open Government—which allows users to search, sort, and download data in bulk—CompStat 2.0 has a long way to go.

Reinvent Albany would argue that providing CompStat data (a step NYPD took in June 2015) as machine-readable files was a far more meaningful step in opening police data than launching this portal without a download button.