Betabeat has had open government on the brain this week, so we called up Girish Chhugani, senior advisor at the Department of Information Technology and Telecommunications (DOITT, pronounced “Do it!”) to talk about the city’s newly launched NYC Open Data initiative.
Mr. Chhugani is both the project lead for NYC Big Apps–Mayor Bloomberg just launched version 3.0 of the contest at NYTM–and the man responsible for transferring the city’s heterogeneous data sets on everything from traffic to crime to electricity consumption to school attendance into something more developer-friendly.
As Mr. Chhugani explained, before this week’s launch, NYC Open Data used to be called NYC Data Mine, a collection of Excel sheets, text files or XML files, depending on the way individual city agencies were used to storing information. Previous Big Apps contestants who wanted to make this information accessible via smartphone, “literally had to download that data set, then import it into their database,” he said. “I’m sure they wrote some script to do those kind of things, but that extra step had to be taken every time. In that sense the data was manually downloaded.”
After evaluating different tools to bring New York into the open government era, the city opted to go with Socrata, the same web services platform used by the federal government for Data.gov as well as the city governments of Seattle and Chicago. Now developers “can directly hook into a data set instead of having to download it,” said Mr. Chhugani. The sets are presented in both machine-readable formats and as APIs. The new site also allows for enhanced browsing and search, so developers can look for a full data set or information within one. Visualization tools can display the information in map, chart, and graph form. DOITT expects to add a field soon that tells users how often a data set will be updated.
However, Mr. Chhugani was quick to note, Open Data is still a work in progress. Data from the Department of Transportation, for example, which has real-time feeds on things like speeding, is presented as a link to the DOT’s data set for now–until there’s an automated way of keeping what’s on one site updated with the other. Mr. Chhugani said DOITT is also working on inputting 311 data. Starting next month, for example, data about all service requests from 2010 to the present will be updated on a nightly basis.
It’s easy to picture an app that, say, shows you a map of zipcodes with the longest wait times and most complaints. And, by extension, how that might empower citizens trying to prove to local officials that there’s a problem that needs to be addressed. Historical 311 data sets, covering complaints from 2004 to 2009 are also available.
NYC Open Data was structured to allow for some user feedback. “People can comment on the data set itself,” said Mr. Chhugani. “You can even go down to the cell level and comment around a single cell. You can suggest a data set that’s good.”
Nicholas Sbordone DOITT’s director of external affairs, noted that the number of data sets available for Big Apps 3.0 is about 750. In fact, the 230 new data sets available exceeds the total number of data sets from the first Big Apps contest.
The big push now seems to be towards getting city agencies to automate the publishing of their data, perhaps by writing a script that connects to NYC Open Data. “Wherever it makes sense and it’s not cost prohibitive, at least,” said Mr. Chhugani. “Even if its daily or twice a week, so we don’t have to keep uploading the data sets manually. In the past, we really only had one way of doing it, which was we keep bothering them and calling them and just be the biggest pain in the neck to try to get the data.”