311 Project Proposal - Ayaz20009/311-Data-Visualization GitHub Wiki

Members

  • Si Mai
  • Joseph Park
  • Ayaz Shah
  • Sharfaraz Hassan

Background and Motivation

Since the creation of New York City’s open data law (Local Law 11 of 2012), the growing amount of data that is publicly recorded has burgeoned. And while the open data movement has made the data more accessible, interpreting it has not been any easier. According to Fast.Co Design, a business media brand, as quoted from this article, “… there wasn’t a tool to generate broad, location-based insights for people at the top”. As a result, a new dashboard was created for the Mayor’s Office in January of 2017. However the dashboard is not available for public use, and it is in this context that we believe we can offer our services to bridge this gap for those interested in exploring this data.

Project Objectives

The primary question that we are trying to answer with our visualization is: “How can we make sense of the massive 311 dataset?” This question is very broad, and can range from being able to spot outliers in the data to checking if the visualization conforms to expectations. We would like to learn how to utilize and integrate all the necessary tools to create an interactive dashboard that has a real world purpose. Doing so allows us to:

  • Communicate effectively with domain-experts to transform domain-specific knowledge into abstract tasks.
  • Learn tools that are relevant and used by many others in the real world.
  • Attain knowledge of how the various tools are assembled to form a working product.
  • Understand and apply visualization techniques and best practices.
  • Improve on being able to work successfully as a team.
  • Become proficient at identifying problems and resolving them.

Data

The NYC 311 data is public and is available from the NYC Open Data site. The complaints have been collected since 2010, and as of March 2017, contain nearly 15 million rows and 53 columns. Some of the columns that are of interest are Date Created, Complaint Type, Descriptor, City, and Location. The dataset can be accessed via its API or downloaded for offline manipulation.

Data Processing

Although the dataset contains 53 attributes, not all of them are complete. Some of the attributes may not be relevant to a complaint, and they are left empty or given an “unspecified” label. Examples include a set of school related attributes (“School Name”, “School Number”, “School Region”, “School Code”, “School Phone Number”, etc.), streets, vehicles, and more. The attributes of interest that we mentioned in the Data section are completed and require little to no data cleanup. As of now we not decided what quantities to derive from the dataset. With nearly 15 million records, we will need to process the data by aggregating it. More research will be needed to implement the data process.

Visualization

Since the main purpose of this project is to discover the unknown, we want to be able to visualize the data in many ways. A browser based dashboard application provides this at-a-glance view that is easily accessible. Discovery also involves interactivity; therefore we will have options that allow the user to filter and aggregate the data, which provide the capability to focus on areas of interest or see an overview. Note that the criteria above is not final and many include more after we have spoken to our domain expert. Based on the criteria mentioned above and our dataset, our initial visualization design will include a heatmap to display a spatial visualization. A color scale will indicate the magnitude of the 311 complaints in the heatmap. A filter option on the types of 311 complaints provides the user with the capability to drill down on. Another filter option that will further enhance drilling is time. Below the heatmap and options will consist of a number of visualizations between the attributes from the dataset.

Must-Have Features

The following must-have features are subject to change

  • Heatmap
  • Filtering/Aggregate options

Project Schedule

  • April 7 - Create static visualizations that are of interest
  • April 14 - Begin d3 dashboard – create the interactive vis from the static vis individually
  • April 28 - Integrate the vis together
  • May 4 - Testing of dashboard with feedback from domain expert
  • May 11 - Revised dashboard from feedback
  • May18 - Complete dashboard with presentation ready