Architecture Diagram

View a larger version of this diagram here.

Some Notes on the Redesign and the Platform's Architecture

The platform uses two Golang projects, Pivot and Diecast, to "decouple" parts of the system from each other. This means that instead of being intertwined, the two components can exist separately. Decoupling is a popular concept in system architecture because it allows you to swap out components of a system without rewriting all of the individual parts.

One of our goals with the redesign of the platform was to make it easier to change or upgrade pieces of the platform (like which database we use, or what backend language we use) without having to do a full rewrite. The original platform was built based off of a 2015 fork of MEAN.JS, and we found that it meant we were stuck with an old version of AngularJS (Angular 1) that discouraged possible pro-bono developers from contributing. (Angular 1 is a depreciated version that is significantly different from later versions, so developers would have to have been using it and working in the industry circa 2011-2015 when this version was popular. A complete rewrite of Angular (version 2) was released in 2016 and is completely different than v1, which is a source of controversy. ) This meant that contributing to the project was not only unappealing for experienced professionals, but extremely difficult for newer programmers to jump into, especially students- and it would have more limited pedagogical value (why teach students an old version of a framework?).

The current architecture gives us flexibility as versions, frameworks, and even languages enter and fall out of fashion, or as people with different talents join the BOP community. The BOP Digital Platform is always going to be a work in progress, but we're hoping that our architectural decisions will make it possible for developers from many different technical backgrounds to contribute.

Pivot

Pivot is a database abstraction service used to access, query, and aggregate data across a variety of database systems, written in Golang. It has been used/is currently in use in production-level systems at several for-profit companies dealing in at least two million database operations/day (the platform sees a couple hundred/day). You can learn more about Pivot or contact its developer through Github: https://github.com/ghetzel/pivot

In this project, Pivot is used to interact with the MongoDB database (https://www.mongodb.com/) we’re currently using. The benefit of using Pivot is that it allows you to write the backend code once, regardless of which (supported) database you use. (Usually, you write backend code that is specific to one/a handful of databases.) Pivot allows us to change or add databases in the future without rewriting the backend code.

Diecast

Diecast is a standalone web server that consumes REST services, passes the response into templates, and serves the results. This means that Diecast takes data from the platform’s API, puts it into the correct spots in a template, and creates the HTML, CSS, and/or JavaScript files that run in your browser.

Usually, the backend language/framework of an application dictates which templating languages you'll use. For example, many Python applications use the Django templating language, and many Ruby applications use Liquid or ERB.

The benefit of using Diecast is that it allows you to build the frontend of a web application completely independently of the backend, without being locked into any particular backend language or framework. While Diecast itself is built in Golang, and is essentially an extension of Go's templating language, you do not need to use Go as the backend language of your application to use Diecast- in our case, the BOP Digital Platform uses Python on the backend.

You can learn more about Diecast or contact its developer through Github: https://github.com/ghetzel/diecast

List of Services

MongoDB (formerly accessed via mLab, a database-as-a-service which was acquired by MongoDB in October 2018)

MongoDB is our database. This is a paid service.

Cloudflare

Cloudflare includes a suite of services- we are using it as a Content Distribution Network (CDN), Distributed Denial of Service (DDoS) mitigation service, and Domain Name System (DNS) nameserver. This is a free service (you can pay for additional features and upgrades).

• A CDN is used for caching and speeding up the delivery of static web assets (CSS, JavaScript, images). This means that Cloudflare serves certain platform files for us that are unlikely to change, instead of having to hit our servers directly. This speeds up how fast the site loads, and reduces the load on our servers (along with the corresponding costs in Amazon Web Services). Currently, Cloudflare diverts 37% of all requests to the platform.

• We are unlikely to suffer a DDoS attack, but the protection comes for free from Cloudflare, so it's a useful feature to have. If a DDoS attack actually did happen, the platform administrator should log into the Cloudflare interface and click the "I'm under attack" button.

• We are using Cloudflare as a DNS because it enables us to use the CDN functionality. As a bonus, for the platform administrator, their interface tends to be better than other DNS services.

GoDaddy

Amazon Web Services

Elastic Compute Cloud (EC2)

Elastic Beanstalk

Elastic Loadbalancing

Simple Storage Service (S3)

Route53

Simple Email Service

Simple Notification Service

Codeship

Sentry

Github (now acquired by Microsoft)

Mapbox

ArcGIS Online

NYC Open Data

NYC Open Data is the city's repository for official city government data. You can read more about open data in general in our wiki page, "Introduction to 'open data,' 'open source,' and how Billion Oyster Project uses these tools for restoration and education." You can read about our specific use of NYC Open Data on this page of our wiki: Accessing NYC Open Data's API to Import Relevant City Datasets

The platform relies on the following datasets from NYC Open Data's API:

• 2013-2018 Demographic Snapshot School

• 2017-2018 School Locations