High level System Design - AndreSand/airquality-mx GitHub Wiki

Good documentation example:

AirQualityMx high level system design Definitions Acronyms AQI: Air Quality Index AWS: Amazon Web Services API: application program interface SQS: Simple Queue Service (from AWS) SNS: Simple notification service(from AWS) US EPA: United States Environmental Protection Agency MIT License: Permissive License by Massachusetts Institute of Technology used in open-source software. WAQI: The World Air Quality Index project City feed: represents the data feed from a supported city, includes one or more stations. Abstract This project aims to use public data of air quality stations located in Mexico, and use this data to send notifications to a given user in a near-realtime fashion. The system will be Cloud based and notifications may be published to the public while providing corresponding attribution(s) to the data source(s).

Goal/Objectives Provide a suitable system architecture which would freely provide near-realtime notifications regarding air quality in Mexico. This will allow users to get notified if air quality changes allowing them to knowingly take actions to safeguard their health, for instance deciding not to go out if airquality is not good, or doing outdoors activities when air quality is good. Provided AQI will use the AQI US-EPA 2016 standard. This high level design does not inted to dive into the specifics of implementation of the presented components.

Additionally, this project targets the open-source software development community, to share the code of this project, as long as guidance/practices/methodologies used in its development.

Stakeholders Project owner: @Jorge Villalobos Users: anyone interested in using or actively using this system with the intention of receiving air quality notifications in a supported area(Mexico). Open-source community: anyone insterested in using this software for learning purposes or those allowed by MIT License. Assumptions This is a high level design, is not ment to hold specifics of implementation of the presented components on this system. Each "feed" or "city feed" represents all the stations within a Mexican city. Mexico city is an exception and counts as 16 "feeds" due to it's town hall("delegaciones") system. Adding support to a new city(becoming a "supported city feed") is treated as separate from this design, assume this is done manually. Data obtained from the World Air Quality Project programmatic API is considered the source of truth Data retrieval frequency is initially set at every 5 minutes. Limitations & Unknowns As per WAQI's terms "The data can not be redistributed as cached or archived data" WAQI's data seems presented as "real-time", however there's no confirmed update frequency for specific cities. There's no known push method to listen for data changes in WAQI's API, so we "pull" every 5 minutes via GET HTTP request. Supported use-cases System is able to access AQI data from WAQI's programmatic API. Users can be notified when any station located in a determined city reaches 51 points or more in the AQI scale Users are notified via twitter(users should follow a determined Twitter account to access messages) Notified users can access a text message containing the AQI, Air Pollution Level(Good, Moderate, Unhealthy for sensitive groups, Unhealthy and Very Unhealthy) according to the AQI scale System is able to publish text messages as tweets on a determined twitter account with AQI Text messages delivered by the System must include attribution to WAQI and any other data source(such as, but not limited to organizations, institutions, public APIs, person/people, etc). System design should to support email notifications in the long term. Out of scope Any form of user data collection: no data will be collected at this stage. However if a thirdparty uses the source code of this project under the terms of the MIT License, they might be able to do so. Notifications messages specific to an individual "station" within a city. Notifications on areas that are not located in Mexico. Search/Lookup functionality Automation of setup for adding support of cities. Proposal Architecture The proposed system will used Cloud based services in AWS, it relies on Cloudwatch events to trigger periodic calls to the data source, and uses AWS Lambdas, SNS topics and SQS queues to deliver the data to its final destination.

Architecture diagram In the following diagram find the components that are both external and part of the the system.

airQuality-mx-design

External components: WAQI API: source of data through a JSON API, it is limited to 1000 requests per second. Twitter: Platform where a "Twitter Publisher" lambda would post tweets. Other/Email: Any email platform that would require it's own lambda or mechanism in order to send in notifications. Owned components: WAQI CityFeed Lambda: Triggered every 5 minutes by a programmatic Cloudwatch event, This lambda performs one or more GET requests to the WAQI API to obtain data for a single city feed. If a city has N air stations collecting data, the lambda would perform 10 requets(one per station) to gather complete data for such city. Data will be packed in a single JSON object and publish into an SQS queue. CityFeed SQS: will serve as delivery mechanism for raw city feed data obtained by WAQI CityFeed Lambda. Feed Processor Lambda: consumes CityFeed SQS messages, and processes them to produce a Notification object. Notification object which contains the final notification message. Feed Processor Lambda will publish this to one or more SNS topics for publishing through different delivery methods. Publisher SNS Topics: each publisher sns topics represents a delivery mechanism of a notification, as such a Twitter publisher SNS topic should be consumed by a subscriber(lambda) capable of delivering the message to the Twitter platform. This pattern allow extension of other specialized delivery mechanisms, such as emal, or any other not yet implemented. Publisher Lambdas: a publisher lambda funcions consumes a designed SNS topic, to obtain the notification message and executes the logic to deliver it to it's final recipient. An example of a publisher Lambda, is a Twitter Publisher lambda, which should be capable of taking any message received through the SNS topic, and publish it directly on the Twitter platform, for instance, through a "tweet" of a determined account. End-to-End flow of notifications sequence diagram airQuality-mx-sequence-diagram

Data structures Sample WAQI API /feed/:city response { "status": "ok", "data": { "aqi": 33, "idx": 391, "attributions": [ { "url": "http://sinaica.inecc.gob.mx/", "name": "INECC - Instituto Nacional de Ecología y Cambio Climático", "logo": "Mexico-INECC.png" }, { "url": "https://waqi.info/", "name": "World Air Quality Index Project" } ], "city": { "geo": [ 20.680450840176, -103.39873310432 ], "name": "Vallarta, Guadalajara, Mexico", "url": "https://aqicn.org/city/mexico/guadalajara/vallarta" }, "dominentpol": "o3", "iaqi": { "co": { "v": 2.1 }, "h": { "v": 11.3 }, "no2": { "v": 5.8 }, "o3": { "v": 33 }, "p": { "v": 845 }, "pm10": { "v": 21 }, "t": { "v": 33.1 }, "w": { "v": 5 }, "wg": { "v": 13 } }, "time": { "s": "2020-04-09 19:00:00", "tz": "-05:00", "v": 1586458800 }, "debug": { "sync": "2020-04-10T09:26:23+09:00" } } } Note: it was decided that specific structures in this project a not part of this high level design.

Implementation plan See assumptions I'm just one developer, so I'll keep this simple.

Complete everything -> days Never surrender -> weeks Publish process on yotube -> not even god knows Costs Monthly Cost (30 day) Costs exclude email/sms publishing

Up to $20.66 (Mexico city, Guadalajara, Monterrey) Up to $58.63 for 100 cities (Mexico city is special case, counts as 16 cities on its own) Details: how cost was calculated Main cities to support are Mexico City, Guadalajara and Monterrey, as Mexico city is separated in 16 town halls, we shall consider initial estimation of 18 city feeds. And also calculate cost in the long term to support up to 100 city feeds in Mexico.

Initial monthly cost (Considering up to 18 city feeds): 19.90 + 0.57(CloudWatch) + 0.19(Sqs) = $20.66 USD

Long Term monthly cost (up to 100 city feeds): 55.00 + 2.60(CloudWatch) + 1.03 (Sqs)= 58.63

Estiamted cost goes as follows (per supported city feed):

Cloudwatch events: $1.00US per 1 Million Estimated calls per month: 8640 (every 5 mins for 30 days) cloudwatch cost per feed per month: $0.02592 USD SNS topic messages: No charge for deliveries to Lambdas or SQS on proposed design If opt for SNS email/email-JSON delivery, cost is $2.00 per 100,000_ SMS messages maximum price for both USA and Mexico networks is 0.027$USD per message SQS standard queue: $0.0000004 per request Max Monthly cost per feed: $0.010368‬ USD Lambda functions: combined cost based on number of requests (assuming 128MB memory and execution lasting 10 seconds) Waqi Get City Feed (avg 8640 requests per 30 day executions per City feed) Feed Processor (avg 8640 requests per 30 day executions per City feed) Publisher Lambda:(avg 8640 requests per 30 day execution per City feed) total cost detailed: given (1M requests / (3*8640) requests) -> 1Million is 38.59x monthly requests per feed. Hence 21.03 / 38.59 would be the approximate cost of monthly usage. per feed. -- total cost: ~$0.55USD x NumberCityFeeds Monthly(30 day). -- ** 30 day number of requests per City Feed **: 25920 Lambda detailed pricing reference Memory: $0.0000002083 - 128MB memory on 100ms -> 0.00002083 per 10 seconds/128MB Request: $0.20 per 1M requests

Combined: assuming each lambda request lasted 10 seconds and used 128mb of memory during this execution. Duration cost per 1 Million requests: 0.00002083 * 1000000 = $20.83 Estimated combined cost of 1 Million requests: 20.83 + 0.20 = $21.03 USD