7. Cloud Setup - sunny-day-flooding-project/tutorials_v1 GitHub Wiki

Table of Contents

  1. Tutorial Overview
  2. Equipment
  3. Overview of Tools
  4. Setup OpenShift
  5. Build Containers
  6. Connect To and Format the Postgres Database
  7. Setup Sensors and Cameras

Tutorial Overview

This page walks through how to set up cloud components for the Sunny Day Flooding Sensors (SuDS) framework.

Equipment List

Access to a computer, with applications for running Python or R code.

Overview of Tools

Main tools

The SuDS framework relies on a few tools that require users to sign up and create accounts:

Other tools

Most code required for SuDS is written in R or Python, so be sure to have those languages and any IDEs (integrated development environments) installed on your local machine for testing/debugging. For R, we recommend RStudio, and for Python, we like VS Code

To connect to the Postgres database that will be created in OpenShift, it is helpful to have pgAdmin or some other database management software installed on your local machine. Openshift Command Line tools are needed to set up port forwarding and access the database.

The repos that are built into Docker containers include a Dockerfile that describes the build characteristics. Understanding how Docker works is helpful, but you do not need to have it installed locally unless you want to test builds locally.

Setup OpenShift

The SuDS uses the OpenShift platform to host docker containers that contain the:

  • 🐘 Postgres database - stores all of our data
  • 🔄 Processing functions - Recurring functions that processes raw sensor data, removes drift, sends flood alerts, etc. Code exists as two repos - sdfp-processing for converting raw pressure data into water depth, and sdfp-drift for removing drift and sending alerts.
  • 🗂️ Data API - How we send data from sensors to database. Code is here
  • 📷 Photo API - How we send pictures from gateways to our Google Drive storage. Code is here
  • 📈 Data Viewer - Visualizes real-time data from sensors and cameras. Code is here

Structure

Each docker container is its own "Deployment" within OpenShift (the little blue circle thingamabobs in the image below), with each Deployment pulling code from a GitHub repo and building a container. Together, the containers make up a Kubernetes cluster.

Here is an annotated screenshot of the setup:

✏️ NOTE

We use Red Hat's OpenShift platform because we are fortunate to have access through UNC ITS' Carolina CloudApps program. We have not tested a set up with any other container hosting platform. A similar structure could be created using other container hosting platforms or cloud database systems.

Creating Deployments

To create a deployment:

  1. Click +Add in the left sidebar
  2. To create the Postgres database container, click the Developer Catalog/Database option and follow the instructions to build a Postgres database container (w/persistent storage) from an OpenShift-supplied template.
  3. To create any of the other containers, click the Git Repository/Import from Git option. Paste in the URL of the GitHub repo for the container you want to create (These are linked at the top of this page under Setting up OpenShift)
  4. For any deployment that will need to be accessible from a URL, make sure to set up a Route that connects to the proper port for each container. This can be found in the Dockerfile in each repo. For example, the port that would need to be exposed with a route for the data viewer would be port 3838 (see Dockerfile, line 33)

Environment variables

Each container has a few environment variables that are supplied through OpenShift. This strategy keeps the environment variables secret, so instead of including sensitive password info in your code, your code references an environment variable that is then supplied by OpenShift.

All environment variables

This is a list of all environment variables needed:

Variable Description
GOOGLE_JSON_KEY A string denoting the content of a JSON key file that allows access to your Google Account
GOOGLE_SHARED_DRIVE_ID The ID of the shared drive where images are stored. The ID can be found at the end of the URL of the drive you wish to save images to
GOOGLE_SHEET_ID The ID of the Google Sheet where flood measurements are automatically written
GOOGLE_DRIVE_FOLDER_ID The ID of the folder within the shared drive that you wish to save images to
POSTGRESQL_PORT Usually 5432. Port of your Postgres database
POSTGRESQL_USER Admin
POSTGRESQL_PASSWORD The password of your Postgres database
POSTGRESQL_HOSTNAME or POSTGRESQL_HOST Ours is structured [postgres deployment name].[project name].svc.cluster.local, replacing the deployment and project names with your own
POSTGRESQL_DATABASE The name of your database
MAILCHIMP_KEY The Mailchimp API Key
MAILCHIMP_LIST_ID The ID of your audience list
MAILCHIMP_INTEREST_ID The ID of your audience interest category, which in this case is the location names which people have signed up to receive alerts from
ADMIN_PSWD Password to sign in as Admin on Data Viewer to access additional sensor info
FIMAN_URL API URL to collect data from NC FIMAN gauges
username Username to authenticate API usage
password Password to authenticate API usage

This is a list of all environment variables broken down for each container:

🔄 Processing function

  • GOOGLE_JSON_KEY
  • GOOGLE_SHARED_DRIVE_ID
  • GOOGLE_SHEET_ID
  • GOOGLE_DRIVE_FOLDER_ID
  • POSTGRESQL_PORT
  • POSTGRESQL_USER
  • POSTGRESQL_PASSWORD
  • POSTGRESQL_HOSTNAME
  • POSTGRESQL_DATABASE
  • MAILCHIMP_KEY
  • MAILCHIMP_LIST_ID
  • MAILCHIMP_INTEREST_ID
  • FIMAN_URL

🗂️ Data API

  • POSTGRESQL_PORT
  • POSTGRESQL_USER
  • POSTGRESQL_PASSWORD
  • POSTGRESQL_HOSTNAME
  • POSTGRESQL_DATABASE
  • username
  • password

📷 Photo API

  • POSTGRESQL_PORT
  • POSTGRESQL_USER
  • POSTGRESQL_PASSWORD
  • POSTGRESQL_HOSTNAME
  • POSTGRESQL_DATABASE
  • GOOGLE_JSON_KEY
  • GOOGLE_DRIVE_FOLDER_ID
  • username
  • password

📈 Data Viewer

  • POSTGRESQL_PORT
  • POSTGRESQL_USER
  • POSTGRESQL_PASSWORD
  • POSTGRESQL_HOST - Note this one ends with "HOST". This should be changed to match others in the future
  • POSTGRESQL_DATABASE
  • ADMIN_PSWD
  • FIMAN_URL

PersistentVolumeClaims

There are 2 PersistentVolumeClaims, which are persistent storage locations that are mounted to a deployment.

  1. postgresql - This volume is automatically generated when following the OpenShift template for creating a persistent storage Postgres database. Ours is 2 GiB capacity.
  2. photo-storage - This volume must be manually created and mounted to the Photo API deployment at the path /photo_storage. To do this, click on the upper left Developer dropdown tab and select Administrator. Expand the Storage dropdown on the left sidebar and select PersistentVolumeClaims. Click the blue button at the top right of the screen called Create PersistentVolumeClaim and follow prompts to name it, select Shared access (RWX), set the size to 2 GiB, and click Create. After creation, this volume can be mounted to a deployment by editing the volume.

Build Containers

After creating your OpenShift Deployments and Postgres database deployment configuration, you can work locally to update any code served in the containers.

To build a new version of a container:

  1. Push your code changes to your GitHub repo that is linked to your Deployment
  2. Navigate to you OpenShift Console, click on the Topology tab on the left while in Developer view.
  3. Click on the Deployment that you want to update. A right sidebar will appear - click the Resources tab.
  4. There will be a subheading called Builds, and you can click on the Start Build button to build the latest code from GitHub.

Connect To and Format the Postgres Database

Once you have created your Deployment of the Postgres database container, you can connect to it from your local machine using the OpenShift Command Line tools and pgAdmin.

The SuDS uses particular table names and schema in a Postgres database, and we provide a .txt file that can be used to recreate the structure of our database.

OpenShift Command Line Interface (CLI)

  1. Install the CLI following instructions here

✏️ NOTE

We use the VS Code IDE for the CLI.

  1. Once installed, follow the directions to login

  2. Navigate to your OpenShift Console, the Topology tab, then click on the postgresql deployment. A right sidebar will appear - click on the Resources tab and note the name of the Pod. If your Postgres Deployment is called "postgresql", the pod name will likely be postgresql-1-[some string of numbers and letters]. Note the name of this pod.

  3. Port forward your local Postgres port that pgAdmin will use to connect to the Postgres pod's 5432 port. You can use the following code in the OpenShift CLI.

oc port-forward [insert pod name here] [local port]:5432

  1. Connect to the remote database using pgAdmin (or another database management software) using your local port that is forwarding to the remote port.

  2. "Restore" your database using this .txt file (link coming soon. May become a .sql file). The .txt file is the output of pg_dump using the schema_only option.

  3. Once you have the database structured, you can move on to adding data & sensor/camera sites! 👇

Setup Sensors and Cameras

Sensor and camera sites can be set up using the data API and photo API.

Adding a sensor site

Sensor site info is stored in a database table called sensor_surveys. Make a new site by adding a survey with the write_survey function:

  1. Navigate to the data API write_survey endpoint
  2. Click the "Try it out" button on the right side of the screen.
  3. A window with a JSON array is now editable underneath the heading "Request Body". Edit this with the site info that you want to add. The date_surveyed should be in '%Y%m%d%H%M%S' UTC time.
  4. Press Execute

Viewing sensor site info

  1. Navigate to the data API get_surveys endpoint
  2. Click the "Try it out" button on the right side of the screen.
  3. Fill the sensor_ID box with the name of a sensor site (e.g., "BF_01") or "all"
  4. Press Execute

Editing a sensor site

This is currently only able to be changed by directly editing the Postgres table. In the future, an API endpoint on the photo API will be created to edit sensor locations.

Adding a camera site

Camera info is stored in a database table called camera_locations. Make a new camera by adding camera info with the write_camera function:

  1. Navigate to the photo API write_camera endpoint
  2. Click the "Try it out" button on the right side of the screen.
  3. Fill in the required values. Place should be the full name of the city and state. For example, "Carolina Beach, North Carolina". The camera ID, should be the letters "CAM", underscore, a two letter abbreviation of the place, another underscore, and a unique two digit number. For example, "CAM_CB_01".
  4. Press Execute. You should see a "SUCCESS!" response

Viewing camera site info

  1. Navigate to the photo API get_cameras endpoint
  2. Click the "Try it out" button on the right side of the screen.
  3. Fill the camera_ID box with the name of a camera site (e.g., "CAM_BF_01") or "all"
  4. Press Execute

Editing a camera site

This is currently only able to be changed by directly editing the postgres table. In the future, an API endpoint on the photo API will be created to edit camera locations.