7. Cloud Setup - sunny-day-flooding-project/tutorials_v1 GitHub Wiki
Table of Contents
- Tutorial Overview
- Equipment
- Overview of Tools
- Setup OpenShift
- Build Containers
- Connect To and Format the Postgres Database
- Setup Sensors and Cameras
Tutorial Overview
This page walks through how to set up cloud components for the Sunny Day Flooding Sensors (SuDS) framework.
Equipment List
Access to a computer, with applications for running Python or R code.
Overview of Tools
Main tools
The SuDS framework relies on a few tools that require users to sign up and create accounts:
Other tools
Most code required for SuDS is written in R or Python, so be sure to have those languages and any IDEs (integrated development environments) installed on your local machine for testing/debugging. For R, we recommend RStudio, and for Python, we like VS Code
To connect to the Postgres database that will be created in OpenShift, it is helpful to have pgAdmin or some other database management software installed on your local machine. Openshift Command Line tools are needed to set up port forwarding and access the database.
The repos that are built into Docker containers include a Dockerfile that describes the build characteristics. Understanding how Docker works is helpful, but you do not need to have it installed locally unless you want to test builds locally.
Setup OpenShift
The SuDS uses the OpenShift platform to host docker containers that contain the:
- 🐘 Postgres database - stores all of our data
- 🔄 Processing functions - Recurring functions that processes raw sensor data, removes drift, sends flood alerts, etc. Code exists as two repos - sdfp-processing for converting raw pressure data into water depth, and sdfp-drift for removing drift and sending alerts.
- 🗂️ Data API - How we send data from sensors to database. Code is here
- 📷 Photo API - How we send pictures from gateways to our Google Drive storage. Code is here
- 📈 Data Viewer - Visualizes real-time data from sensors and cameras. Code is here
Structure
Each docker container is its own "Deployment" within OpenShift (the little blue circle thingamabobs in the image below), with each Deployment pulling code from a GitHub repo and building a container. Together, the containers make up a Kubernetes cluster.
Here is an annotated screenshot of the setup:
✏️ NOTE
We use Red Hat's OpenShift platform because we are fortunate to have access through UNC ITS' Carolina CloudApps program. We have not tested a set up with any other container hosting platform. A similar structure could be created using other container hosting platforms or cloud database systems.
Creating Deployments
To create a deployment:
- Click
+Add
in the left sidebar - To create the Postgres database container, click the
Developer Catalog/Database
option and follow the instructions to build a Postgres database container (w/persistent storage) from an OpenShift-supplied template. - To create any of the other containers, click the
Git Repository/Import from Git
option. Paste in the URL of the GitHub repo for the container you want to create (These are linked at the top of this page under Setting up OpenShift) - For any deployment that will need to be accessible from a URL, make sure to set up a Route that connects to the proper port for each container. This can be found in the
Dockerfile
in each repo. For example, the port that would need to be exposed with a route for the data viewer would be port 3838 (seeDockerfile
, line 33)
Environment variables
Each container has a few environment variables that are supplied through OpenShift. This strategy keeps the environment variables secret, so instead of including sensitive password info in your code, your code references an environment variable that is then supplied by OpenShift.
All environment variables
This is a list of all environment variables needed:
Variable | Description |
---|---|
GOOGLE_JSON_KEY |
A string denoting the content of a JSON key file that allows access to your Google Account |
GOOGLE_SHARED_DRIVE_ID |
The ID of the shared drive where images are stored. The ID can be found at the end of the URL of the drive you wish to save images to |
GOOGLE_SHEET_ID |
The ID of the Google Sheet where flood measurements are automatically written |
GOOGLE_DRIVE_FOLDER_ID |
The ID of the folder within the shared drive that you wish to save images to |
POSTGRESQL_PORT |
Usually 5432 . Port of your Postgres database |
POSTGRESQL_USER |
Admin |
POSTGRESQL_PASSWORD |
The password of your Postgres database |
POSTGRESQL_HOSTNAME or POSTGRESQL_HOST |
Ours is structured [postgres deployment name].[project name].svc.cluster.local , replacing the deployment and project names with your own |
POSTGRESQL_DATABASE |
The name of your database |
MAILCHIMP_KEY |
The Mailchimp API Key |
MAILCHIMP_LIST_ID |
The ID of your audience list |
MAILCHIMP_INTEREST_ID |
The ID of your audience interest category, which in this case is the location names which people have signed up to receive alerts from |
ADMIN_PSWD |
Password to sign in as Admin on Data Viewer to access additional sensor info |
FIMAN_URL |
API URL to collect data from NC FIMAN gauges |
username |
Username to authenticate API usage |
password |
Password to authenticate API usage |
This is a list of all environment variables broken down for each container:
🔄 Processing function
GOOGLE_JSON_KEY
GOOGLE_SHARED_DRIVE_ID
GOOGLE_SHEET_ID
GOOGLE_DRIVE_FOLDER_ID
POSTGRESQL_PORT
POSTGRESQL_USER
POSTGRESQL_PASSWORD
POSTGRESQL_HOSTNAME
POSTGRESQL_DATABASE
MAILCHIMP_KEY
MAILCHIMP_LIST_ID
MAILCHIMP_INTEREST_ID
FIMAN_URL
Data API
🗂️POSTGRESQL_PORT
POSTGRESQL_USER
POSTGRESQL_PASSWORD
POSTGRESQL_HOSTNAME
POSTGRESQL_DATABASE
username
password
Photo API
📷POSTGRESQL_PORT
POSTGRESQL_USER
POSTGRESQL_PASSWORD
POSTGRESQL_HOSTNAME
POSTGRESQL_DATABASE
GOOGLE_JSON_KEY
GOOGLE_DRIVE_FOLDER_ID
username
password
Data Viewer
📈POSTGRESQL_PORT
POSTGRESQL_USER
POSTGRESQL_PASSWORD
POSTGRESQL_HOST
- Note this one ends with "HOST". This should be changed to match others in the futurePOSTGRESQL_DATABASE
ADMIN_PSWD
FIMAN_URL
PersistentVolumeClaims
There are 2 PersistentVolumeClaims, which are persistent storage locations that are mounted to a deployment.
postgresql
- This volume is automatically generated when following the OpenShift template for creating a persistent storage Postgres database. Ours is 2 GiB capacity.photo-storage
- This volume must be manually created and mounted to the Photo API deployment at the path/photo_storage
. To do this, click on the upper leftDeveloper
dropdown tab and selectAdministrator
. Expand theStorage
dropdown on the left sidebar and selectPersistentVolumeClaims
. Click the blue button at the top right of the screen calledCreate PersistentVolumeClaim
and follow prompts to name it, selectShared access (RWX)
, set the size to 2 GiB, and clickCreate
. After creation, this volume can be mounted to a deployment by editing the volume.
Build Containers
After creating your OpenShift Deployments and Postgres database deployment configuration, you can work locally to update any code served in the containers.
To build a new version of a container:
- Push your code changes to your GitHub repo that is linked to your Deployment
- Navigate to you OpenShift Console, click on the
Topology
tab on the left while inDeveloper
view. - Click on the Deployment that you want to update. A right sidebar will appear - click the
Resources
tab. - There will be a subheading called
Builds
, and you can click on theStart Build
button to build the latest code from GitHub.
Connect To and Format the Postgres Database
Once you have created your Deployment of the Postgres database container, you can connect to it from your local machine using the OpenShift Command Line tools and pgAdmin.
The SuDS uses particular table names and schema in a Postgres database, and we provide a .txt
file that can be used to recreate the structure of our database.
OpenShift Command Line Interface (CLI)
- Install the CLI following instructions here
✏️ NOTE
We use the VS Code IDE for the CLI.
-
Once installed, follow the directions to login
-
Navigate to your OpenShift Console, the Topology tab, then click on the
postgresql
deployment. A right sidebar will appear - click on theResources
tab and note the name of thePod
. If your Postgres Deployment is called "postgresql", the pod name will likely bepostgresql-1-[some string of numbers and letters]
. Note the name of this pod. -
Port forward your local Postgres port that pgAdmin will use to connect to the Postgres pod's
5432
port. You can use the following code in the OpenShift CLI.
oc port-forward [insert pod name here] [local port]:5432
-
Connect to the remote database using pgAdmin (or another database management software) using your local port that is forwarding to the remote port.
-
"Restore" your database using this
.txt
file (link coming soon. May become a.sql
file). The.txt
file is the output ofpg_dump
using theschema_only
option. -
Once you have the database structured, you can move on to adding data & sensor/camera sites! 👇
Setup Sensors and Cameras
Sensor and camera sites can be set up using the data API and photo API.
Adding a sensor site
Sensor site info is stored in a database table called sensor_surveys
. Make a new site by adding a survey with the write_survey
function:
- Navigate to the data API
write_survey
endpoint - Click the "Try it out" button on the right side of the screen.
- A window with a JSON array is now editable underneath the heading "Request Body". Edit this with the site info that you want to add. The date_surveyed should be in '%Y%m%d%H%M%S' UTC time.
- Press
Execute
Viewing sensor site info
- Navigate to the data API
get_surveys
endpoint - Click the "Try it out" button on the right side of the screen.
- Fill the
sensor_ID
box with the name of a sensor site (e.g., "BF_01") or "all" - Press
Execute
Editing a sensor site
This is currently only able to be changed by directly editing the Postgres table. In the future, an API endpoint on the photo API will be created to edit sensor locations.
Adding a camera site
Camera info is stored in a database table called camera_locations
. Make a new camera by adding camera info with the write_camera
function:
- Navigate to the photo API
write_camera
endpoint - Click the "Try it out" button on the right side of the screen.
- Fill in the required values.
Place
should be the full name of the city and state. For example, "Carolina Beach, North Carolina". The camera ID, should be the letters "CAM", underscore, a two letter abbreviation of the place, another underscore, and a unique two digit number. For example, "CAM_CB_01". - Press
Execute
. You should see a "SUCCESS!" response
Viewing camera site info
- Navigate to the photo API
get_cameras
endpoint - Click the "Try it out" button on the right side of the screen.
- Fill the
camera_ID
box with the name of a camera site (e.g., "CAM_BF_01") or "all" - Press
Execute
Editing a camera site
This is currently only able to be changed by directly editing the postgres table. In the future, an API endpoint on the photo API will be created to edit camera locations.