WS GCP Part4: Waffle server deployment to GCP - Gapminder/waffle-server GitHub Wiki

WARNING! This article may change without notice since both GCP and Waffle Server are in active development

This instruction will help you to deploy Waffle Server into Google Cloud
Be sure you've followed the previous chapters:



1. Decide on an environment you want to deploy to.

The possible values are dev, test, prod. Here we will use test as example.



2. Create a config file for test environment

touch deployment/gcp_scripts/settings_gapminder_test.json
nano deployment/gcp_scripts/settings_gapminder_test.json

note: here "test" in the end stands for environment and "gapminder" is hardcoded

Paste the following config into the file and replace as instructed:

{
  "BILLING_ACCOUNT": "<GO TO GOOGLE CLOUD GUI AND TAKE IT FROM BILLING>",
  "OWNER_ACCOUNT": "<ENTER YOUR GCLOUD ACCOUNT EMAIL>",
  "DEFAULT_PROJECT_NAME": "<ENTER PROJECT NAME WITHOUT ENVIRONMENT IN THE END, example: umbrella-ws1>",
  "REDIS_PORT": 6379,
  "MONGO_PORT": 27017,
  "MONGO_DB": "ws_ddf_test",
  "DEFAULT_USER_PASSWORD": "<ENTER PASSWORD YOU WILL LATER USE TO IMPORT DATASET, like 123456>",
  "PATH_TO_DDF_REPOSITORIES": "ddf",
  "NEW_RELIC_LICENSE_KEY": "",
  "LOGS_SYNC_DISABLED": true
}

Exit and save the file (Ctrl + X, then Y, then Enter)



3. Initialise google cloud SDK:

gcloud init

note: if you get permissions error, run sudo chown -R $USER ~/.config/

  • Choose Log in with a new account. Confirm action: yes

  • You will be given a long URL. Copy it and paste it to browser

  • Pick your google account for Google Cloud, authenticate if required

  • Press ALLOW — give Google Cloud permissions for gcloud SDK

  • Copy the pass code

  • Paste it to the terminal, where it asks "Enter verification code"

  • Choose Create a new project

  • Enter project name same as in config but with the environment -test in the end.

  • Example: umbrella-ws1-test (compare with the example in the config JSON given above)

  • Check that you have succeeded. Run gcloud config list

The result should contain your Google account used in GCP and your project name with -test in the end



4. Set billing account for the new project

  • Return to online Google cloud GUI

  • Press on the project selector, then manage resources, refresh the page

  • Click on the new project umbrella-ws1-test (or whatever your project name was)

  • Go to Menu > Billing and press [Link a billing account], press [SET ACCOUNT]

  • You should now have 2 projects linked in your billing page



5. Enable the necessary APIs for your project

  • Go to Menu > APIs and services, then Library

  • Check that the correct project name is displayed in the top bar.

  • Search and enable the following APIs. Confirm “Enable billing” if needed.

    • Container registry API
    • Google Kubernetes Engine API
    • Logging API (Stackdriver)
    • Google Compute Engine API (if not already)

  • Give it a minute to spin up



6. Run the deploy script

Return to the terminal again

cd //home/$USER/waffle-server
npm run deploy:test

note: the deploy takes about 15 minutes, when it doesn’t crash
note: if this deploy fails at some point and you need to do it again, you need to first clean up the mess (but don’t do it yet). You do that by running the "unpublish" script VERSION=2.12.2 NODE_ENV=test npm run unpublish. Here test is your environment name, version you can look up by running cat package.json

Troubleshooting

  • Some red lines are ok during deploy script

  • If no docker permission, run sudo usermod -a -G docker $USER, then sudo reboot, then run unpublish script, then try to deploy again. It is important to completely log out and log back in again or restart the system after usermod.

  • If you get error after a command but then you run the same command from what you see in the terminal history and you get a nice response (see example below), then it’s a timing issue. Unpublish and try again.

  • Deploy process may crash if the billing is not connected or if the APIs are not enabled. Check if you have properly done steps 5 and 6



7. Checking the deploy results

  • The deploy should result in two IP-addresses: one for threshing machine (TM), one for load balancer (LB). Note them down!
RESULTS:
 TM: <TM EXTERNAL IP>
 LB: <LB EXTERNAL IP>

note: TM is used to import datasets and serve assets, like maps. LB is used to serve DDF data

note: Sometimes the deploy ends with null for LB address. In this case wait a minute after deploy has finished and execute the latest command from the log: kubectl get service test-lb-2-12-2 --output=json. The IP will be in one of the objects below

  • You can also find both addresses in Google Cloud web GUI: TM external ip is in Compute engine, at the instance with "tm" in it. LB external ip is in Kubernates engine —> Discovery and LB, under “Endpoints”

  • To test if the machines are up, open both IP’s in new browser tabs:

<TM EXTERNAL IP>
<LB EXTERNAL IP>

Expected result for each would be:

If you have it, then you know the machines are up and running.

  • A more elaborate test is the following. Go to these links:

http://<TM EXTERNAL IP>/api/ddf/assets/my-dataset/assets/world-50m.json

http://<LB EXTERNAL IP>/api/ddf/ql/?_select_key@=concept;&value@=concept/type;;&from=concepts&where;&language=en&dataset=my-dataset

The expected response for each is the following:
{"success":false,"message":"Dataset with given name was not found"}

This means WS is working, but the dataset you're looking for isn't imported. Proceed to installing a client ws-cli and uploading a dataset.

⚠️ **GitHub.com Fallback** ⚠️