Google Cloud - multiparty/cardinal GitHub Wiki

This is a guide to setting up Cardinal as an orchestrator on Google Cloud.

These steps are adapted from Google's docs with some Cardinal-specific steps.

  1. Google Kubernetes Engine - Creating the cluster

    • Log into your Google Cloud account and navigate to your project. If you are the root account, you should not have any permissions issues for the remaining steps. If working with an IAM account, make sure the account has "Editor" permissions at least.
    • Navigate to the "Kubernetes Engine" tab from the menu on the left side.
    • Click "Create" under the "Clusters" tab, and choose the "Standard" option
    • Name your cluster in the "Name" field.
    • Select a zone from the drop-down.
    • On the sidebar on the left side, navigate to Nodes Pools > default-pool > Security and click the "Allow full access to all Cloud APIs" option. Cardinal cannot use the Google Cloud APIs without setting this option.
    • Finish by clicking "Create" at the bottom of the page.

  2. Repo setup

    • Open the shell in your project ("code" icon at the top right of the page).

    • Clone this repo into your project workspace using the shell, and change directories into it.

      git clone https://github.com/multiparty/cardinal.git

      cd cardinal

  3. Docker

    • Run the following command to set your project ID to make future steps easier. (You can find your project ID on the landing page of the project under "Project Info" on the top left of the main project page)

      export PROJECT_ID={PROJECT_ID}

    • Move the Dockerfile in the gke_docker directory into the main cardinal directory.

      mv gke_docker/Dockerfile .

    • Edit the following fields in the Dockerfile:

      • Change PROJECT_ID to match your Google Cloud project ID.
      • Change REGION to match the region of your cluster.
      • Set CONGREGATION to the image URI of the Congregation image you would like to use. See note below about Congregation images.
      • Set CHAMBERLAIN to the IP address or hostname of your deployed chamberlain server.
      • Set PROFILE to 'true' or 'false' depending on whether you would like this deployment to save profiling information for each workflow.
      • Enter the AWS credentials for the account where your cardinal results will be sent.
      • Set DESTINATION_BUCKET to the name of the bucket in the above AWS account where your results will be sent.
      • If you intend to start multiple clusters (i.e. one for each party) in this project, you will need to edit, build, and push this Dockerfile for each one. Please see below for a template Dockerfile.

    • The easiest way to work with Docker images in GCP is using Google's Container Registry, so we will be pushing our image(s) there. If prompted, authorize Cloud Shell to make Google Cloud API calls.

      • Run docker build -t gcr.io/${PROJECT_ID}/cardinal:{tag} . where tag can be filled with anything that helps you identify what version of this image you are using (i.e. cardinal:gke-east1).
      • Run gcloud services enable containerregistry.googleapis.com to enable the Container Registry API.
      • Run gcloud auth configure-docker to authenticate to the Container Registry API
      • Run docker push gcr.io/${PROJECT_ID}/cardinal:{tag} to push the image to the Container Registry.

  4. Kubernetes - Deploying your Cardinal image(s)

    You will need to give your cluster the right RBAC permissions in order for Cardinal to spin up the necessary pods and services for running computations.

    • Run gcloud container clusters get-credentials {NAME} --zone {ZONE} where ZONE is the zone you specified when you made the cluster. This will allow you to run kubectl commands on this cluster in the shell.
    • Run kubectl apply -f role_def.yaml and kubectl apply -f role_binding_def.yaml to give the cluster full Kubernetes API permissions.
    • Return to the Google Kubernetes Engine console and go to the "Workloads" tab on the left side.
      • Click "Deploy" either in the top menu or in the center of the page.
      • In the "Container" section, select "Existing container image".
      • In the "Image path" field, click "Select".
      • In the "Select container image" pane, select the cardinal image you pushed to Container Registry and click "Select".
      • In the "Container" section, click "Done", then click "Continue".
      • In the "Configuration" section, under "Labels", enter "app" for "Key" and your deployment name for "Value" (these should auto-fill anyway).
      • If you want to view the generated YAML file for your deployment, click "View YAML" under "Configuration YAML". This opens a YAML configuration file representing the two Kubernetes API resources about to be deployed into your cluster: one Deployment, and one HorizontalPodAutoscaler for that Deployment.
      • Close out of the YAML file if you opened it and then click "Deploy".
      • When the Deployment Pods are ready, the Deployment details page opens.
    • Click "Expose" from the Deployment details page. (If there is not already an info box present with this button, it will be under the "Actions" menu)
      • Set the "Target port" to 5000. If you changed the PORT environment variable in your Dockerfile, then set the "Target port" to that value.
      • From the "Service type" drop-down list, select "Load balancer".
      • Click "Expose" to create the Kubernetes Service.
      • When the Load Balancer is ready, the Service details page opens.
      • You can find the external IP address near the bottom of this page. Test that the deployment worked by going to this IP address - you should just see the word "homepage" when you do so.

You are now set up to run workflows using Cardinal! You can begin by trying to send a submit request from chamberlain.

Congregation

If you do not know of a publicly available Congregation Docker image that you would like to use, you are welcome to build, push, and use your own. Please see the repository catechism to see what the current configuration of the pushed Congregation images are. There you will see directories labeled by which backend is being used in that particular image. Inside each directory will be a push_pull.py script, a bash script, and a Dockerfile that specifies which libraries to clone to the image and then runs the bash script. To make a new congregation image, simply copy the file structure of an existing one, make the changes you would like to make, and then build and push the image in the same way that we built and pushed the cardinal images.

Dockerfile

Below is a template Dockerfile for deploying cardinal on GKE:

FROM python:3.7

RUN mkdir /cardinal
WORKDIR /cardinal
ADD . /cardinal/
RUN pip install --upgrade pip
RUN pip install -r requirements.txt

#------------------#
# CHANGE AS NEEDED #
#------------------#
# Environment information
ENV PORT=5000
ENV CLOUD_PROVIDER="GKE"
ENV INFRA="GCP"
ENV PROJECT="{project ID}"
ENV REGION="{cluster region}"
ENV CONGREGATION="{image URI of congregation image you would like to use}"
ENV CHAMBERLAIN="{IP/hostname of deployed chamberlain server}"

# profile flag - set to "true" to receive profiling timestamps with each workflow
ENV PROFILE="false"

# Information for pushing results to AWS S3 bucket
ENV AWS_REGION="{region of public-read results bucket}"
ENV AWS_ACCESS_KEY_ID="{access key for owner account of bucket}"
ENV AWS_SECRET_ACCESS_KEY="{secret access key for owner account of bucket}"
ENV DESTINATION_BUCKET="{name of public-read results bucket}"

CMD ["python", "/cardinal/wsgi.py"]