Setting Up GCP - justinjfu/doodad GitHub Wiki

Install GCP Python Libraries

Currently there is no automated setup script for GCP. You will have to follow these instructions here to set up the Python clients for GCP:

  1. https://cloud.google.com/sdk/docs/quickstart-debian-ubuntu
  2. https://cloud.google.com/compute/docs/tutorials/python-guide
  3. https://cloud.google.com/storage/docs/reference/libraries#client-libraries-install-python

The two python packages you should have installed by the end of this are google-cloud-storage and google-api-python-client. You should also have created a GCP project - the name (project ID) of this project will be required as an argument by Doodad.

Set up a GCP Image

Next, you will need to create a GCE image with docker installed.

A simple way to do this is the following:

  1. Launch an f1-micro instance on GCP by opening the VM Console and creating an image with default settings. You may wish to select "Ubuntu" as your base OS to make installing docker easier.

  1. SSH into the instance, and install docker

  2. Add yourself to the docker group, and test that you can run the docker hello world (this may require a re-login).

sudo usermod -aG docker $USER
docker run hello-world

  1. Stop the instance on the GCP console (square button at the top bar)

  2. Go to the "Images", click "Create Image" and select the source disk corresponding to the image you made.

  1. Done!

(Optional) Add GPU support

  1. Install the CUDA drivers.

  2. Follow the installation instructions for nvidia-docker

  3. Test your new installation with the command:

sudo docker run --gpus all nvidia/cuda:9.0-base nvidia-smi

Create a Google Cloud Storage Bucket

  1. Go to the cloud storage console
  1. Click "Create Bucket" at the top, and configure your bucket name and pricing settings (multi-regional is quick and easy)

When using Doodad

GCP should be completely configured now. When using doodad, you can now pass the appropriate options to GCPMode:

mode.GCPMode(
        gcp_bucket=<YOUR BUCKET NAME HERE>,
        gcp_log_path='path/to/log/within/bucket',
        gcp_project=<YOUR PROJECT ID HERE>,
        gcp_image=<YOUR IMAGE HERE>,
        gcp_image_project=<YOUR PROJECT ID HERE>
        instance_type='f1-micro',
        zone='us-west1-a',
    )