Google setup - RenderToolbox/RenderToolbox4 GitHub Wiki
Overview
The function rtbBatchRenderer (Render Toolbox Batch Renderer) and related functions manage the call of a docker container (usually pbrt-v2-spectral). This is a critical container that implements the rendering. The master version of the container is stored on dockerHub, and the normal control checks whether your local copy matches the master. If it does, your local copy is used. Otherwise, the new copy is downloaded to your disk or to a bucket on Google Cloud.
Whether you run locally or on a Google Cloud Engine (GCE) is determined by how you set up critical slots in the "hints" variable (see below). In many cases we want to render multiple images, and using a kubernetes cluster on the GCE greatly speeds up the rendering. This page describes how to configure your computer and set the parameters so that the Docker container with pbrt is executed on Google Cloud Engine.
Create project
To run on the Google cloud you need an account; to set up an account, you (or your lab) needs to provide a credit card.
Once you have an account, you can create a project from the cloud console web page. Click on the project pull down at the upper left of the web page and add a project, giving it a reasonable name. (Google adds some numbers to insure uniqueness). You will have to configure the project resources, mainly the properties of the storage bucket. The compute cluster used for running kubernetes will be created using Matlab functions within RenderToolbox4, so you don't have to worry about that.
Add user to the project
You control access to your project from the console page. Select the IAM & admin tab on the left. Choose the 'Add' button at the top of the screen and fill out the forms. You will have to assign a role. Typically, lab members are Project -> Editors. One person is the Project -> Owner.
Gcloud command line tools
The RenderToolbox4 functions address the GCE using a set of command line tools. The Matlab calls are just wrappers that invoke these tools. For the tools to run, the user must have access to the gcloud SDK. On a Linux machine, these should be installed for everybody.
On a Mac, you will have to download them for yourself. The SDK package can be downloaded for different operating systems, and it includes these functions gcloud, gsutil, kubectl. Once the archive is downloaded and extracted, run
./google-cloud-sdk/install.sh
Note that you will need to make an explicit call to install kubernetess tools
gcloud components install kubectl
Once you download and install, initialization is simply this
gcloud init
A typical output of the gcloud init is included at the bottom of this page.
(BW note) I have not yet set up the port and am getting a port 8080 error. I am reading these issues pages.
RenderToolbox4 hints settings
To invoke the GCE with a kubernetes cluster, you must set certain slots in the 'hints' variable created by RenderToolbox4. The critical variables are in hints.bathRenderStrategy.renderer. If these are set properly, then the function rtbCloudInit() will call the relevant system commands to spin up the kubernetes cluster on the GCE.
hints.batchRenderStrategy.renderer = RtbPBRTCloudRenderer(hints); % NOT SURE ABOUT ORDERING BUT THIS WORKS
hints.batchRenderStrategy.renderer.pbrt.dockerImage = 'gcr.io/primal-surfer-140120/pbrt-v2-spectral-gcloud';
hints.batchRenderStrategy.renderer.cloudFolder = ...
fullfile('gs://primal-surfer-140120.appspot.com',hints.recipeName);
%% Initialize for kubernetes on google cloud
rtbCloudInit(hints);
Data management
To run the calculation you will need to move data back and forth to the cluster. Right now, we generate most of the data locally (e.g., pbrt scene files). The only thing that we do on the cloud is move the fully configured files to the GCE and then run PBRT remotely. Of course, we also copy the outputs of the rendering back to the local machine.
The functions that manage the upload and download are rtbCloudUpload() and rtbCloudDownload().
We are implementing utilities to list the files that are on the cloud and to remove them (using the gcloud class that are developing the RenderToolbox).
When you render on the cloud, the rtbBatchRenderer() returns almost immediately; it just submits the jobs to be run. Before downloading, you want to check whether the jobs are completed. The current logic is that rtbCloudDownload() checks whether the jobs are completed. If they are not, it returns empty. You keep polling until the jobs are completed, at which point the downloads are executed.
There is an issue about pbrt-v2-spectral. Running on GCE we have to move the data from the bucket into the local machine. (This technology is in flux, because it is now possible to mount the same data on multiple machines). For now, however, we handle the data movement using a script that downloads from the storage bucket to the machine. This is an edit of the pbrt-v2-spectral container that is essential to run on GCE. The script downloads the data from the gcloud bucket, runs PBRT, and uploads the results to bucket.
Useful commands
Useful GCE and kubectl commands include finding the status of your running jobs
'kubectl get jobs --all-namespaces' - to check the running jobs
`gcloud --help` - to see the Cloud Platform services you can interact with.
`gcloud help COMMAND` to get help on any gcloud command
gcloud topic -h` - to learn about advanced features of the SDK like arg files and output formatting
RenderToolbox4 utilties
We have the classes.
gcloud
kubernetes
These are in progress.
We have
rtbKubectlConfig
rtbDockerConfig
rtbKubernetesExists
rtbDockerExists
rtbDockerTest
to make the paths to these programs right. The kubectl one is pretty lame, but at least it reminds us of what to do.
Running "gcloud init"
gcloud init
Welcome! This command will take you through the configuration of gcloud.
Your current configuration has been set to: [default]
You can skip diagnostics next time by using the following flag:
gcloud init --skip-diagnostics
Network diagnostic detects and fixes local network connection issues.
Checking network connection...done.
Reachability Check passed.
Network diagnostic (1/1 checks) passed.
You must log in to continue. Would you like to log in (Y/n)? Y
Your browser has been opened to visit:
https://XXXX-VERY LONG --- XXX
[90209:90243:0807/154855.970886:ERROR:browser_gpu_channel_host_factory.cc(103)] Failed to launch GPU process.
Created new window in existing browser session.
Created new window in existing browser session.
You are logged in as: [[email protected]].
Pick cloud project to use:
[1] primal-surfer-140120
[2] Create a new project
Please enter numeric choice or text value (must exactly match list
item): 1
Your current project has been set to: [primal-surfer-140120].
Your project default Compute Engine zone has been set to [us-west1-b].
You can change it by running [gcloud config set compute/zone NAME].
Your project default Compute Engine region has been set to [us-west1].
You can change it by running [gcloud config set compute/region NAME].
Created a default .boto configuration file at [/home/wandell/.boto]. See this file and
[https://cloud.google.com/storage/docs/gsutil/commands/config] for more
information about configuring Google Cloud Storage.
Your Google Cloud SDK is configured and ready to use!
* Commands that require authentication will use [email protected] by default
* Commands will reference project `primal-surfer-140120` by default
* Compute Engine commands will use region `us-west1` by default
* Compute Engine commands will use zone `us-west1-b` by default
Run `gcloud help config` to learn how to change individual settings
This gcloud configuration is called [default]. You can create additional configurations if you work with multiple accounts and/or projects.
Run `gcloud topic configurations` to learn more.