Adding a New Non Java Client Library - GoogleCloudPlatform/pubsub GitHub Wiki

There are four main steps here:

1. Write a Task that exercises the publish/subscribe path
2. Add the Task to the main Framework
3. Write a startup script to run your Task on GCE
4. Add your Task to be packaged in run.py

We will assume that you have taken care of the first step, and cover the three remaining steps in detail below.

You can see an example of this with the CPS_GCLOUD_PYTHON_PUBLISHER.

The main idea is that you must create binary that starts a gRPC Server that implements LoadtestWorker. It should default to port 6000, but take a command line argument of --worker_port to override this, listening only on localhost. Your Task should be added to load-test-framework/<lang>_src/clients.

On Start, the worker should use the StartRequest to initialize itself, and on Execute, should publish batchSize messages or make one Pull request for cpsMaxMessagesPerPull messages, and then populate the ExecuteResponse with the results of the operation.

For a Publisher, it should be a latency value for each user message, that is [latency] * batchSize in Python.

For a Subscriber, it should be a latency per user message received, and ExecuteResponse must also be populated with the identifiers received. These should be parallel arrays, meaning received_messages[i] should be the same message that latencies[i] was recorded from.

There are a couple places you will now need to update in the load test framework. First you should add new command line flags to the Driver. The flag should be named --<cps|kafka>_<client_library_name>_<language>_<publisher/subscriber>_count.

In the run method of Driver, you will need to add a clause near the others to add your type to the Map, something like below except with Gcloud and Java replaced with the name and language of your client library:

if (cpsGcloudJavaPublisherCount > 0) {
  clientParamsMap.put(
      new ClientParams(ClientType.CPS_GCLOUD_JAVA_PUBLISHER, null), cpsGcloudJavaPublisherCount);
}

You will now need to add it to the ClientType enum in Client. You will also need to add cases for it in isPublisher and isCpsPublisher appropriately. You will also need to add a case to getSubscriberType that returns the type of the subscriber that will receive messages published by your task, or add it to a publisher case so you can subscribe to messages.

Last you will need to add it to the spreadsheet output in SheetsService and increment cpsPublisherCount, cpsSubscriberCount, kafkaPublisherCount, or kafkaSubscriberCount accordingly.

You must also create a startup script. You can see an example here. Mainly you will need to install and prerequisites and your client library worker, and start it before starting the Driver.

You must also add a clause in the main method that zips non-Java files so they can be installed in GCE. An example change is below:

  if not os.path.isfile('./target/classes/gce/cps.zip'):
    subprocess.call([
        'zip', './target/classes/gce/cps.zip',
        './python_src/clients/cps_publisher_task.py',
        './python_src/clients/loadtest_pb2.py',
        './python_src/clients/requirements.txt'
        './<my_client_library>/some/file.cc'
    ])