Cloud Services Overview

The Cloud Services API has been developed to allow providers of web based solution (e.g. Today's Plan, Xert) to enable upload, download and sync with GoldenCheetah. We have developed a standard approach for this, driven mostly by the desire to re-use common code when developing integration with lots of cloud services in version 3.5. The mechanism we developed means you don't have to worry about developing lots of code to support Oauth and zipfiles and async read and write. Instead, you develop using a framework.

This page describes how you do that.

When implementing an integration you will need to implement the CloudService class, re-implementing a number of core methods. In this wiki we are going to walk through the main methods and concepts you will need to re-implement. For an example of a real service, you should examine the Today's Plan implementation in TodaysPlan.h and TodaysPlan.cpp.

CloudService API - Service Definition and Configuration

Before we start, lets just introduce the CloudService class. The header CloudService.h defines the base class you will need to re-implement.

It has a number of methods you will need to implement that describe your web-services, the capabilities they have and the configuration they need.

Service Descriptions

We need to describe the service by giving it a name, description and logo.

virtual QString name() const;
virtual QString description() const;
virtual QImage logo() const;

When re-implementing the name should be short and not require translation. Typically it will be the main part of the website URL, examples like "Strava", "Today's Plan", "Polar Flow". Avoid long names as this is used to uniquely identify the service.

The description should be a short snappy tag-line, since it is used when listing services. Examples include "The smart cycling training site.", "The training social network site".

The image returned should be as hi-resolution as possible. Remember GC runs on platforms with 4k and 5k screens so the images should be large enough to render well on such hi-dpi screens. GC manages scaling down to smaller screens so you just need to provide the image. Please make sure the alpha (transparency) channel is set so the image can be rendered with a transparent background.

You will need to add the image to the src/Resources/images/services folder as a PNG file. It will also need to be added to the src/Resources/application.qrc file to ensure it is embedded within the binary when GC is built.

Service Type

We currently support syncing with services that provide Activity data and Measures data. Activities are workout recordings that contain sample-by-sample data, whilst Measures are things like weight, body-fat, HRV. Activity based sites will be integrated with the syncing processes at startup, import or manually by users. Measures sites will need to update code separately (see Measures code updates below).

virtual int type() { return Activities; }

The default implementation will set the type to CloudService::Activities, so you don't need to implement in this case. If your service provides Measures you will. It is possible to support both, in this case you would reimplement, with:

virtual int type() { return Activities | Measures; }

Service Capabilities

We will work with sites that offer lots of different services; some offer upload only, some will let you download, and some will let you query what's there as well as upload and download. When you implement you will need to define the capabilities of the service by re-implementing the capabilities method.

virtual int capabilities() const { return OAuth | Upload | Download | Query; }

The default implementation will define a typical service; it supports OAuth authentication, and provides Upload, Download and Query operations. If your service differs, you will need to reimplement. There are a few capabilities you can use:

• OAuth - The service can authenticate using an OAuth token, which will need to be authorised via the OAuth authentication process. Note that you will need to update OauthDialog to support your process flow. This is documented below in the section OAuthDialog.
• UserPass - The service authenticates using a simple user and password scheme.
• Upload - The service allows users to upload activities or measures.
• Download - The service allows users to download activities or measures.
• Query - The service allows users to query what activities or measures are available and stored on the site.

File Formats and Compression

The base class CloudService contains code to convert activity files between formats and to compress and uncompress data. So when implementing a CloudService you do not have to handle such processing. Insted you can specific the file formats and compression you need to use with your service.

To specify compression, you shoud set uploadCompression and downloadCompression to one of:

• none - no compression should be applied
• gzip - GNU zip compression should be applied
• zip - Standard zip compression

For file formats we already support any kind of file format in the download, but your cloud service may only be able to support a certain type, you can specify the type by setting filetype to one of:

• JSON - GoldenCheetah json format [the default, if not specified]
• TCX - Garmin Training Centre XML format
• FIT - Garmin / Dynastream Binary file format
• PWX - TrainingPeaks XML format

NOTE: FIT files do not support textual content, so any notes or metadata maintained in GC or the cloud service will be lost if this file format is used to transferring activity files

Here is the constructor from the TodaysPlan code, defining the file formats and compression to be used, note that the default format of JSON is used, since it is not specified:

uploadCompression = gzip; // gzip
downloadCompression = none;

Service keys and secrets

Most cloud services require two tokens to authenticate the application; a client key which is supplied by the service provider to the application and is a public (not secret) key, and a second consumer key which is supplied by the service provider and is private and must be kept secret.

Public keys can be defined in the source code, typically you will see them defined in settings.h, but they could equally be defined in your own source files. The name of the private key must be declared in Secrets.h. But the actual value is never posted in source files. Instead, the private keys are maintained by the build team who add the setting to the private build settings file gcconfig.pri.

As an example, the Strava, consumer key is declared in Secrets.h:

#ifndef GC_STRAVA_CLIENT_SECRET
#define GC_STRAVA_CLIENT_SECRET "__GC_STRAVA_CLIENT_SECRET__"
#endif

The value is then declared in gcconfig.pri (note the escaping used):

DEFINES += GC_STRAVA_CLIENT_SECRET=\\\"378b13........\\\"

And in the Strava specific section of OAuthDialog.cpp (See OAuthDialog below) it is accessed:

#ifdef GC_STRAVA_CLIENT_SECRET
    params.addQueryItem("client_secret", GC_STRAVA_CLIENT_SECRET);
#endif

This approach means that client secrets are never posted to public repositories (GoldenCheetah code on GitHub for instance), but the keys will need to be shared and maintained with the GC build team.

Service Registration and Instantiation

We have a cloud service factory that you will need to register your service with. A reference to the cloud service factory can be attained with:

const CloudServiceFactory &factory = CloudServiceFactory::instance();

The constructor for a CloudService must accept a context pointer, which may be NULL. This provides information to the service about the athlete and other important objects the service will need to interact with.

When you register with the factory you pass an object that has been created for a NULL context. This means the object you register cannot (and will not) ever be used to execute service operations. Typically the cloud service will be registered as follows:

static bool addTodaysPlan() {
    CloudServiceFactory::instance().addService(new TodaysPlan(NULL));
    return true;
}

static bool add = addTodaysPlan();

This will take place before any application code has been executed, and means the service will not have any context. In GC parlance, the context provides information about the current athlete and other information. It is expected that a service will map data from a GC athlete to an athlete in your service as a 1:1 relationship.

This means that before any operations can be performed the service will need to be cloned with a context provided and configuration injected. This process is performed by the cloud factory, but you will need to offer a clone method it can call for this purpose. Here is an example:

CloudService *clone(Context *context) { return new TodaysPlan(context); }

Across the rest of the GC codebase, whenever a service object is required it will be requested from the CloudServiceFactory. The request passes the service name required, and the context it is to be created for:

CloudService *newone = CloudServiceFactory::instance().newService("Today's Plan", context);

Additionally, the newService factory method will interrogate the configuration requirements of the cloud service and inject any available configuration.

Service Configuration

All cloud services need configuration settings, things like user names, passwords, athlete id, folder and so on. So to support this, the CloudService API uses a declarative model. Each CloudService is expected to populate a QHash called settings with a list of all the settings it needs.

These settings are then used by the add cloud wizard to control what the user is prompted for when setting up an account. At runtime, the CloudServiceFactory will inject the user suppled configuration into another QHash called configuration when the object is instantiated via newService() as detailed above.

In short; your cloud service specifies the configuration it needs via QHash settings and receives the configuration as supplied by the user via another QHash configuration. These settings can be set and queried using a getSetting and setSettting method.

Declaring configuration requirements

When populating the settings you provide a setting type such as CloudService::Username and a global settings symbol such as GC_MYSERVICE_USERNAME. The global settings symbol needs to be defined in Settings.h and follow the semantics there (see the comments in the file).

The following types of configuration can be declared, with each having a slightly different prompt or validation required during account setup.

• CloudService::Username - a username to be used when connecting to the service
• CloudService::Password - a password (not clear text) to provide when connecting to the service
• CloudService::OAuthToken - an Oauth token (clear text) provided via the OAuth protocol
• CloudService::Key - A service specific key that will be used during authentication
• CloudService::URL - The service URL to use for this account
• CloudService::DefaultURL - The default value to use for a URL (not used to prompt user)
• CloudService::Consent - A message to ask the user to consent to terms of service
• CloudService::Folder - A folder to use on your service (requires readdir/createfolder semantics)
• CloudService::AthleteID - The athlete identifer (requires listAthlete/selectAthlete semantics)
• CloudService::Localx - A setting maintained by your service but no user prompt required
• CloudService::Combox - A combo box selection that has predefined values the user selects from
• CloudService::Metadatax - A combo box selection of metadata field names

By way of an example, here is the code to setup the configuration for the Today's Plan cloud service (taken from the constructor):

// config
settings.insert(OAuthToken, GC_TODAYSPLAN_TOKEN);
settings.insert(URL, GC_TODAYSPLAN_URL);
settings.insert(DefaultURL, "https://whats.todaysplan.com.au");
settings.insert(Key, GC_TODAYSPLAN_USERKEY);
settings.insert(AthleteID, GC_TODAYSPLAN_ATHLETE_ID);
settings.insert(Local1, GC_TODAYSPLAN_ATHLETE_NAME);

The service requires OAuth credentials (see OAuthDialog below) and the user can override the url to connect to, but by default would use "https://whats.todaysplan.com.au". For some users who have private tenants they offer additional security whereby special key is required to connect (which can be expired on a per user basis). And additionally, you can select the athlete you want to sync with (coaches on Today's Plan can see data for the riders they coach) and during the athlete selection process the Today's Plan implementation also stores the athlete name.

As a result. the authentication page of the add account wizard looks like this:

Folder selection and creation

Storage based services like Google Drive and Dropbox require a folder to be selected to read and write activity files to. So to support this you can declare a CloudService::Folder setting. This is selected using a CloudServiceFileDialog which is a classic file dialog. In order for it to work your cloud service will need to offer endpoints for readdir (see Endpoints below) and also createFolder() since the dialog uses these endpoints to browse the storage and create a new folder.

Athlete selection

Training based sites like Today's Plan and Sixcycle require an athlete to be selected to sync with. When the athlete is self-coached they will typically select from a list of one athlete, which is themself. For coaches, they will typically authenticate as themselves but then select from a list of athletes they have a coaching relationship with.

The athlete selection is performed within the wizard and will offer a list of athletes from a call to the endpoint listAthletes and when an athlete is selected call the selectAthlete endpoint to notify the service, during the configuration process, that an athlete has been selected. At this point the service can collect its own internal settings or perform other setup. See listAthletes in the endpoints section.

Combo box

Some services will have configuration options that are highly specific to the service, to this end a combo box can be described and used to provide selections to the user. One example of this is the Google Drive storage service. The user is expected to select the scope of access being authorised, they can select from one of three options: drive, drive.file and drive.folder. To do this the following is added to the settings:

// config
settings.insert(Combo1, QString("%1::Scope::drive::drive.appdata::drive.file").arg(GC_GOOGLE_DRIVE_AUTH_SCOPE));

Presently the Combo1 is shown on the authentication screen, in later releases we may add Combo2, Combo3 and so on to other screens (if they are needed).

To unpick the specifier, it is provided as 4 or more strings separated by ::. Each of the parameters are used to control the behaviour of a combo box:

• 1 - setting name (typically defined in Settings.h)
• 2 - parameter name (a user friendly name for the parameter that is used to label the combo box)
• 3 - default value (the value it will have on first selection)
• 4-n - other values the user can select from.

Metadata field

You may want to post values using metadata fields. For example the Strava cloud service lets the user select a metadata field to use when populating the activity name on upload. The metada field setting is configured similarly to the combo box above, except there are only two parameters, separated by ::.

The first parameter is the setting name (typically defined in Settings.h), whilst the second parameter is the label to use when prompting the users.

As an example, here is the setup code for the Strava cloud service:

// config
settings.insert(Metadata1, QString("%1::Activity Name").arg(GC_STRAVA_ACTIVITY_NAME));

Configuration Injection, saveSettings, getSetting and setSetting

As noted above the CloudServiceFactory will inject previously configured settings for the service into the QHash configuration when the service is instantiated via CloudServiceFactory::newService(). The configuration that is injected is only the configuration that was requested in settings and as supplied by the user when setting up a cloud account in AddDeviceWizard.

The key here is that the AddDeviceWizard will store user supplied configuration stored in the configuration QHash. At the end of the wizard it will lookup settings that have been stored in the configuration QHash and write them to global settings using the appsettings methods.

If you need to add your own settings during this process (that do not involve user interaction) then you should also update the settings QHash and declare them in advance as local settings. Every local setting must use a separate key, at present there are 6 local keys supported (CloudService::Local1 thru CloudService::Local6), more can be added later if needed.

To set and get values in the settings QHash you should use getSetting and setSetting, both of which take two parameters; the setting name such as GC_TODAYSPLAN_ATHLETE_NAME, and a value e.g. "Mark Liversedge", in getSetting the second parameter is a default value to return if it is not present, whilst in setSetting it is the value to set.

You should never read or write appsettings directly, instead you must use setSetting and getSetting. This is because we may change the storage mechanism in the future -- and this is likely to happen if we ever support setting up more than one account per athlete for a single service.

The values are stored as QVariants (since we might store text, numbers, bools) this is example code from the TodaysPlan.cpp that is setting and getting the local setting GC_TODAYSPLAN_ATHLETE_NAME:

Set:

setSetting(GC_TODAYSPLAN_ATHLETE_NAME, athlete.name);

Get:

getSetting(GC_TODAYSPLAN_ATHLETE_NAME, "").toString();

Save:

CloudServiceFactory::instance().saveSettings(this, context);

With some services you will need to refresh tokens or save state along the way (after initial setup with the add cloud wizard). In this case you should update the local settings with setSetting and ask the CloudServiceFactory to save these permanently using the call above. For an example, see the open method of the SportTracks.mobi cloud service. It refreshes the access token on open and saves the retreived values using this approach.

Cloud Service API - Endpoints

Service endpoint implementations are optional depending upon the capabilities you declared and the configuration options you require. No implementations are absolutely required, since the base implementation can always be used. But the table below indicates the implicit dependencies between capabilities, settings and CloudService methods.

Dependency	open close	readdir	readFile	writeFile	listAthletes selectAthlete	createFolder home root	OAuthDialog
Capability
Upload	X			X
Download	X	X	X
Query	X	X
Config
OAuthToken	X						M
Folder	X	X				X
AthleteID	X				X

bool open(QStringList &errors)

Since most services require a connection to be established and things like session tokens and authentication to be performed before any operations can be performed we always call bool open(QStringList &errors) to establish such a connection.

If it returns false this indicates the open failed, and the QStringList &errors parameter will be updated with texts that can be displayed to the user to indicate why. A return of true indicates a successful open operation.

void close()

One all operations are completed the close() endpoint is called to clean-up any resources and close the connection. It is assumed this will always be successful.

bool readFile(QByteArray *data, QString remotename, QString remoteid)

The data pointer is used to store the received data, whilst the remotename and remoteid are as received from a prior readdir() call (see below).

Since there is no file type it is expected that the remotename extension is known to GoldenCheetah. It is essential that this is the case. So if the download file type isn't known (eg. the Strava streams API provides data in a special JSON format) then you will need to convert to a known file type in readFile and report that in the filename extension when returning the file list in readdir() (e.g. remotename might be aaa901722.json or aaa901722.fit to signal the file type you will respond with.

In addition, if the data is provided as compressed data then this can be signalled by setting downloadCompression as described above. In this instance, the remotename is parsed to identify the file type by removing the .zip or .gz from the filename. So compressed FIT files would have an extension .fit.gz or .fit.zip depending upon the compression type used.

If the request cannot be successfully initiated then the implementation should return false otherwise true indicates the request is being processed.

Asynchronous response

The call to readFile is asynchronous, it is expected that one the data has been read by your service you will call notifyReadComplete(QByteArray *data, QString name, QString message) with the data to inform the caller that the data has been received.

Converting and renaming the response

In some cloud services (notably Todays Plan) the downloaded data may be converted from the source format to another. In the case of Todays Plan downloaded files are updated to add 'RPE' metadata, and so the source file (perhaps a .FIT) is processed and tags added but returned as JSON. In this case the type conversion needs to be passed back to the caller.

To notify the caller you can change the name passed via notifyReadComplete. In the example above the name might change from abc1234.fit to abc1234.json.

Compressed content

The response is uncompressed by the caller if needed using CloudService::uncompressRide(). Where the file is assumed to be compressed if you set compressDownload as described above. However, is is possible to also indicate compressed content by appending .zip or .gz to the filename returned.

The uncompressRide method will handle this using the right compression routines and ride file reader. As an example, the data returned by the Sixcycle API may have .gz as a filename suffix, but the file type is prefixed to the filename in the URL it uses: https://s3.amazonaws.com/sixcycle/rawFiles/tcx_ba4319c2-a6dd-4503-9a50-0cb1f0f832ab.gz. To ensure this is processed correctly, all the Sixcycle::readFile method needs to do is rename the file taking the suffix and prefix to create a filename with a new suffix of .tcx.gz.

Threaded requests

In future we may initiate multiple concurrent calls to read files in parallel (this is not the case at present). In this instance the implementation will need to map network calls to the response data in order to collect the right response data to the correct request. This is already handled natively by the CloudService base implementation.

When initiating the QNetworkRequest the QNetworkReply can be placed into a QMap within implementation. In this way, every response can be mapped back. Here is a snippet of code to explain:

Initiating a request:

// put the file
QNetworkReply *reply = nam->get(request);

// remember
mapReply(reply,remotename);
buffers.insert(reply,data);

Mapping the responses to the original request:

QNetworkReply *reply = static_cast<QNetworkReply*>(QObject::sender());
buffers.value(reply)->append(reply->readAll());

Returning once read is complete:

QNetworkReply *reply = static_cast<QNetworkReply*>(QObject::sender());
notifyReadComplete(buffers.value(reply), replyName(reply), tr("Completed."));

For more detail on this approach please review the implementation of readFile, readyRead and readFileComplete in the TodaysPlan.cpp example.

bool writeFile(QByteArray &data, QString remotename, RideFile *ride)

The data passed has been converted and compressed using the fileType and uploadCompression settings you should setup in the constructor. The remotename is constructed from the local file name, but with the correct extensions applied to indicate the content being passed.

The call is expected to be asynchronous, although at present this is not multi-threaded, files are uploaded one at a time to avoid race conditions or capacity issues at the remote server. This may change in the future, so should be coded for (see multi-threading in readFile above). If the call fails it should return false to indicate this.

Once the file has been written, the writeComplete(QString name, QString message) signal should be emitted. This notifies the caller that the file has been uploaded. There is a utility function notifyWriteComplete(QString name,QString message) to emit this. The first parameter name should be the original remotename passed in the call, whilst the message should indicate success. The readers will look for "Completed." to indicate success, whilst any other response indicates an error (this is also a translated text, so should be returned as tr("Completed.".

bool upload(QWidget *parent, CloudService *store, RideItem *item)

We do not expect this function to be re-implemented for a cloud service. It provides the GUI and caller semantics for the writeFile API that can be called by the GUI. If you need to add your own GUI elements around the upload function (perhaps to prepate data / confirm with the user etc), then you may reimplement this function.

To date, we have not needed to reimplement this for a specific service, and may choose to enhance it in future releases (to add data that may be missing but required, such as RPE, date/time etc). The base implementation calls the CloudServiceUploadDialog to do the main processing.

virtual QList<CloudServiceEntry*> readdir(QString path, QStringList &errors, QDateTime &from, QDateTime&to)

The readdir method is used to query the cloud service for activities and should return a list of CloudServiceEntry objects that have been allocated via the support method CloudServiceEntry *newCloudServiceEntry().

Each entry in the list represents an activity that is stored at the location specified in path. For cloud services that don't have a storage model that includes paths then this can be ignored in your implementation. For services that do use a folder structure (e.g. Dropbox) this is used to traverse.

In the event that there are errors whilst querying the service you can return errors in the errors reference to a QStringList.

It is expected that the query will be limited to those activites between and including the date to and from. If the date range is not (or cannot) be honored the calling methods in the main dialogs will also check for the correct range (but this cannot always be guaranteed in the future).

The utility dialog CloudServiceFileDialog allows the user to browse the cloud service structure when selecting a folder in configuration. It makes repeated calls to readdir for this purpose. The dates to and from are always invalid (created via QDateTime()).

See below, the dialog browsing a Dropbox folder structure:

CloudServiceEntry *root() and QString *home()

These two methods return a CloudServiceEntry for traversing a store. root() should point to the base of the store, whilst home() should point to the pathname for files that will be stored for this athlete. Since these values are passed to readdir() or CloudServiceDialog you do not need to implement them unless you are implementing a filestore service. The default implementations will return NULL and "/" respectively.

bool createFolder(QString pathname)

Again, if you are implementing a file store you will need to reimplement this method for creating a remote folder. The pathname is a fully qualified path from the value provided by root. It is only called during configuration at present (by CloudServiceDialog). It should return false if the folder cannot be created, otherwise true to indicate success. There is no way to indicate why the create folder operation failed at this point.

QList listAthletes()

For services where the athlete needs to be selected the listAthletes method is called to get a list of the athletes available to select. This is then offered to the user when configuring the service. If an athlete is selected then your service will be notified by a call to selectAthlete (see below).

bool selectAthlete(CloudServiceAthlete athlete)

If you need to perform any further configuration setup when an athlete is selected during the service configuration you can perform it when this method is called. It is passed one of the CloudServiceAthlete objects returned by listAthletes. The method should return true to indicate the athlete was selected successfully, otherwise false.