A review of the code - JonHowes/IDOLOnDemandWrapper GitHub Wiki

This is a C# .NET project which uses the HP IDOL OnDemand job API to batch together several actions and associated parameters.

Full details of making an IDOL OnDemand asynchronous job request can be found here.

RestSharp is used to simplify the batch request process. Json.NET is used for building request structures and processing responses.

The underlying basis for the wrapper is that a set of requests are batched together to perform a task with a particular outcome. The particular task addressed in the code involves:

• input data as text, located in files and at a URL,
• the IDOL OnDemand sentiment and entity extraction APIs to process this data
• taking the results from all these analyses to generate an outcome for the task

In reality, this task might run in its own thread. In the demonstration, the console application Program.Main() contains the task.

The task processing steps are:

• prepare the batch
• make the API batch request
• obtain the batch response
• process results in task context

Batch preparation involves creating a list of C# request classes that can be converted into json for the batch request. First, create the actual batch request wrapper and set the IDOL OnDemand API key:

IODJobBatchRequest batchReq = new IODJobBatchRequest();
batchReq.ApiKey = IODApiKey;

Files to be processed are individually added to the batch request. Each file has an associated key which is used as a reference from requests:

batchReq.AddFile("filekey01", filePath1);

A task has one or more requests which are individually added to the batch request wrapper. Handling the response here, inline with the request, greatly simplifies response processing as there is no need to lodge additional information about how to process the response with the request class.

SentimentRequest sr2 = new SentimentRequest(IODSource.file, "filkeye01");
batchReq.AddRequest(sr2);
sr2.OnResponse = (status, response) =>
{
  if (status == IODStatus.finished)
  {
    Console.WriteLine(string.Format("Request: {0}, Aggregate response: {1}", filePath1, response.aggregate.score);
  }
};

Each request class is specific to a particular IDOL OnDemand API call and encapsulates generation of the outgoing json action and parsing of the json response.

The outgoing json for an action consists of a name, version and a set of action specific parameters. These are grouped together as an array for the “job” field:

name="job"
value='{ "actions": [ 
  { "name":"analyzesentiment", "version":"v1", "params":{ "file":"file01", "language":"eng" } }, 
  { "name":"extractentity",    "version":"v1", "params":{ "file":"file01", "entity_type":"people_eng" } }
] }

This is generated in IODJobBatchRequest using the following technique:

public class IODJobBatch
{
  public List<IODJob> actions = new List<IODJob>();
}

IODJobBatch batch = new IODJobBatch();
batch.actions.Add( new IODJob() { 
    name = "analyzesentiment", 
    version = "v1", 
    @params = new { file = "file01", language = "eng" } 
  } );
batch.actions.Add( new IODJob() { 
    name = " extractentity ", 
    version = "v1", 
    @params = new { file = "file01", entity_type = "people_eng" } 
  } );

JsonConvert.SerializeObject(batch);

After files and requests have been added to IODJobBatchRequest, the API call is started with:

batchReq.MakeRequest();

This call will not return until the results have been processed or an error has occurred. The reasoning behind the synchronous nature of the request is that all of the results must be obtained before the outcome of the task can be determined, ie, task processing is an all or nothing operation. Each task would most likely be kicked off in a separate thread.

Within MakeRequest(), the parameters for the initial API job request are assembled:

RestClient client = new RestClient(@"https://api.idolondemand.com");
RestRequest req = new RestRequest(@"1/job", Method.POST);
req.AlwaysMultipartFormData = true;
req.AddParameter("apikey", ApiKey);
req.AddParameter("job", JsonConvert.SerializeObject(_iodJobBatch));
foreach (var v in _dictionaryOfFiles) 
{ 
  req.AddFile(v.FileKey, v.FilePath); 
}

An asynchronous request is made to the job API and the response is awaited in a tight loop. The presence of a response (or a timeout) is used to signal end of wait:

DateTime timeoutTime = DateTime.Now + TimeoutSpan;
string resp = null;
client.ExecuteAsync(req, response => { resp = response.Content; });
while (string.IsNullOrEmpty(resp) && DateTime.Now < timeoutTime)
{
  System.Threading.Thread.Sleep(1000);
}

The response is checked for errors and eventually parsed to give a job id for the batch which is then used to make a synchronous request for the results.

JObject parsedRes = JObject.Parse(resp);
string jobId = (string)resParsed["jobID"];
RestRequest req = new RestRequest(@"1/job/result/{jobId}", Method.POST);
req.AddUrlSegment("jobId", jobId);
req.AddParameter("apikey", ApiKey);
RestResponse rr = (RestResponse)client.Execute(req);

The response is converted to a JobBatchResponse which contains an array of action results. We must assume that the results are in the same order as the initial request and that there is a result for every request. There is some checking for action/result mismatch, but the code is essentially:

JobBatchResponse rs = JsonConvert.DeserializeObject<JobBatchResponse>(rr.Content);
for (int i = 0; i < rs.actions.Count; i++)
{
  Action a = rs.actions[i];
  IODRequest r = _batch.Requests[i];
  r.SetResult(a.status, a.result.ToString());
}

IODRequest.SetResult causes the results to be parsed using the correct C# response class and makes the OnResponse callback to the initial task code so that the results can be processed in context.

Individual response classes are created using sample json from the IDOL OnDemand API documentation pages and the utility here. The demonstrator only contains classes for entity extraction and sentiment analysis. It should be straight-forward to add handling of other APIs as necessary by following the same pattern for request and response classes.