Batch Import and Export - LiquidAnalytics/ld-api-examples GitHub Wiki

Batch Job Status

These are the following status a batch job can be in:

Status Description
Initialized A new job has been received and is being accepted
Accepted The job has been accepted and is ready to run
Processing The job has begun processing
Completed The job has been completed
Rejected There was an error with the job parameters or file that caused the job to be rejected
Canceling A request to cancel the job has been received and the job will be canceled next time it is up for processing
Canceled The job has been canceled
Failed During processing there was an error with the job that caused the job to fail to complete

Batch Job Progress

Use this API to query a previously submitted Batch Job status:

POST /ls/api/batch/status

Post Parameters:

OAuth2 Access Token

Parameter Description
jobId The unique identifier for the job that status is being queried

Example


payload = {'jobId':jobId}

response = requests.post(server + '/ls/api/batch/status'.format(com), headers={'Authorization':'Bearer ' + accessToken}, data=payload)

Response

The response will be in json format and contain the following fields:

Parameter Description
submitTime The time the job was submitted
jobId A unique job identifier that can be used to query job status
notificationEmail Optional: An email address that will be updated with the jobs' status
jobName The name for the import job
contentType The file data type {'application/json','plain/text'}
itemType Item type as per LD schema that the job will be importing
recordsProcessed The total counts of records from the job that have been processed
runTime The actual runtime of the server in milliseconds
status The current job status
reason A message that details the current job status

Example:

{
  "submitTime": "2015-05-26T13:35:25+0000",
  "jobId": "e25a5d4b-f9a5-4794-9ac2-4e026c0df40d",
  "notificationEmail": "[email protected]",
  "jobName": "Attendee Import Job",
  "contentType": "application/json",
  "itemType": "Attendee",
  "recordsProcessed": "0",
  "runTime": "0",
  "status": "Accepted",
  "reason": "The file has been stored and the job is ready for processing"
}

Complete code example can be found here

Import

Use this API to import JSON or CVS files to Liquid Decisions

POST /ls/api/batch/import

Post Parameters:

OAuth2 Access Token

Parameter Description
jobName The name for the import job
itemType Item type as per LD schema that the job will be importing
file The file that contains the items to be imported
template Optional: The template to be used if the file is of CSV type
merge Optional: The value will determine if files are merge or overridden {True, False}. Default is override.
contentType The file data type {'application/json','plain/text'}
notificationEmail Optional: An email address that will be updated with the jobs' status

Example:


files = {'file': ('data.json', open('data.json', 'rb'), 'application/json', {'Expires': '0'})}

parameters = {
              'notificationEmail': '[email protected]', 
              'jobName': 'Attendee Import Job',
              'contentType':'application/json',
              'itemType':'Attendee'
              'merge':False'}

responce = requests.post(
                     server + '/ls/api/batch/import'.format(community), 
                     headers={'Authorization':'Bearer ' + accessToken}, 
                     files=files, 
                     data=parameters)

Job Import Request Response

After the file import is completely uploaded to the LDS server a job status response will be returned with information about the job and its job Id that can be used to further query the job's status. The response will be in json format and follow the [Batch Status Response](Batch Jobs#response).

Import Job Notifications

After the Batch Import job has been accepted it will be run in the LDS server. When the job is completed or any error that prevents the job to finish processing occurs an email will be sent to the notificationEmail (if one is provided) to notify the user of the completion/failure of the job.

Export

Use this API to export data from Liquid Decisions into CSV or JSON format.

POST /ls/api/batch/export

Post Parameters:

OAuth2 Access Token

Parameter Description
jobName The name for the import job
notificationEmail Optional: An email address that will be updated with the jobs' status
itemType Item type as per LD schema that the job will be exporting
contentType The format of the file that the data will be exported to {'application/json','text/plain'}
rowHeader By setting to true the CSV export file will contain a header row with the name of the exported fields
template Optional: The velocity template to be used to convert from the Item JSON to CSV format
sqlQuery Optional*: A SQL query that's result set yields the data to be exported
filters** Optional*: Set of filters will be applied and only Items matching the filters will be exported
Note:
*If no filters and no sqlQuery are provided all of the records for the itemType will be exported
**If both a sqlQuery and filters are provided the filters will be disregarded and the sql query will be prioritized

Example:


filters = { "userId":"[email protected]",
            "processStatus":"/Workflow/processStatus[Editing]"}

parameters = {
                'notificationEmail': options.notificationEmail, 
                'jobName': jobName,
                'itemType': itemType,
                'sqlQuery': sqlQuery,
                'contentType': contentType,
                'filters': str(filters),
                'template': template}

response = requests.post(
                          server + '/ls/api/batch/export'.format(community), 
                          headers={'Authorization':'Bearer ' + accessToken}, 
                          data=parameters)


Complete code example can be found here

Job Export Request Response

After the export request is processed a job status response will be returned with information about the job and its job Id that can be used to further query the job's status. The response will be in json format and follow the [Batch Status Response](Batch Jobs#response).

Export Job Notifications

After the Batch Export job has been accepted it will be run in the LDS server. When the job is completed or any error that prevents the job to finish processing occurs an email will be sent to the notificationEmail (if one is provided) to notify the user of the completion/failure of the job. The response will contain one or more links to download the results of the export job. The links will expire in a week after the job has been completed.