Batch Import and Export - LiquidAnalytics/ld-api-examples GitHub Wiki
Batch Job Status
These are the following status a batch job can be in:
| Status | Description |
|---|---|
| Initialized | A new job has been received and is being accepted |
| Accepted | The job has been accepted and is ready to run |
| Processing | The job has begun processing |
| Completed | The job has been completed |
| Rejected | There was an error with the job parameters or file that caused the job to be rejected |
| Canceling | A request to cancel the job has been received and the job will be canceled next time it is up for processing |
| Canceled | The job has been canceled |
| Failed | During processing there was an error with the job that caused the job to fail to complete |
Batch Job Progress
Use this API to query a previously submitted Batch Job status:
POST /ls/api/batch/status
Post Parameters:
OAuth2 Access Token
| Parameter | Description |
|---|---|
| jobId | The unique identifier for the job that status is being queried |
Example
payload = {'jobId':jobId}
response = requests.post(server + '/ls/api/batch/status'.format(com), headers={'Authorization':'Bearer ' + accessToken}, data=payload)
Response
The response will be in json format and contain the following fields:
| Parameter | Description |
|---|---|
| submitTime | The time the job was submitted |
| jobId | A unique job identifier that can be used to query job status |
| notificationEmail | Optional: An email address that will be updated with the jobs' status |
| jobName | The name for the import job |
| contentType | The file data type {'application/json','plain/text'} |
| itemType | Item type as per LD schema that the job will be importing |
| recordsProcessed | The total counts of records from the job that have been processed |
| runTime | The actual runtime of the server in milliseconds |
| status | The current job status |
| reason | A message that details the current job status |
Example:
{
"submitTime": "2015-05-26T13:35:25+0000",
"jobId": "e25a5d4b-f9a5-4794-9ac2-4e026c0df40d",
"notificationEmail": "[email protected]",
"jobName": "Attendee Import Job",
"contentType": "application/json",
"itemType": "Attendee",
"recordsProcessed": "0",
"runTime": "0",
"status": "Accepted",
"reason": "The file has been stored and the job is ready for processing"
}
Complete code example can be found here
Import
Use this API to import JSON or CVS files to Liquid Decisions
POST /ls/api/batch/import
Post Parameters:
OAuth2 Access Token
| Parameter | Description |
|---|---|
| jobName | The name for the import job |
| itemType | Item type as per LD schema that the job will be importing |
| file | The file that contains the items to be imported |
| template | Optional: The template to be used if the file is of CSV type |
| merge | Optional: The value will determine if files are merge or overridden {True, False}. Default is override. |
| contentType | The file data type {'application/json','plain/text'} |
| notificationEmail | Optional: An email address that will be updated with the jobs' status |
Example:
files = {'file': ('data.json', open('data.json', 'rb'), 'application/json', {'Expires': '0'})}
parameters = {
'notificationEmail': '[email protected]',
'jobName': 'Attendee Import Job',
'contentType':'application/json',
'itemType':'Attendee'
'merge':False'}
responce = requests.post(
server + '/ls/api/batch/import'.format(community),
headers={'Authorization':'Bearer ' + accessToken},
files=files,
data=parameters)
Job Import Request Response
After the file import is completely uploaded to the LDS server a job status response will be returned with information about the job and its job Id that can be used to further query the job's status. The response will be in json format and follow the [Batch Status Response](Batch Jobs#response).
Import Job Notifications
After the Batch Import job has been accepted it will be run in the LDS server. When the job is completed or any error that prevents the job to finish processing occurs an email will be sent to the notificationEmail (if one is provided) to notify the user of the completion/failure of the job.
Export
Use this API to export data from Liquid Decisions into CSV or JSON format.
POST /ls/api/batch/export
Post Parameters:
OAuth2 Access Token
| Parameter | Description |
|---|---|
| jobName | The name for the import job |
| notificationEmail | Optional: An email address that will be updated with the jobs' status |
| itemType | Item type as per LD schema that the job will be exporting |
| contentType | The format of the file that the data will be exported to {'application/json','text/plain'} |
| rowHeader | By setting to true the CSV export file will contain a header row with the name of the exported fields |
| template | Optional: The velocity template to be used to convert from the Item JSON to CSV format |
| sqlQuery | Optional*: A SQL query that's result set yields the data to be exported |
| filters** | Optional*: Set of filters will be applied and only Items matching the filters will be exported |
| Note: |
|---|
| *If no filters and no sqlQuery are provided all of the records for the itemType will be exported |
| **If both a sqlQuery and filters are provided the filters will be disregarded and the sql query will be prioritized |
Example:
filters = { "userId":"[email protected]",
"processStatus":"/Workflow/processStatus[Editing]"}
parameters = {
'notificationEmail': options.notificationEmail,
'jobName': jobName,
'itemType': itemType,
'sqlQuery': sqlQuery,
'contentType': contentType,
'filters': str(filters),
'template': template}
response = requests.post(
server + '/ls/api/batch/export'.format(community),
headers={'Authorization':'Bearer ' + accessToken},
data=parameters)
Complete code example can be found here
Job Export Request Response
After the export request is processed a job status response will be returned with information about the job and its job Id that can be used to further query the job's status. The response will be in json format and follow the [Batch Status Response](Batch Jobs#response).
Export Job Notifications
After the Batch Export job has been accepted it will be run in the LDS server. When the job is completed or any error that prevents the job to finish processing occurs an email will be sent to the notificationEmail (if one is provided) to notify the user of the completion/failure of the job. The response will contain one or more links to download the results of the export job. The links will expire in a week after the job has been completed.