Request Manager (ReqMgr2) REST APIs - dmwm/WMCore GitHub Wiki

This wiki provides some instructions and documentation on the Request Manager 2 RESTful APIs.

A general requirement of the WMCore REST framework is that clients must provide a Accept HTTP header in their request. Hence, if the end user wants to retrieve data in a JSON format, it needs to provide the following HTTP header request: Accept: application/json.

How to request compressed data through the WMCore REST APIs

Starting in the WMCore release HG2211 - from November, 2022 - using WMCore version 2.1.4, the capability of gzip compressed response has been added to the WMCore REST framework. End users are invited to request compressed data, especially for heavy APIs transferring (many) megabytes of data, including most of the wmstatsserver RESTful APIs.

When the user is creating their HTTP request, an extra HTTP header has to be provided to communicate to the WMCore server that gzip'ed content is accepted by the client. The user has to provide this key/value parameter in their HTTP request: Accept-Encoding: gzip.

This does not necessarily mean that the server will, so the user must check the HTTP response headers to decide how to read the response body, which might or not be compressed. In case the server has responded with compressed data, the following HTTP response header will be sent back to the client Content-Encoding: gzip, flagging that that response body is in a binary/compressed format.

In order to decompress the body data, the client can use the gzip third-party python library and decompress the data as:

gzip.decompress(body)

If HTTP requests are made with the curl Unix tool, the same header has to be provided and the output data can be redirected to a file, example:

curl -L -k --cert $X509_USER_CERT --key $X509_USER_KEY --cacert $X509_USER_CERT https://cmsweb.cern.ch/reqmgr2/data/status -vvv -H "Accept: application/json" -H "Accept-Encoding: gzip" > out.data

now to see the content of out.data, one can use the zcat tool, example:

zcat out.data

To summarize the use of gzip, the client needs to provide the correct Accept-Encoding HTTP request header and when parsing the HTTP response object, a check for the HTTP response header Content-Encoding is required to decide how to deal with that object.

To create and update request (data need to passed as request body part)

To create the request - need POST request

POST /reqmgr2/data/request HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

json format of data example: ReReco example

To clone the request - need POST request

POST /reqmgr2/data/request/clone/[request_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

i.e.
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X POST -H "Content-type: application/json"  -H "Accept: application/json"  -d '{"Step1":{"RequestNumEvents": 4000}}' https://cmsweb.cern.ch/reqmgr2/data/request/clone/wmagent_dmason_NERSCTEST_171102_171002_8020

update ([] here means user input not the list)

PUT /reqmgr2/data/request/[request_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

To update request priority
i.e PUT /reqmgr2/data/request/[request_name]
Content-type: application/json
Host: cmsweb.cern.ch
Body: {"RequestPriority": 112000}
 
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X PUT -H "Content-type: application/json"  -d '{"RequestPriority": 112000}' https://cmsweb.cern.ch/reqmgr2/data/request/pdmvserv_task_PPD-RunIISpring18wmLHEGS-00001__v1_T_180220_111353_7996

json format of data example: ReReco example

4. To assign request with other properties
i.e PUT /reqmgr2/data/request/[request_name]
Content-type: application/json
Host: cmsweb.cern.ch
Body: {"RequestStatus": "assigned", "SiteWhitelist": ["T1_US_FNAL"], ...}}
 
curl --cert $X509_USER_CERT --key $X509_USER_KEY -X PUT -H "Content-type: application/json"  -d "@assingment.json" https://cmsweb.cern.ch/reqmgr2/data/request/pdmvserv_task_PPD-RunIISpring18wmLHEGS-00001__v1_T_180220_111353_7996

To retrieve the data

GET /reqmgr2/data/request?name=[request_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

GET /reqmgr2/data/request?status=[request_status] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

* you can't retrieve *-archived status with this call. There is special status **ACTIVE** which retrieves all non archived status

GET /reqmgr2/data/request?status=[request_status]&request_type=[request_type] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

* you can't just use request_type only it needs to combined with status

GET /reqmgr2/data/request?status=[request_status]&team=[team] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

* you can't just use team only it needs to combined with status

** following calls can be combined with other key

GET /reqmgr2/data/request?prep_id=[prep_id] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

GET /reqmgr2/data/request?inputdataset=[inputdataset] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

GET /reqmgr2/data/request?outputdataset=[outputdataset] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

GET /reqmgr2/data/request?campaign=[campaign] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

GET /reqmgr2/data/request?mc_pileup=[mc_pileup] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

GET /reqmgr2/data/request?data_pileup=[data_pileup] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

* You can also filter out output by adding mask following will return only OutputDatasets and InputDatasets

GET /reqmgr2/data/request?data_pileup=[data_pileup]&mask=OutputDatasets&mask=InputDataset HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

return format

{"result": [ {requestname: {key: value}, ....}, 
             {requestname: {key: value}, ....}, ...]}

example)

https://cmsweb.cern.ch/reqmgr2/data/request?name=pdmvserv_TSG-RunIISpring15MiniAODv2-00006_00082_v0__151012_002749_6757

{"result": [
 {"pdmvserv_TSG-RunIISpring15MiniAODv2-00006_00082_v0__151012_002749_6757": 
  {"InputDataset": "/QCD_Pt-170to300_EMEnriched_TuneCUETP8M1_13TeV_pythia8/RunIISpring15DR74-Asympt25ns_MCRUN2_74_V9-v1/AODSIM", 
   "KeepStepOneOutput": true, 
   "Group": "ppd", 
   "CustodialSites": [], 
   "TimePerEvent": 1.5, 
   "Requestor": "pdmvserv", 
   "ProcessingString": "74X_mcRun2_asymptotic_v2", 
   "ScramArch": "slc6_amd64_gcc491", 
   "SizePerEvent": 60, 
   "Memory": 1980, 
   "PrepID": "TSG-RunIISpring15MiniAODv2-00006", 
   "AutoApproveSubscriptionSites": ["T1_US_FNAL_Disk"], 
   "CustodialSubType": "Replica", 
   "RequestType": "ReDigi", 
   "OpenRunningTimeout": 43200, 
   "InputDatasetTypes": {}, 
   "OutputDatasets": ["/QCD_Pt-170to300_EMEnriched_TuneCUETP8M1_13TeV_pythia8/RunIISpring15MiniAODv2-74X_mcRun2_asymptotic_v2-v1/MINIAODSIM"], 
   "TotalInputEvents": 5735584, 
   "RequestDate": [2015, 10, 11, 22, 27, 49], 
   "AcquisitionEra": "RunIISpring15MiniAODv2", 
   "PrimaryDataset": "QCD_Pt-170to300_EMEnriched_TuneCUETP8M1_13TeV_pythia8", 
   "RequestPriority": 85000, 
   "SiteWhitelist": ["T1_IT_CNAF", "T1_FR_CCIN2P3", "T2_CH_CSCS", "T1_US_FNAL", "T1_UK_RAL"],      
   "SubscriptionPriority": "Low", 
   "ProcessingVersion": "1", 
   "Teams": ["production"], 
   "TotalEstimatedJobs": 308, 
   "RequestTransition": [{"Status": "new", "DN": null, "UpdateTime": 1444602475},
                         {"Status": "assignment-approved", "DN": null, "UpdateTime": 1444602478},       
                         {"Status": "assigned", "DN": null, "UpdateTime": 1444626489}, 
                         {"Status": "acquired", "DN": null, "UpdateTime": 1444628716}, 
                         {"Status": "running-open", "DN": null, "UpdateTime": 1444634598}],      i     
   "RequestName": "pdmvserv_TSG-RunIISpring15MiniAODv2-00006_00082_v0__151012_002749_6757", 
   ...}
]}

REST APIs for ReqMgr2 Auxiliary

retrieve an agent configuration document from central CouchDB

GET /reqmgr2/data/wmagentconfig/[agent_host_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://alancc7-cloud1.cern.ch/reqmgr2/data/wmagentconfig/vocms0261.cern.ch

create a new agent configuration document in central CouchDB

The complete content needs to be provided in the request body

POST /reqmgr2/data/wmagentconfig/[agent_host_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

in request body, something like {'UserDrainMode':True, 'key2': 'value2', ...}

If the document already exists, the call will return a 'Document conflict' error and no changes will be made.

Example creating a new agent configuration:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X POST -H "Content-type: application/json" -d '{"ALAN":true, "MALTA":[1,2,3]}' https://alancc7-cloud1.cern.ch/reqmgr2/data/wmagentconfig/vocms0222.cern.ch

update an agent configuration document in central CouchDB

Following the good practices of REST implementation, the whole content needs to be provided in the request body. This PUT request replaces the old document by a new one with the new content.

PUT /reqmgr2/data/wmagentconfig/[agent_host_name] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

in request body, something like {'UserDrainMode':True, 'key2': 'value2', ...}

For these auxiliary specific APIs, if the resource does not exist yet, a new document gets created.

Example for an agent document update:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X PUT -H "Content-type: application/json" -d '{"UserDrainMode":true, "ALAN":"test"}' https://alancc7-cloud1.cern.ch/reqmgr2/data/wmagentconfig/vocms0222.cern.ch

retrieve a campaign configuration document

GET /reqmgr2/data/campaignconfig/[CAMPAIGN_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

create a new campaign configuration document

The complete content needs to be provided in the request body

POST /reqmgr2/data/campaignconfig/[CAMPAIGN_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

in request body, something like {'CampaignName':'test', 'datasets': ['blah'], ...}

If the document already exists, the call will return a 'Document conflict' error and no changes will be made.

update a campaign configuration document in central CouchDB

Note that the whole content needs to be provided in the request body. The PUT request replaces the old document by a new one with the new content.

PUT /reqmgr2/data/campaignconfig/[CAMPAIGN_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

in request body, something like {'UserDrainMode':True, 'key2': 'value2', ...}

For these auxiliary specific APIs, if the resource does not exist yet, a new document gets created.

retrieve the unified configuration document

GET /reqmgr2/data/unifiedconfig/[UNIFIED_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

create a unified configuration document

The complete content needs to be provided in the request body

POST /reqmgr2/data/unifiedconfig/[UNIFIED_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

in request body, something like {'goodsites':'test', 'key1': 'value1', ...}

If the document already exists, the call will return a 'Document conflict' error and no changes will be made.

update a unified configuration document

Note that the whole content needs to be provided in the request body. The PUT request replaces the old document by a new one with the new content.

PUT /reqmgr2/data/unifiedconfig/[UNIFIED_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

in request body, something like {'UserDrainMode':True, 'key2': 'value2', ...}

For these auxiliary specific APIs, if the resource does not exist yet, a new document gets created.

retrieve a workflow transfer document

GET /reqmgr2/data/transferinfo/[TRANSFER_DOC_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

If one wants to fetch all transfer documents in one single call, then the document name has to be ALL_DOCS, e.g.:

GET /reqmgr2/data/transferinfo/ALL_DOCS HTTP/1.1

create a workflow transfer document

The complete content needs to be provided in the request body

POST /reqmgr2/data/transferinfo/[TRANSFER_DOC_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

in request body, something like {'workflowName':'test', ...}

If the document already exists, the call will return a 'Document conflict' error and no changes will be made.

update a workflow transfer document

Note that the whole content needs to be provided in the request body. The PUT request replaces the old document by a new one with the new content.

PUT /reqmgr2/data/transferinfo/[TRANSFER_DOC_NAME] HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

in request body, something like {'workflowName':'test', 'key2': 'value2', ...}

For these auxiliary specific APIs, if the resource does not exist yet, a new document gets created.

Retrieving a list of locked parent datasets

This API returns a list of parent datasets that are in use by workflows with "IncludeParents": True and the following statuses:

['assignment-approved', 'assigned', 'staging', 'staged', 'acquired', 'failed',
 'running-open', 'running-closed', 'force-complete', 'completed', 'closed-out']

A list of InputDatasets for these workflows is obtained via a CouchDB view and DBS is queried via DBSReader.listDatasetParents() to create the list of parent datasets.

The REST API is parentlocks

GET /reqmgr2/data/parentlocks HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

Example output

{"result": [
 {
  "ConfigType": "PARENT_LOCKS", 
  "parentlocks": [
    "/Cosmics/Commissioning2015-v1/RAW", 
    "/HLTPhysics/Run2017B-v1/RAW"
  ]
}]}