ReqMgr2 MicroService APIs - dmwm/WMCore GitHub Wiki

This wiki will list the RESTP APIs for all WMCore Microservices. They return data in json, so it requires the header "Accept: application/json" to be set.

MSTransferor APIs
MSMonitoring APIs
MSOutput APIs
MSUnmerged APIs
MSPileup APIs
MSPileup tasks run as independent service (daemon) which provides data placement to MSPileup workflows. This daemon does not have external APIs exposed to end-users.

MSTransferor REST APIs

This microservice is responsible for the workflow input data placement. It has CouchDB as a backend database, where it persists the workflow transfers summary and keeps updating the transfers completion in specified intervals. Once all the data placement requests have been made, the workflow is moved to the next status staging.

Fetch the status of the service

This returns very basic information of the service, like its status and version. One can also provide the detail=true query string in order to retrieve a few more details of the service. This extra information contains data regarding its last execution cycle.

GET /ms-transferor/data/status HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-transferor/data/status?detail=true

Fetch information for a given request

This API can be used to retrieve information for a given workflow. The information returned corresponds to the transfer document created when handling the input data to be transferred. It includes the dataset names, transfer request IDs, transfer completion, etc. Note that those documents are only available while the workflow is active, basically between staging and running-* statuses.

GET /ms-transferor/data/info?request=WORKFLOW_NAME HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-transferor/data/info?request=pdmvserv_task_B2G-RunIISummer16NanoAODv6-02873__v1_T_200331_212615_94

MSMonitor REST APIs

This microservice is responsible for monitoring the data transfer requests created by MSTransferor. It keeps updating the local couch document with the transfers completion in a periodic cycle (X hours), once transfers have completed, the workflow is moved to the next status staged).

Fetch the status of the service

GET /ms-monitor/data/status HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-monitor/data/status?detail=true

Fetch information for a given request

This API is equivalent to the ms-transferor info API. Thus, it can be used to retrieve information for a given workflow. The information returned corresponds to the transfer document created when handling the input data to be transferred. It includes the dataset names, transfer request IDs, transfer completion, etc. Note that those documents are only available while the workflow is active, basically between staging and running-* statuses.

GET /ms-monitor/data/info?request=WORKFLOW_NAME HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-monitor/data/info?request=pdmvserv_task_B2G-RunIISummer16NanoAODv6-02873__v1_T_200331_212615_94

MSOutput REST APIs

This microservice is responsible for the workflow output data placement. It has MongoDB as a backend database, where it persists a workflow record for the output data placement. This microservice fetches workflows in the status closed-out or announced.

Fetch the status of the service

GET /ms-output/data/status HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-output/data/status?detail=true

Fetch information for a given request

This API can be used to retrieve information for a given workflow. The information returned contains information regarding the output data produced, amount of Disk copies made, destinations, replication rule IDs, others. Once all the output data transfers have been successfully done, the workflow record is marked as TransferStatus=done (instead of pending) status. Note that such workflow records should only be created for workflows that went through either the closed-out or the announced status.

GET /ms-output/data/info?request=WORKFLOW_NAME HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-output/data/info?request=pdmvserv_RVCMSSW_12_0_0_pre5BuMixing_14_210730_160440_5107

MSUnmerged REST APIs

This microservice is responsible for the unmerged area cleanup for the sites storage elements.

This service is executed with 3 different classes - corresponding to 3 different services/replicas in the Kubernetes realm - in order to contact each of those replicas, we have initially implemented a specific query string, where:

rse_type=t1: will fetch information from the MSUnmerged replica responsible for all the T1 storage elements;
rse_type=t2t3us: will fetch information from the MSUnmerged replica responsible for all the T2 and T3 storage elements, in US (United States);
rse_type=t2t3: will fetch information from the MSUnmerged replica responsible for all the remaining T2 and T3 storage elements.

Currently the rse_type option is used only int the status interface, as shown bellow.

Fetch the status of the service

GET /ms-unmerged/data/status?rse_type=t2t3 HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-unmerged/data/status?&rse_type=t2t3

Fetch information for a given RSE

This API can be used to retrieve information for a given storage element (RSE). For directory detailed information, one needs to provide detail=true as a query string, in addition to the RSE name. At its current state.

GET /ms-unmerged/data/info?&rse=RSE_NAME HTTP/1.1
Accept: application/json
Host: cmsweb.cern.ch

An example of a curl command to read the agent configuration is:

curl --cert $X509_USER_CERT --key $X509_USER_KEY -X GET https://cmsweb.cern.ch/ms-unmerged/data/info?rse=T2_IT_Rome&detail=true

MSPileup REST APIs

This micro-service is responsible for handling Premix pileup data. It has the following REST APIs:

HTTP GET APIs

These set of APIs are used to fetch pileup data from the micro-service. The main endpoint is /ms-pileip/data/access, e.g.

curl -v "http://localhost:8249/ms-pileup/data/access?pileupName=pileup"
curl -v "http://localhost:8249/ms-pileup/data/access?campaign=Campaign"

and, we can use filters to extract certain fields from the doc, e.g.

curl -v "http://localhost:8249/ms-pileup/data/access?pileupName=pileup&filters=["campaigns", "pileupType"]"

HTTP POST APIs

The POST APIs have several roles:

create document in MSPileup service

curl -v -X POST -H "Content-type: application/json" -d '{"bla":1}' \
       http://localhost:8249/ms-pileup/data/access

fetch data via provided JSON query (spec)

curl -v -X POST -H "Content-type: application/json" \
       -d '{"query":{"pileupName":"bla"}}' \
       http://localhost:8249/ms-pileup/data/access

fetch data by using query and filters

curl -v -X POST -H "Content-type: application/json" \
       -d '{"query":{"pileupName": "bla"}, "filters":["campaigns"]}' \
        http://localhost:8249/ms-pileup/data/access

HTTP PUT APIs

The PUT HTTP API are used to update specific data in MSPileup service, e.g.

curl -v -X PUT -H "Content-type: application/json" -d '{"bla":1}' \
        http://localhost:8249/ms-pileup/data/access

HTTP DELETE APIs

The DELETE APIs are used to delete specific data in MSPileup service, e.g.

curl -v -X DELETE -H "Content-type: application/json" -d '{"bla":1}' \
        http://localhost:8249/ms-pileup/data/access
curl -v -X DELETE -H "Content-type: application/json" -d '{"pileupName":"pileup"}' \
        http://localhost:8249/ms-pileup/data/access