Unified Porting - dmwm/WMCore GitHub Wiki

NOTE: most of this content has been adapted and migrated to https://github.com/dmwm/WMCore/wiki/ReqMgr2-MicroService-Transferor , reflecting the latest discussions as of 2019. So we will probably deprecated this wiki in the near future.

Transferer.py porting

  1. Provide API for the information which can determine datasets to transfer:

    • Create "Hold" states before "Available" state for Global Workqueue elements. (don't replicate to local workqueue when it is "Hold" state
    • When request is assign-approved states, populates GQ elements with "Hold" state.
    • Provide the API to get jobs per dataset, site white list, cpu requirement, (Alan, Jean-Roch - which information need to determine the dataset to transfer? - do we have to consider block level transfer?)

  2. Trigger the transfer and keep track of the status.

    • MicroService (daemon) use GQ api and Unified config to determine which dataset need to be transferred periodically.
    • MicroService makes PhEDEx subscription call for the datasets selected.
    • For the request which contains "not selected" datasets, update the request status in request manager to "assigned" (Provide the api to change request status and WQE status at the same time - change WQE status first).
    • MicroService keeps track of the subscription made until data is finished. (Memory queue, local database)
    • Update the request status in request manager to "assigned" using the same API above.

  3. Dataset deletion

    • DDM service will query to reqmgr to check whether given dataset is ready to be deleted.
    • DDM will handle internal locks from other sources as well as reqmgr query result to handle the deletion