CUDA RNDV protocols - openucx/ucx GitHub Wiki
The following are default rndv protocols with default UCX_RNDV_SCHEME=auto
Inter-node:
GPUDIrectRDMA enabled
-
Sender stages cuda fragments and PUTs to destination Device buffer D-D
-
Sender stages cuda fragments and PUTs to destination Host buffer D-H
-
put_zcopy protocol H-D
-
get-zcopy protocol H-H
GPUDIrectRDMA disabled
-
Receiver requests fragments from Sender (frag rtr) Sender stages fragments and PUTs to the receiver staging buffer and notifies (frag ATP) Receiver PUTs fragment to Device buffer. D-D
-
Sender pipelines cuda fragments and PUTs to destination Host buffer D-H
-
Receiver GETs host fragments and PUTs to destination Device buffer H-D
-
get-zcopy protocol H-H
Intra-node
CUDA-IPC
-
put_zcopy protocol D-D
-
Sender stages cuda fragments and PUTs to destination Host buffer D-H
-
put_zcopy protocol H-D
-
get-zcopy protocol H-H
No CUDA_IPC
-
Sender stages cuda fragments and PUTs(IB)to destination Device buffer D-D
-
Sender pipelines cuda fragments and PUTs to destination Host buffer D-H
-
put_zcopy protocol (IB). H-D
-
get-zcopy protocol H-H