gnix_rma - a-abraham/docs GitHub Wiki

RMA Requests support Reads and Writes.
GNI Supports writing to local and remote addresses with any alignment and length.
GNI requires reads to use four byte aligned remote address and length.
For reads smaller than one cacheline, aligned data is read into an intermediate buffer, then partially copied to the user buffer. This is referred to as an INDIRECT Transfer.
For larger unaligned reads, the interior, aligned portion of remote data is pulled directly into the user provided buffer. The four bytes at the head and tail of an unaligned read are pulled into an intermediate buffer, then partially copied into the user buffer. This is referred to as a CHAINED transfer.
Unaligned reads smaller than the RDMA threshold can perform these 3 distinct transactions (head, middle, tail) in a single GNI chained FMA operation (resulting in a single GNI CQE).
For unaligned reads larger than than the RDMA threshold, two GNI posts are used. One RDMA TX to transfer the bulk of the data, and then an FMA TX to transfer the head and/or tail data.

gnix_rma

Basic sanity checking. A few examples are listed below, but these may or may not change.
Find VC for target
Set up fab request
Determine if RDMA by checking if the length exceeds the rma rdma threshold.
Default work_fn is set to be gnix_rma_post_req
Determine if using Chained Transfer or INDIRECT Transfer
If RDMA set the work_fn to be gnix_rma_post_rdma_chain_req
If using ```RDMA` write or read, we need to auto-register the source buffer and get a memory descriptor.
FI_INJECT - ??? Different behavior for req->rma.loc_addr
Some completion stuff Inject interfaces always suppress completions. If SELECTIVE_COMPLETION is set, honor any setting. Otherwise, always deliver a completion.
return gnix_vc_queue_tx_req This attempts to initiate a TX request. If the TX queue is blocked (due to low resources or a FI_FENCE request), the request is scheduled to be sent later. gnix_vc_queue_tx_req will eventually call on the work_fn to process the request.