Rendezvous Protocol threshold for multilane mode - openucx/ucx GitHub Wiki

Assumptions:

1. Sender & receiver are configured by same manner & using same HW (latency, registration costs, etc. on both sides are equal).
2. Eager & RNDV protocols use different lanes
3. Cost of transfer single message is latency + overhead

Definitions:

• ecost/rcost – eager/rndv registration cost
• egro/rgro – eager/rndv registration growth
• ebw/rbw – eager/rndv bandwidth
• eover/rover – eager/rndv overhead
• elat/rlat – eager/rndv latency
• etime/rtime – eager/rndv delivery time
• d - differentiate factor (bit less 1)

bw=∑_n⁰lane_i→bandwidth

cost=∑_n⁰md_i→registration cost

gro=∑_n⁰md_i→registration grow

Eager protocol (zcopy):

1. Registration (eager reg cost + size * eager reg grow)

2. Transfer (size/bw + eager overhead + eager latency)

   EAGER:etime = ecost + size • egro + ^size⁄_ebw+eover

RNDV AM protocol (zcopy):

1. Registration on sender size (rndv reg cost + size * rndv reg grow)

2. RTS (rndv latency + rndv overhead)

3. RTR (rndv latency + rndv overhead)

4. Transfer (size/bw + rndv latency + rndv overhead + rndv latency)

   AM:rtime=d(rcost + size • rgro + 4rlat + 3rover + ^size⁄_rbw)

RNDV RMA protocol (get):

1. Registration on sender size (rndv reg cost + size * rndv reg grow)

2. RTS (rndv latency + rndv overhead)

3. Registration on receiver size (rndv reg cost + size * rndv reg grow)

4. Transfer (size/bw + rndv latency + rndv overhead + rndv latency)

5. ACK (rndv latency + rndv overhead)

   RMA:rtime=d(2(rcost + size • rgro) + 4rlat + 3rover + ^size⁄_rbw)

To unify RNDV formulas let’s add one more argument: rrc – receive registration cost, which is 0 for AM RNDV and 1 for RMA RNDV, then RNDV time may be calculated as:

RNDV:rtime=d((1+rrc)(rcost + size • rgro) + 4rlat + 3rover + ^size⁄_rbw)

Transformations:

EAGER:etime = ecost + size • egro + ^size⁄_ebw+ eover = size(egro+¹⁄_ebw)+ ecost + eover

RNDV:rtime=d((1 + rrc)(rcost + size • rgro) + 4rlat + 3rover + ^size⁄_rbw) = d(size•rgro(1 + rrc)+rcost(1 + rrc) + 4rlat + 3rover + ^size⁄_rbw) = d(size(rgro(1 + rrc) + ¹⁄_rbw) + rcost(1 + rrc) + 4rlat + 3rover) = size • d(rgro(1 + rrc) + ¹⁄_rbw) + d(rcost(1 + rrc) + 4rlat + 3rover)

Have equation:

etime == rtime

size(egro + ¹⁄_ebw) + ecost + eover = size • d(rgro(1 + rrc) + ¹⁄_rbw) + d(rcost(1 + rrc) + 4rlat + 3rover)

size(egro + ¹⁄_ebw) - size • d(rgro(1 + rrc) + ¹⁄_rbw) = d(rcost(1 + rrc) + 4rlat + 3rover) - ecost - eover

size(egro + ¹⁄_ebw - d(rgro(1 + rrc) + ¹⁄_rbw)) = d(rcost(1 + rrc) + 4rlat + 3rover) - ecost - eover

size = ^{d(rcost(1 + rrc) + 4rlat + 3rover) - ecost - eover}⁄_{egro + ¹⁄ebw - d(rgro(1 + rrc) + ¹⁄rbw)}