NCCL Testing - chhwang/devel-note GitHub Wiki

mpirun --allow-run-as-root --tag-output -hostfile hostfile -map-by ppr:8:node --bind-to numa -mca pml ob1 -mca btl ^openib -mca btl_tcp_if_include eth0 -x PATH -x NCCL_IB_PCI_RELAXED_ORDERING=1 -x NCCL_SOCKET_IFNAME=eth0 -x CUDA_DEVICE_ORDER=PCI_BUS
_ID -x NCCL_NET_GDR_LEVEL=5 -x NCCL_TOPO_FILE=/opt/microsoft/ndv4-topo.xml -x NCCL_DEBUG=WARN ./all_reduce_perf -b 1 -e 16G -f 2 -g 1 -c 1 -w 10 -n 10 -G 10

optional: -x NCCL_P2P_READ_ENABLE=0