LR R2 Manual deployment on HW.rst - Seagate/cortx-motr GitHub Wiki
- # cat /etc/redhat-release
- CentOS Linux release 7.8.2003 (Core)
- # uname -r
- 3.10.0-1127.el7.x86_64
# ssh-keygen
# ssh-copy-id [email protected]
# ssh smc66-m14.pun.seagate.com
# ssh-copy-id [email protected]
# ssh smc67-m14.colo.seagate.com
Check password less login between all the nodes.
Refer https://github.com/Seagate/cortx-prvsnr/wiki/Seagate-Gallium-Storage-Controller-Configuration
- Check whether 5u84 ip's 10.0.0.2 or 10.0.0.3 are reachable or not
- If it's not working then check status of scsi-network-relay
# systemctl status scsi-network-relay
# systemctl start scsi-network-relay
- To see the current volumes setup in the 5u84
- # ./controller-cli.sh host -h 10.0.0.2 -u manage -p '!Manage20' prov -s
OR
# ssh -l manage 10.0.0.2
> show-volumes
Then cleanup old volumes and create them as per LR R2 requirement,
# ./controller-cli.sh host -h 10.0.0.2 -u manage -p '!Manage20' prov -c -a
# yum install device-mapper-multipath
- Update /etc/multipath.conf
- refer Multipath configuration provided at the end of this document.
# systemctl start multipathd.service
# Validate with "multipath -ll"
- # Define two IO disk pool for data and meta data using "multipath -ll | grep mpath"
- It will give you all drive with pool IDs like below:
mpathn (3600c0ff00050f0 cdc4120b6002000000) dm-19 SEAGATE ,5565 mpatha (3600c0ff00050f0 ebb3120b6001000000) dm-6 SEAGATE ,5565
Identify two different pools using ID differance.
# yum install fio
- # fio write-mpath.fio
- refer fio workloadd configuration provided at the end of this document.
# fio read-mpath.fio
# show_gids
Node-1, # ibv_rc_pingpong -d mlx5_3 -g 3
Node-2, # ibv_rc_pingpong -d mlx5_3 -g 3 192.168.80.
Validate all the network interfaces (IB devices : mlx5_0, mlx5_1, mlx5_2, mlx5_3)
Check iperf thouthput,
# yum install iperf
Node-1, # iperf -s -B <data network IP>
Ex. iperf -s -B 192.168.49.15Node-2, # iperf -B <local address> -c <server address> -P 8
Ex. iperf -s -B 192.168.49.12 -c 192.168.49.15 -P 8Libfabrics Throughput,
# yum install libfabrics fabtests
Node-1, # fi_msg_bw -p sockets|verbs -s 192.168.49.15 -I 100000 -S all
Node-2, # fi_msg_bw -p sockets|verbs 192.168.49.15 -I 100000 -S all
Please disable and clean previously installed repos before proceeding further. Also, verify /etc/yum.repos.d does not contain old version repo files.
# yum-config-manager --add-repo=http://cortx-storage.colo.seagate.com/releases/cortx_builds/centos-7.8.2003/538/3rd_party/
# yum-config-manager --add-repo=http://cortx-storage.colo.seagate.com/releases/cortx_builds/centos-7.8.2003/538/cortx_iso/
# yum install cortx-motr --nogpgcheck -y
# yum install cortx-hare --nogpgcheck -y
# yum install cortx-s3server --nogpgcheck -y
Lnet setup,
update Lnet interfaces as per interface being used enp175s0f1(eth1 for vm) and o2ib(tcp for vm),
- # vi /etc/modprobe.d/lnet.conf
- options lnet networks=o2ib(enp175s0f1) config_on_load=1 lnet_peer_discovery_disabled=1
# systemctl restart lnet
# lctl list_nids
# For Lnet self test refer lnet configuration provided at the end of this document.
Check with lctl ping <nid>, if ping is not working then do the following steps.
- wget https://raw.githubusercontent.com/Mellanox/gpu_direct_rdma_access/master/arp_announce_conf.sh
- sh arp_announce_conf.sh
- sysctl -w net.ipv4.conf.all.rp_filter=0
- sysctl -w net.ipv4.conf.enp175s0f0.rp_filter=0
- sysctl -w net.ipv4.conf.enp175s0f1.rp_filter=0
- sysctl -w net.ipv4.conf.enp216s0f0.rp_filter=0
- sysctl -w net.ipv4.conf.enp216s0f1.rp_filter=0
- ip neigh flush all
- systemctl restart lnet
# yum-config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
# yum -y install consul-1.7.8
Create metadata partition on each node and update it on CDF file.
# hctl bootstrap --mkfs cluster-mp-R1.yaml
# /opt/seagate/cortx/hare/libexec/m0crate-io-conf > crate.yaml
# dd if=/dev/urandom of=/tmp/128M bs=1M count=128
# m0crate -S crate.yaml
# hctl shutdown
Setup HAproxy,
# yum install haproxy
# cp /opt/seagate/cortx/s3/install/haproxy/haproxy_osver7.cfg /etc/haproxy/haproxy.cfg
# mkdir /etc/haproxy/errors/
# cp /opt/seagate/cortx/s3/install/haproxy/503.http /etc/haproxy/errors/
# Update haproxy config for number s3server instances, /etc/haproxy/haproxy.cfg
# systemctl restart haproxy
Setup Ldap,
# cd /opt/seagate/cortx/s3/install/ldap/
# ./setup_ldap.sh --defaultpasswd --skipssl --forceclean
# systemctl restart slapd
- # vim /etc/hosts
127.0.0.1 - s3.seagate.com sts.seagate.com iam.seagate.com sts.cloud.seagate.com
# systemctl restart s3authserver
Setup S3 client,
# easy_install pip
# pip3 install awscli
# pip install awscli-plugin-endpoint
# aws configure
# aws configure set plugins.endpoint awscli_plugin_endpoint
# aws configure set s3.endpoint_url http://s3.seagate.com
# aws configure set s3api.endpoint_url http://s3.seagate.com
# s3iamcli CreateAccount -n motr -e [email protected] --ldapuser sgiamadmin --ldappasswd ldapadmin
# aws s3 ls
# aws s3 mb s3://testbucket1
# aws s3 cp /tmp/128M s3://testbucket1
Run s3bench,
# yum install go
# go get github.com/igneous-systems/s3bench
# /root/go/bin/s3bench -accessKey AKIAwk0geOx8SnCfiveCzzV0Uw -accessSecret rD2FQIeMf0ZBsiEaKfAEafeNoP0K/B9p0Bm+ox3+ -bucket firstbucket7 -endpoint http://s3.seagate.com -numClients 512 -numSamples 4096 -objectSize 134217728b -verbose
- https://github.com/Seagate/cortx/blob/main/doc/scaleout/README.rst
- https://github.com/Seagate/cortx/blob/main/doc/Cluster_Setup.md
- https://github.com/Seagate/cortx-s3server/blob/main/docs/R2-setup/3%20Node%20Manual%20Hare%20Cluster%20Setup.md
- https://seagatetechnology.sharepoint.com/:w:/r/sites/gteamdrv1/tdrive1224/_layouts/15/guestaccess.aspx?e=D9OLR8&share=EfvqJjlha8pNkOeIfCgDywUBn0YNIYGT6-tWMREx9iGxpw
- https://github.com/Seagate/cortx-hare/blob/dev/README_developers.md
- https://seagatetechnology.sharepoint.com/:w:/r/sites/gteamdrv1/tdrive1224/_layouts/15/Doc.aspx?sourcedoc=%7B8c2c808f-3b39-4331-9a93-c67009ae7fdd%7D&action=edit&wdPreviousSession=27cb5705-c553-4852-bdc1-6e01de59f8ef&cid=3e7b558b-8aa7-4fea-a50e-91816f8be2b9
[root@smc65-m14 ~]# cat /etc/multipath.conf
defaults {
polling_interval 10
max_fds 8192
user_friendly_names yes
find_multipaths yes
}
devices {
device {
vendor "SEAGATE"
product "*"
path_grouping_policy group_by_prio
uid_attribute "ID_SERIAL"
prio alua
path_selector "service-time 0"
path_checker tur
hardware_handler "1 alua"
failback immediate
rr_weight uniform
rr_min_io_rq 1
no_path_retry 18
}
}
- [root@smc65-m14 ~]# cat cluster-R2.yaml
-
# Cluster Description File (CDF). # See `cfgen --help-schema` for the format description. nodes: - hostname: smc65-m14.colo.seagate.com # [user@]hostname data_iface: enp175s0f0 # name of data network interface data_iface_type: o2ib # LNet type of network interface (optional); # supported values: "tcp" (default), "tcp" m0_servers: - runs_confd: true io_disks: data: [] - io_disks: data: - /dev/mapper/mpatha - /dev/mapper/mpathb - /dev/mapper/mpathc - /dev/mapper/mpathd - /dev/mapper/mpathe - /dev/mapper/mpathf - /dev/mapper/mpathg - /dev/mapper/mpathh - io_disks: data: - /dev/mapper/mpathi - /dev/mapper/mpathj - /dev/mapper/mpathk - /dev/mapper/mpathl - /dev/mapper/mpathm - /dev/mapper/mpathn - /dev/mapper/mpatho - /dev/mapper/mpathp m0_clients: s3: 21 # number of S3 servers to start other: 2 # max quantity of other Mero clients this node may have - hostname: smc66-m14.colo.seagate.com data_iface: enp175s0f0 data_iface_type: o2ib m0_servers: - runs_confd: true io_disks: data: [] - io_disks: data: - /dev/mapper/mpatha - /dev/mapper/mpathb - /dev/mapper/mpathc - /dev/mapper/mpathd - /dev/mapper/mpathe - /dev/mapper/mpathf - /dev/mapper/mpathg - /dev/mapper/mpathh - io_disks: data: - /dev/mapper/mpathi - /dev/mapper/mpathj - /dev/mapper/mpathk - /dev/mapper/mpathl - /dev/mapper/mpathm - /dev/mapper/mpathn - /dev/mapper/mpatho - /dev/mapper/mpathp m0_clients: s3: 21 # number of S3 servers to start other: 2 - hostname: smc67-m14.colo.seagate.com # [user@]hostname data_iface: enp175s0f0 # name of data network interface data_iface_type: o2ib # LNet type of network interface (optional); # supported values: "tcp" (default), "tcp" m0_servers: - runs_confd: true io_disks: data: [] - io_disks: data: - /dev/mapper/mpatha - /dev/mapper/mpathb - /dev/mapper/mpathc - /dev/mapper/mpathd - /dev/mapper/mpathe - /dev/mapper/mpathf - /dev/mapper/mpathg - /dev/mapper/mpathh - io_disks: data: - /dev/mapper/mpathi - /dev/mapper/mpathj - /dev/mapper/mpathk - /dev/mapper/mpathl - /dev/mapper/mpathm - /dev/mapper/mpathn - /dev/mapper/mpatho - /dev/mapper/mpathp m0_clients: s3: 21 # number of S3 servers to start other: 2 # max quantity of other Mero clients this node may have pools: - name: the pool type: sns data_units: 4 parity_units: 2 # allowed_failures: { site: 0, rack: 0, encl: 0, ctrl: 0, disk: 0 }
# cat write.fio
[global] direct=1 ioengine=libaio iodepth=16 ;invalidate=1 ramp_time=5 runtime=60 time_based bs=1M rw=write numjobs=32 [job0] filename=/dev/mapper/mpatha [job1] filename=/dev/mapper/mpathb [job2] filename=/dev/mapper/mpathc [job3] filename=/dev/mapper/mpathd [job4] filename=/dev/mapper/mpathe [job5] filename=/dev/mapper/mpathf [job6] filename=/dev/mapper/mpathg [job7] filename=/dev/mapper/mpathh [job8] filename=/dev/mapper/mpathi [job9] filename=/dev/mapper/mpathj [job10] filename=/dev/mapper/mpathk [job11] filename=/dev/mapper/mpathl [job12] filename=/dev/mapper/mpathm [job13] filename=/dev/mapper/mpathn [job14] filename=/dev/mapper/mpatho [job15] filename=/dev/mapper/mpathp
# cat lst.sh
export LST_SESSION=$$ lst new_session twonoderead lst add_group server 192.168.49.49@o2ib lst add_group client 192.168.49.3@o2ib lst add_batch bulk_rw lst add_test --concurrency 90 --batch bulk_rw --from client --to server brw write size=1M lst run bulk_rw lst stat client server & sleep 30; kill $! lst stop bulk_read lst end_session
Run following command on both the nodes,
# modprobe lnet_selftest
# sh lst.sh
Currently R2 setups comes up with four NIC ports of 100G each, which is not yet supported. So need to convert them to 50G and have to use only two ports, one for public data and other for private data.
- ethtool -s enp175s0f0 speed 50000 duplex full autoneg off
- ethtool -s enp175s0f1 speed 50000 duplex full autoneg off
The following steps too are needed as iperf gives 100G with them being enabled.
- ifdown enp216s0f0
- ifdown enp216s0f1
Steps to revert back to 100G,
- ethtool enp175s0f0 | grep Speed
- ethtool enp175s0f1 | grep Speed
- ethtool -s enp175s0f0 speed 100000 duplex full autoneg off
- ethtool -s enp175s0f1 speed 100000 duplex full autoneg off
- ethtool -s enp175s0f1 autoneg on
- ethtool -s enp175s0f0 autoneg on
- ifup enp216s0f0
- ifup enp216s0f1
Super Micro Setup,
- smc65-m14.colo.seagate.com
- smc66-m14.colo.seagate.com
- smc67-m14.colo.seagate.com
- client: smc75-m14.colo.seagate.com
- smc74-m14.colo.seagate.com
Dell Setup,
- dell05-m02.colo.seagate.com
- dell06-m02.colo.seagate.com
- dell07-m02.colo.seagate.com
client: smc73-m02.colo.seagate.com
Pune setup-1
- sm6-r4.pun.seagate.com
- sm20-r5.pun.seagate.com
- sm13-r5.pun.seagate.com
client: sm36-r2.pun.seagate.com
Pune setup-2
- sm13-r1.pun.seagate.com
- sm20-r1.pun.seagate.com
- sm27-r1.pun.seagate.com
Client: sm35-r1.pun.seagate.com