Huawei MN - shawfdong/hyades GitHub Wiki

MN is the Management Node of the Huawei Universal Distributed Storage. It is a 2U Huawei Tecal RH2288 V2 server, equipped with 2x quad-core Intel Xeon E5-2609 CPUs (2.40GHz), 64 GB memory, and 2x 1TB SATA HDDs configured into a RAID-1 group (managed by an LSI SAS2208 RAID controller). There are 4x Gigabit Network Interfaces and 2x 10-Gigabit Network Interfaces: 

# lspci | grep -i Ethernet
02:00.0 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
02:00.2 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
02:00.3 Ethernet controller: Intel Corporation 82580 Gigabit Network Connection (rev 01)
81:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)
81:00.1 Ethernet controller: Intel Corporation 82599EB 10-Gigabit Network Connection (rev 01)

MN runs a heavily modified version of SUSE Linux Enterprise Server 11. MN was installed using the INU4Win software from a laptop, and was then used to install the rest of the Huawei Universal Distributed Storage.

Table of Contents

Network Configuration

The 4x Gigabit Network Interfaces are eth0 – eth3; and the 2x 10-Gigabit Network Interface are eth4 & eth5.

The 2x GbE NICs, eth0 & eth1, are bonded together into a channel bonding interface bondinstall: 

# cat /etc/sysconfig/network/ifcfg-bondinstall 
BONDING_MASTER='yes'
BONDING_MODULE_OPTS='mode=2 miimon=100'
BONDING_SLAVE0='eth0'
BONDING_SLAVE1='eth1'
BOOTPROTO=static
IPADDR=192.168.0.1
NETMASK=255.255.255.0
STARTMODE='auto'
USERCONTROL='no'
which was only used during the initial installation of MN using INU4Win; but otherwise serves no purpose in UDS's operation.

The 2x 10GbE NICs, eth4 & eth5, are bonded together into bond0: 

# cat /etc/sysconfig/network/ifcfg-bond0       
BONDING_MASTER='yes'
BONDING_MODULE_OPTS='mode=2 miimon=100'
BONDING_SLAVE0='eth4'
BONDING_SLAVE1='eth5'
BOOTPROTO=static
IPADDR=172.16.0.11
NETMASK=255.255.255.0
STARTMODE='auto'
USERCONTROL='no'

A VLAN interface vlan3076 is then created on bond0: 

# cat /etc/sysconfig/network/ifcfg-vlan3076 
BOOTPROTO='static'
ETHERDEVICE='bond0'
STARTMODE='auto'
USERCONTROL='no'

A bridge interface br2 is then created on vlan3076: 

# cat /etc/sysconfig/network/ifcfg-br2 
BOOTPROTO='static'
BRIDGE='yes'
BRIDGE_FORWARDDELAY='0'
BRIDGE_PORTS='vlan3076'
BRIDGE_STP='off'
IPADDR='172.16.6.2'
NETMASK='255.255.255.0'
STARTMODE='auto'
USERCONTROL='no'

NOTE the setup (by Huawei) is unnecessarily convoluted! The bridge interface br2 is superfluous. One can simply assign an IP address to the VLAN interface vlan3076. See: 

# man ifcfg-vlan

xCAT

MN is the Management Node of xCAT (Extreme Cluster/Cloud Administration Toolkit), the toolkit for the deployment and administration of the Huawei Universal Distributed Storage. xCAT depends on the following services:

  • Advanced TFTP server (atftpd)
  • DHCP server
  • PostgreSQL server
  • NTP daemon
  • xcatd

Other Services

Apache HTTP server, which serves the HealthCheck report and the documentation for the HealthCheck tool (in /UDS/Tools/healthcheck/report) : 

http://mn/hcheck/

vsftpd (very secure FTP daemon), presumably for installation of nodes via FTP

BitTorrent tracker, presumably to speed up the installation of the nodes

Ganglia Monitoring Daemon (gmond)

WatchDog (in /opt/galax/gcs/watchdog): 

java -Ddecryptjar.path=/opt/galax/gcs/watchdog -Dclass.name=com.huawei.galax.gcs.watchdog.Main \
  -classpath /usr/java/jdk1.6.0_24/jre/bin:/opt/galax/gcs/watchdog/lib/Galax-config.jar:/opt/galax/gcs/watchdog/lib/commons-lang-2.6.jar:/opt/galax/gcs/watchdog/lib/dom4j-1.6.1.jar:/opt/galax/gcs/watchdog/lib/galax-decrypt.jar:/opt/galax/gcs/watchdog/lib/init.jar:/opt/galax/gcs/watchdog/lib/jaxen-1.1.1.jar:/opt/galax/gcs/watchdog/lib/jfig-1.5.2.jar:/opt/galax/gcs/watchdog/lib/log4j-1.2.15.jar:/opt/galax/gcs/watchdog/lib/postgresql-8.3-605.jdbc4.jar:/opt/galax/gcs/watchdog/lib/slf4j-api-1.6.1.jar:/opt/galax/gcs/watchdog/lib/slf4j-log4j12-1.6.1-selffix.jar \
  com.huawei.galax.gcs.encryption.decrypt.MainSystem

NOTE watchdog is started by the following line towards the end of /etc/init.d/rc: 

sh /opt/galax/gcs/watchdog/watchdog.sh -check
and by both /etc/rc.d/rc3.d/S13watchdog and cron.

Cron also starts the following services (see /etc/crontab):

OMS agent (in /usr/galax/oms), which appears to be a Ganglia plugin.

ZooKeeper (in /opt/sod/zookeeper): 

java -Dzookeeper.log.dir=/opt/sod/zookeeper \
  -Dzookeeper.root.logger=INFO,ROLLINGFILE,SYSLOG \
  -Djute.maxbuffer=10485760 -Dzookeeper.snapCount=300 \
  -cp /opt/sod/zookeeper/build/classes:/opt/sod/zookeeper/build/lib/*.jar:/opt/sod/zookeeper/zookeeper-3.3.4.jar:/opt/sod/zookeeper/lib/zookeeper-check-1.0.jar:/opt/sod/zookeeper/lib/log4j-1.2.15.jar:/opt/sod/zookeeper/lib/jline-0.9.94.jar:/opt/sod/zookeeper/lib/commons-lang-2.6.jar:/opt/sod/zookeeper/lib/commons-collections-3.2.jar:/opt/sod/zookeeper/lib/commons-cli-1.1.jar:/opt/sod/zookeeper/lib/apache-rat-tasks-0.6.jar:/opt/sod/zookeeper/lib/apache-rat-core-0.6.jar:/opt/sod/zookeeper/src/java/lib/*.jar \
  org.apache.zookeeper.server.quorum.QuorumPeerMain /opt/sod/zookeeper/conf/zoo.cfg

Additional cron jobs are specified in /var/spool/cron/tabs/root.

See Also

References

⚠️ **GitHub.com Fallback** ⚠️