Huawei SoD - shawfdong/hyades GitHub Wiki
SoD is the storage node of the Huawei Universal Distributed Storage. Each node is armed with a Marvell Kirkwood processor at 1.6 GHz (ARMADA 300 SoC, utilizing the ARMv5 instruction set)[1], 1GB memory, and a 2TB enterprise-class SATA hard drive.
SoD runs embedded Linux:
root@sod0:~# uname -a Linux sod0 2.6.34.6-WR4.0.0.0_standard #1 Tue Aug 21 11:23:39 CST 2012 armv5tel GNU/Linux
There is a 512MB NAND flash in each SoD node, divided into 6 partitions:
# cat /proc/mtd dev: size erasesize name mtd0: 00100000 00020000 "uboot" mtd1: 00400000 00020000 "kernel" mtd2: 00400000 00020000 "kernel_back" mtd3: 00a00000 00020000 "ramdisk" mtd4: 00a00000 00020000 "ramdisk_back" mtd5: 1dd00000 00020000 "usr" mtd6: 00600000 00020000 "config" # mtd_debug info /dev/mtd0
The entries are mostly self-explanatory[2]. For example, mtd0 is the Bootloader partition for U-Boot[3]; mtd5 holds a Unsorted Block Images (UBI) file system and is mounted at /usr.
# df -h Filesystem Size Used Avail Use% Mounted on rootfs 61M 20M 41M 33% / /dev/root 61M 20M 41M 33% / tmpfs 501M 32K 501M 1% /var/run tmpfs 501M 0 501M 0% /var/lock devpts 501M 4.0K 501M 1% /dev/pts udev 501M 4.0K 501M 1% /dev ubi5:usr 421M 238M 184M 57% /usr /dev/sda 1.8T 424G 1.3T 25% /opt
NOTE there is no partition table on the 2TB SATA drive. The whole disk is formatted as an EXT3 file system, and is mounted at /opt.
There are 2 Gigabit Ethernet MACs with Media Independent Interface (MII) ports. The 2 NICs are are bonded together into a channel bonding interface bond0. A VLAN interface bond0.4092 and a sub interface (alias) bond0.4092:0 are then created on bond0.
| Interface | IP Address | Netmask | VLAN |
|---|---|---|---|
| bond0 | 172.16.$rack.$rack-rank | 255.255.255.0 | |
| bond0.4092 | 192.168.2.$UDSN-rank | 255.255.255.0 | 4092 |
| bond0.4092:0 | 192.168.3.$UDSN-rank | 255.255.255.0 | 4092 |
where $rack is the rack # of the SoD node (0 – 2); $rack-rank the rank # in the rack (16 – 201); $UDSN-rank the rank # inside its UDSN enclosure (1 – 15). For example:
| SoD node | bond0 | bond0.4092 | bond0.4092:0 | Description |
|---|---|---|---|---|
| sod0 | 172.16.0.16 | 192.168.2.1 | 192.168.3.1 | $rack=0, $rack-rank=16, $UDSN-rank=1 |
| sod15 | 172.16.0.31 | 192.168.2.16 | 192.168.3.16 | $rack=0, $rack-rank=31, $UDSN-rank=16 |
| sod144 | 172.16.0.169 | 192.168.2.1 | 192.168.3.1 | $rack=0, $rack-rank=169, $UDSN-rank=1 |
| sod159 | 172.16.0.184 | 192.168.2.1 | 192.168.3.1 | $rack=0, $rack-rank=184, $UDSN-rank=16 |
| sod160 | 172.16.1.16 | 192.168.2.1 | 192.168.3.1 | $rack=1, $rack-rank=16, $UDSN-rank=1 |
| sod175 | 172.16.1.31 | 192.168.2.16 | 192.168.3.16 | $rack=1, $rack-rank=31, $UDSN-rank=16 |
| sod320 | 172.16.1.186 | 192.168.2.1 | 192.168.3.1 | $rack=1, $rack-rank=186, $UDSN-rank=1 |
| sod335 | 172.16.1.201 | 192.168.2.16 | 192.168.3.16 | $rack=1, $rack-rank=201, $UDSN-rank=16 |
| sod336 | 172.16.2.16 | 192.168.2.1 | 192.168.3.1 | $rack=2, $rack-rank=16, $UDSN-rank=1 |
| sod351 | 172.16.2.31 | 192.168.2.16 | 192.168.3.16 | $rack=2, $rack-rank=31, $UDSN-rank=16 |
| sod496 | 172.16.2.186 | 192.168.2.1 | 192.168.3.1 | $rack=2, $rack-rank=186, $UDSN-rank=1 |
| sod511 | 172.16.2.201 | 192.168.2.16 | 192.168.3.16 | $rack=2, $rack-rank=201, $UDSN-rank=16 |
NOTE each UDSN holds two (2x) independent UM680 blade servers, which are installed in slot 1 and slot 2 located in the front of the chassis; and each UM680 blade carries a Gigabit Ethernet switch and eight (8x) Sod nodes. The switch in slot 1 is connected to the first 8 SoD nodes ($UDSN-rank = 1 – 8) and eth0 of the UDSN's BMC, whose IP addresses are 192.168.2.1/192.168.3.1 – 192.168.2.8/192.168.3.8, and 192.168.2.17. The switch in slot 2 is connected to the second 8 SoD nodes ($UDSN-rank = 9 – 16) and eth1 of the UDSN's BMC, whose IP addresses are 192.168.2.9/192.168.3.9 – 192.168.2.16/192.168.3.16, and 192.168.3.17. So for example, 192.168.2.1 can ping 192.168.2.8 and 192.168.2.17, but can't reach 192.168.2.16; similarly, 192.168.2.16 can ping 192.168.2.9, but can reach neither 192.168.2.1 nor 192.168.2.17. Apparently, this scheme is unnecessary convoluted and confusing!
Dropbear SSH server
sod-agent (in /usr/sod/sod-agent/)
java -Dlog4j.configuration=config/log4j.xml -Duser.timezone=UTC \ -cp :/usr/sod/sod-agent/lib/commons-codec-1.3.jar:/usr/sod/sod-agent/lib/commons-collections-3.2.1.jar:/usr/sod/sod-agent/lib/commons-dbcp-1.2.2.jar:/usr/sod/sod-agent/lib/commons-io-2.1.jar:/usr/sod/sod-agent/lib/commons-lang-2.6.jar:/usr/sod/sod-agent/lib/commons-pool-1.5.2.jar:/usr/sod/sod-agent/lib/google-collect-1.0.jar:/usr/sod/sod-agent/lib/jackson-core-asl-1.4.0.jar:/usr/sod/sod-agent/lib/jdom-1.1.jar:/usr/sod/sod-agent/lib/jna.jar:/usr/sod/sod-agent/lib/log4j-1.2.15.jar:/usr/sod/sod-agent/lib/pb_agent.jar:/usr/sod/sod-agent/lib/pb_server.jar:/usr/sod/sod-agent/lib/perf4j.jar:/usr/sod/sod-agent/lib/protobuf_sod.jar:/usr/sod/sod-agent/lib/sod-agent-1.00.jar:/usr/sod/sod-agent/lib/sod-lib.jar:/usr/sod/sod-agent/lib/tokyocabinet.jar \ com.huawei.sod.agent.server.AgentServer config/config.xml
sod-server (in /opt/sod/sod-server/)
java -Dlog4j.configuration=config/log4j.xml -Xmx384M -Xms384M \ -server -XX:MaxDirectMemorySize=256M -Dsod.jmxport=9300 -Duser.timezone=UTC \ -cp :/opt/sod/sod-server/lib/commons-codec-1.3.jar:/opt/sod/sod-server/lib/commons-collections-3.2.1.jar:/opt/sod/sod-server/lib/commons-dbcp-1.2.2.jar:/opt/sod/sod-server/lib/commons-io-2.1.jar:/opt/sod/sod-server/lib/commons-lang-2.6.jar:/opt/sod/sod-server/lib/commons-logging-1.1.1.jar:/opt/sod/sod-server/lib/commons-pool-1.5.2.jar:/opt/sod/sod-server/lib/google-collect-1.0.jar:/opt/sod/sod-server/lib/jackson-core-asl-1.4.0.jar:/opt/sod/sod-server/lib/jackson-mapper-asl-1.4.0.jar:/opt/sod/sod-server/lib/jdom-1.1.jar:/opt/sod/sod-server/lib/jline-0.9.94.jar:/opt/sod/sod-server/lib/jna.jar:/opt/sod/sod-server/lib/jopt-simple-3.1.jar:/opt/sod/sod-server/lib/log4j-1.2.15.jar:/opt/sod/sod-server/lib/paranamer-2.1.jar:/opt/sod/sod-server/lib/pb_agent.jar:/opt/sod/sod-server/lib/pb_server.jar:/opt/sod/sod-server/lib/perf4j.jar:/opt/sod/sod-server/lib/protobuf_sod.jar:/opt/sod/sod-server/lib/slf4j-api-1.6.1.jar:/opt/sod/sod-server/lib/slf4j-log4j12-1.6.1.jar:/opt/sod/sod-server/lib/sod-server-1.00.jar:/opt/sod/sod-server/lib/tokyocabinet.jar \ voldemort.server.VoldemortServer /opt/sod/sod-node
om-agent (in /usr/galax/oms/)