Huawei SoD - shawfdong/hyades GitHub Wiki

SoD is the storage node of the Huawei Universal Distributed Storage. Each node is armed with a Marvell Kirkwood processor at 1.6 GHz (ARMADA 300 SoC, utilizing the ARMv5 instruction set)[1], 1GB memory, and a 2TB enterprise-class SATA hard drive.

SoD runs embedded Linux:

root@sod0:~# uname -a
Linux sod0 2.6.34.6-WR4.0.0.0_standard #1 Tue Aug 21 11:23:39 CST 2012 armv5tel GNU/Linux

Table of Contents

Storage

There is a 512MB NAND flash in each SoD node, divided into 6 partitions:

# cat /proc/mtd 
dev:    size   erasesize  name
mtd0: 00100000 00020000 "uboot"
mtd1: 00400000 00020000 "kernel"
mtd2: 00400000 00020000 "kernel_back"
mtd3: 00a00000 00020000 "ramdisk"
mtd4: 00a00000 00020000 "ramdisk_back"
mtd5: 1dd00000 00020000 "usr"
mtd6: 00600000 00020000 "config"

# mtd_debug info /dev/mtd0

The entries are mostly self-explanatory[2]. For example, mtd0 is the Bootloader partition for U-Boot[3]; mtd5 holds a Unsorted Block Images (UBI) file system and is mounted at /usr.



# df -h
Filesystem            Size  Used Avail Use% Mounted on
rootfs                 61M   20M   41M  33% /
/dev/root              61M   20M   41M  33% /
tmpfs                 501M   32K  501M   1% /var/run
tmpfs                 501M     0  501M   0% /var/lock
devpts                501M  4.0K  501M   1% /dev/pts
udev                  501M  4.0K  501M   1% /dev
ubi5:usr              421M  238M  184M  57% /usr
/dev/sda              1.8T  424G  1.3T  25% /opt

NOTE there is no partition table on the 2TB SATA drive. The whole disk is formatted as an EXT3 file system, and is mounted at /opt.




Network Configuration

There are 2 Gigabit Ethernet MACs with Media Independent Interface (MII) ports. The 2 NICs are are bonded together into a channel bonding interface bond0. A VLAN interface bond0.4092 and a sub interface (alias) bond0.4092:0 are then created on bond0.

Interface IP Address Netmask VLAN
bond0 172.16.$rack.$rack-rank 255.255.255.0
bond0.4092 192.168.2.$UDSN-rank 255.255.255.0 4092
bond0.4092:0 192.168.3.$UDSN-rank 255.255.255.0 4092

where $rack is the rack # of the SoD node (0 – 2); $rack-rank the rank # in the rack (16 – 201); $UDSN-rank the rank # inside its UDSN enclosure (1 – 15). For example:

SoD node bond0 bond0.4092 bond0.4092:0 Description
sod0 172.16.0.16 192.168.2.1 192.168.3.1 $rack=0, $rack-rank=16, $UDSN-rank=1
sod15 172.16.0.31 192.168.2.16 192.168.3.16 $rack=0, $rack-rank=31, $UDSN-rank=16
sod144 172.16.0.169 192.168.2.1 192.168.3.1 $rack=0, $rack-rank=169, $UDSN-rank=1
sod159 172.16.0.184 192.168.2.1 192.168.3.1 $rack=0, $rack-rank=184, $UDSN-rank=16
sod160 172.16.1.16 192.168.2.1 192.168.3.1 $rack=1, $rack-rank=16, $UDSN-rank=1
sod175 172.16.1.31 192.168.2.16 192.168.3.16 $rack=1, $rack-rank=31, $UDSN-rank=16
sod320 172.16.1.186 192.168.2.1 192.168.3.1 $rack=1, $rack-rank=186, $UDSN-rank=1
sod335 172.16.1.201 192.168.2.16 192.168.3.16 $rack=1, $rack-rank=201, $UDSN-rank=16
sod336 172.16.2.16 192.168.2.1 192.168.3.1 $rack=2, $rack-rank=16, $UDSN-rank=1
sod351 172.16.2.31 192.168.2.16 192.168.3.16 $rack=2, $rack-rank=31, $UDSN-rank=16
sod496 172.16.2.186 192.168.2.1 192.168.3.1 $rack=2, $rack-rank=186, $UDSN-rank=1
sod511 172.16.2.201 192.168.2.16 192.168.3.16 $rack=2, $rack-rank=201, $UDSN-rank=16

NOTE each UDSN holds two (2x) independent UM680 blade servers, which are installed in slot 1 and slot 2 located in the front of the chassis; and each UM680 blade carries a Gigabit Ethernet switch and eight (8x) Sod nodes. The switch in slot 1 is connected to the first 8 SoD nodes ($UDSN-rank = 1 – 8) and eth0 of the UDSN's BMC, whose IP addresses are 192.168.2.1/192.168.3.1 – 192.168.2.8/192.168.3.8, and 192.168.2.17. The switch in slot 2 is connected to the second 8 SoD nodes ($UDSN-rank = 9 – 16) and eth1 of the UDSN's BMC, whose IP addresses are 192.168.2.9/192.168.3.9 – 192.168.2.16/192.168.3.16, and 192.168.3.17. So for example, 192.168.2.1 can ping 192.168.2.8 and 192.168.2.17, but can't reach 192.168.2.16; similarly, 192.168.2.16 can ping 192.168.2.9, but can reach neither 192.168.2.1 nor 192.168.2.17. Apparently, this scheme is unnecessary convoluted and confusing!

Services

Dropbear SSH server

sod-agent (in /usr/sod/sod-agent/)

java -Dlog4j.configuration=config/log4j.xml -Duser.timezone=UTC \
  -cp :/usr/sod/sod-agent/lib/commons-codec-1.3.jar:/usr/sod/sod-agent/lib/commons-collections-3.2.1.jar:/usr/sod/sod-agent/lib/commons-dbcp-1.2.2.jar:/usr/sod/sod-agent/lib/commons-io-2.1.jar:/usr/sod/sod-agent/lib/commons-lang-2.6.jar:/usr/sod/sod-agent/lib/commons-pool-1.5.2.jar:/usr/sod/sod-agent/lib/google-collect-1.0.jar:/usr/sod/sod-agent/lib/jackson-core-asl-1.4.0.jar:/usr/sod/sod-agent/lib/jdom-1.1.jar:/usr/sod/sod-agent/lib/jna.jar:/usr/sod/sod-agent/lib/log4j-1.2.15.jar:/usr/sod/sod-agent/lib/pb_agent.jar:/usr/sod/sod-agent/lib/pb_server.jar:/usr/sod/sod-agent/lib/perf4j.jar:/usr/sod/sod-agent/lib/protobuf_sod.jar:/usr/sod/sod-agent/lib/sod-agent-1.00.jar:/usr/sod/sod-agent/lib/sod-lib.jar:/usr/sod/sod-agent/lib/tokyocabinet.jar \
  com.huawei.sod.agent.server.AgentServer config/config.xml

sod-server (in /opt/sod/sod-server/)

java -Dlog4j.configuration=config/log4j.xml -Xmx384M -Xms384M \
  -server -XX:MaxDirectMemorySize=256M -Dsod.jmxport=9300  -Duser.timezone=UTC \
  -cp :/opt/sod/sod-server/lib/commons-codec-1.3.jar:/opt/sod/sod-server/lib/commons-collections-3.2.1.jar:/opt/sod/sod-server/lib/commons-dbcp-1.2.2.jar:/opt/sod/sod-server/lib/commons-io-2.1.jar:/opt/sod/sod-server/lib/commons-lang-2.6.jar:/opt/sod/sod-server/lib/commons-logging-1.1.1.jar:/opt/sod/sod-server/lib/commons-pool-1.5.2.jar:/opt/sod/sod-server/lib/google-collect-1.0.jar:/opt/sod/sod-server/lib/jackson-core-asl-1.4.0.jar:/opt/sod/sod-server/lib/jackson-mapper-asl-1.4.0.jar:/opt/sod/sod-server/lib/jdom-1.1.jar:/opt/sod/sod-server/lib/jline-0.9.94.jar:/opt/sod/sod-server/lib/jna.jar:/opt/sod/sod-server/lib/jopt-simple-3.1.jar:/opt/sod/sod-server/lib/log4j-1.2.15.jar:/opt/sod/sod-server/lib/paranamer-2.1.jar:/opt/sod/sod-server/lib/pb_agent.jar:/opt/sod/sod-server/lib/pb_server.jar:/opt/sod/sod-server/lib/perf4j.jar:/opt/sod/sod-server/lib/protobuf_sod.jar:/opt/sod/sod-server/lib/slf4j-api-1.6.1.jar:/opt/sod/sod-server/lib/slf4j-log4j12-1.6.1.jar:/opt/sod/sod-server/lib/sod-server-1.00.jar:/opt/sod/sod-server/lib/tokyocabinet.jar \
  voldemort.server.VoldemortServer /opt/sod/sod-node

om-agent (in /usr/galax/oms/)

References

  1. ^ Marvell ARMADA 300/310 SoCs
  2. ^ Managing flash storage with Linux
  3. ^ Das U-Boot -- the Universal Boot Loader
⚠️ **GitHub.com Fallback** ⚠️