Software - acislab/pragma-cloud-cluster GitHub Wiki

Rocks install

Wrote a USB drive with the kernel roll: http://www.rocksclusters.org/downloads/2017-12-01-download-rocks-7-0-manzanita.html

Followed installation instructions normally: http://central-7-0-x86-64.rocksclusters.org/roll-documentation/base/7.0/install-frontend-7.html

Add the following rolls at install time:

  • area51
  • base
  • CentOS
  • core
  • ganglia
  • kernel
  • kvm
  • python
  • Updates-CentOS-7.4.1708
  • zfs-linux

Fix HTTPD not starting

Apache won't start with:

A needed directory under /run does not exist. It looks like this may be fixed in an update but not sure about updating ROCKS from mainline CENTOS. Just created the directory:

mkdir /run/httpd
chown root:apache /run/httpd
chmod 0710 /run/httpd
service httpd start

Ref:

https://bugzilla.redhat.com/show_bug.cgi?id=1215667

https://ma.ttias.be/apache-2-4-ah01762-ah01760-failed-to-initialize-shm-shared-memory-segment/

ZFS

The ZFS roll no longer requires compilation so after installing all the ZFS tools are there. Make raid10-style zpool and fs for pragma boot directories on head node:

zpool create -f storage mirror ata-WDC_WD5000BPKT-00PK4T0_WD-WX11AA2R2477 ata-WDC_WD5000BPKT-00PK4T0_WD-WX51A92P7624
zpool add -f storage mirror ata-WDC_WD5000BPKT-75PK4T0_WD-WX81A23F9893 ata-WDC_WD5000BPKT-75PK4T0_WD-WX81A23R2662
zpool add -f storage mirror ata-WDC_WD5000BPKT-75PK4T0_WD-WX81A23R3759 ata-WDC_WD5000BPKT-75PK4T0_WD-WXB1E92LUR62
zpool status
zfs create -o compression=on storage/site_temp_directory
zfs create -o compression=on storage/site_repository_dir
zfs create -o compression=on storage/cziso_temp_directory
zfs list

If reinstalling these will already exist if you exported the zpool before wiping the install.

KVM roll

Use insert-ethers to PXE boot VM containers.

Main instructions are here: http://central-7-0-x86-64.rocksclusters.org/roll-documentation/base/7.0/install-compute-nodes.html

PRAGMA boot install

https://github.com/pragmagrid/pragma_boot

Also cziso as referred to in the above page.


Below is deprecated

The following notes were made during the ROCKS 6.x installation which was never completed:

Local CDROM drive was used because swapping CDs over remote emulation didn't work well. The local CDROM needed to have DMA disabled at the boot prompt of the first disk with build ide=nodma.

Ref: https://www.linuxquestions.org/questions/rock-65/after-booting-rocks-asks-for-next-cd-and-says-cd-not-found-501946/

Later, found the jumbo DVD image so CD swapping wasn't needed but still installed from local CDROM as it's a bit faster:

ftp://ftp.rocksclusters.org/pub/rocks/rocks-6.2/linux/area51+base+bio+fingerprint+ganglia+hpc+htcondor+java+kernel+kvm+os+perfSONAR+perl+python+sge+web-server+zfs-linux-6.2.x86_64.disk1.iso

Add the following rolls at install time:

  • ganglia
  • area51
  • kernel
  • os
  • base
  • kvm
  • python

Install for convenience yum --enablerepo=base install screen

Networking

After install, the KVM roll reconfigures the networking to be a physical device peth? with a bridge eth? attached which holds the IP configuration, a typical bridge VM setup. However, the regular devices are not correctly renamed to peth? at boot so networking fails.

Edit /etc/udev/rules.d/70-persistent-net.rules to set the device named to peth? and reboot. Networking should now start up correctly eg:

# PCI device 0x14e4:0x1639 (bnx2)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="e4:1f:13:2e:18:c4", ATTR{type}=="1", KERNEL=="eth*", NAME="peth0"

ZFS roll

To install roll, insert CD and run:

mount /dev/cdrom /mnt/cdrom
rocks add roll
rocks enable roll zfs-linux
cd /export/rocks/install; rocks create distro
rocks run roll zfs-linux | sh

Follow installation instructions from the documentation to build the source and install the compiled roll:

https://pcloud-head/roll-documentation/zfs-linux/0.6.4.1/adding-the-roll.html

After following those instructions, make zpool and fs on head node:

zpool create -f storage mirror ata-WDC_WD5000BPKT-00PK4T0_WD-WX11AA2R2477 ata-WDC_WD5000BPKT-00PK4T0_WD-WX51A92P7624
zpool add -f storage mirror ata-WDC_WD5000BPKT-75PK4T0_WD-WX81A23F9893 ata-WDC_WD5000BPKT-75PK4T0_WD-WX81A23R2662
zpool add -f storage mirror ata-WDC_WD5000BPKT-75PK4T0_WD-WX81A23R3759 ata-WDC_WD5000BPKT-75PK4T0_WD-WXB1E92LUR62
zpool status
zfs create -o compression=on storage/site_temp_directory
zfs create -o compression=on storage/site_repository_dir
zfs create -o compression=on storage/cziso_temp_directory
zfs list

KVM roll

Use insert-ethers to PXE boot VM containers.

PRAGMA boot install

https://github.com/pragmagrid/pragma_boot

Also cziso as referred to in the above page.