Eureka Installation: Computing Node - calab-ntu/gpu-cluster GitHub Wiki

Choose one of the following two approaches.

• Install From Scratch
• Clone From Another Computer

---

0. Component list: CPU, GPU, motherboard, power supply, SSD (256G), RAM (16G*8), water cooler
   
1. Wash hands, dry, and touch metal to discharge
   
2. NVIDIA RTX 2080 Super GPU:
   
   1. Plug-in two (6+2)-pin power cables
   2. Install on the PCIe x16 slot closest to RAM
   3. Remove the protective film on the back of GPU and repaste the NTU property tag
   4. Verify that the GPU BIOS hardware switch is set to P MODE (performance) instead of Q MODE (quiet)
3. Power cable: use the one shipped together with the power supply
4. Water cooler
   1. Cooling fan orientation: consistent with other nodes
   2. Use the PWM instead of AUTO mode
   3. Thermal paste MUST firmly touches CPU
5. Copper pillar x10
6. Reset and power switches
7. Wire trimming
   
8. Labels: MB, power cable, IB cable, Ethernet cable, cabinet
9. Ethernet cable: use the left (black) RJ45 port instead of the right (red) one
10. Connect all power plugs on the MB
11. CPU 8 pin x2
12. ATX 24 pin
13. Water cooler (use CPU_FAN2/WP)
14. Ground both MB and power supply

1. Update BIOS to 1.30

   1. Install the USB stick Packages
   2. Press F2 during boot
   3. Tool -> Instant Flash -> It should automatically find the file \bios\X399PGH1.30 -> Update (use Tab to switch)

2. Enable NUMA

   1. Reboot and F2
   2. Advanced -> AMD CBS -> DF Common Options -> Memory interleaving -> Channel

3. Verification

   1. Reboot and F2
   2. Main: UEFI Version=P1.30
   3. Main:
      1. Total Memory= 131072MB
         --> If less than this number, check the missing slot(s) (e.g., DDR4_A1) and reinstall the RAM
      2. Quadruple-Channel Memory Mode
      3. Frequency of all 8 slots should be DDR4-2933
         --> If the frequency is wrong, click OC Tweaker and set
         
         • Load XMP Setting = XMP 2.0 Profile 1
           
         • DRAM Frequency = DDR4-2933
   4. CPU_FAN2= WMP

4. Shut down the system

1. Install USB stick CentOS

2. Turn on the computer and press F2 to go to the BIOS

   1. Boot -> Boot Option #1 -> UEFI: KingstonDataTraveler 3.0 Partition 2
   2. Exit -> Save Changes and Exit

3. Test this media & install CentOS 7

4. Installation summary:

   1. LANGUAGE: English (United States)
   2. DATE & TIME: Asia, Taipei
   3. SOFTWARE SELECTION: Development and Creative Workstation
      • Additional Development
        
      • Compatibility Libraries
        
      • Development Tools
        
      • Emacs
        
      • File and Storage Server
        
      • Hardware Monitoring Utilities
        
      • Infiniband Support
        
      • Legacy X Window System Compatibility
        
      • Network File System Client
        
      • Platform Development
        
      • Python
        
      • Technical Writing
   4. INSTALLATION DESTINATION:
      1. Select INTEL SSD
      2. Partitioning -> I will configure partitioning -> Done -> Click here to create them automatically
         -> Verify that the total space is about 256 GB (238.47 GiB in my test)
         -> /home: click - to remove it since we will mount a remote home
         /: 221 GiB
         swap: 16 GiB
         -> Done -> Accept Changes
   5. NETWORK & HOST NAME:
      1. Ethernet -> ON
      2. Host name (on the bottom left corner): eurekaXX -> Apply
   6. After finishing (a)-(e) above, press Begin Installation (ignore the warning about SMT)
   7. Set root password

5. The installation will take ~10 min

   1. Reboot -> Accept the EULA agreement -> FINISH CONFIGURATION
   2. Location Services: OFF
      Time Zone: Taipei, Taiwan
Connect Your Online Accounts: Skip
About You: (this account will be removed later)
      Full Name: tmp_account
      Username: tmp_account
      Password: ********
      
   3. Unplug the USB stick

6. Validate kernel (switch to root)

   uname -r # 3.10.0-1062.el7.x86_64

7. Set up network

   1. Edit /etc/sysconfig/network-scripts/ifcfg-XXX, where XXX is usually "enpXXX"

      BOOTPROTO=static
      ONBOOT=yes
      IPADDR=192.168.0.1XX    # Replace XX by the node number + 50 (node number = 00 for the login node)
      GATEWAY=192.168.0.1
      NETMASK=255.255.255.0
      DNS1=140.112.254.4
      

   2. Disable the network interface virbr0
      --> It prevents MPI from working properly
      --> Ref: https://www.thegeekdiary.com/centos-rhel-67-how-to-disable-or-delete-virbr0-interface/
      

      virsh net-autostart default --disable

   3. Disable SELinux and firewall (necessary for torque)

      setenforce 0
      sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config # or just edit "/etc/selinux/config" directly
      systemctl stop firewalld
      systemctl disable firewalld
      
      sestatus # Check the SELinux status
               # --> Before reboot, it should show
               #     SELinux status:                 enabled
               #     Current mode:                   permissive
               # --> After reboot, it should show
               #     SELinux status:                 disabled

   4. Shut down the system, move it to the computer room, and plug in the Ethernet cable

   5. Boot -> validate network

      1. Ethernet port LED is on
      2. Switch LED is on (green for 1 Gb/s)
      3. ping 192.168.0.150
      4. ifconfig: No virbr0 interface

0. Update kernel to kernel-3.10.0-1160.****

   1. Edit /etc/yum.repos.d/CentOS-*: Change the mirror source from mirror.centos.org to vault.centos.org

      ref. https://serverfault.com/questions/1161816/mirrorlist-centos-org-no-longer-resolve

      sed -i s/mirror.centos.org/vault.centos.org/g /etc/yum.repos.d/*.repo
      sed -i s/^#.*baseurl=http/baseurl=http/g /etc/yum.repos.d/*.repo
      sed -i s/^mirrorlist=http/#mirrorlist=http/g /etc/yum.repos.d/*.repo
      

   2. Update kernel

      yum install -y kernel-3.10.0-1160 #press <tab> to fill up availible version
      

1. Remove the temporary user
   userdel -r tmp_account

2. NFS client

   1. Auto mount

      scp 192.168.0.150:/work1/shared/eureka/etc/hosts /etc/hosts
      ssh eureka00 cat /work1/shared/eureka/etc/fstab >> /etc/fstab

   2. Create folders
      

      mkdir /software /work1 /project /projectX /projectY /projectZ

   3. Check the accessibility of the target NFS servers:

      showmount -e eureka00   # /software 192.168.0.0/24
      
      showmount -e tumaz      # /home 192.168.0.0/24
      
      showmount -e ironman    # /volume1/gpucluster1 192.168.0.0/24
                              # /volume2/gpucluster2 192.168.0.0/24
                              # /volume3/gpucluster3 192.168.0.0/24
      
      showmount -e eater      # /volume1/gpucluster3 192.168.0.0/24
                              # /volume2/gpucluster4 192.168.0.0/24

   4. Mount all remote folders

      mount /software; mount /home; mount /work1; mount /project; mount /projectX; mount /projectY; mount /projectZ
      
      df # Check if all folders have been mounted

3. CUDA

   1. [For current system] Update system

      yum -y update
      

   2. Set the text mode as default (since the NVIDIA driver cannot be installed while X window is running)

      systemctl set-default multi-user.target

      Check it with

      systemctl get-default   # It should show "multi-user.target" instead of "graphical.target"

   3. Reboot

   4. Install EPEL and ELRepo

      # [Optional] yum may fail due to incorrect local system time (which causes failure of certification)
      #            --> Solution 1: reset system time directly with, for example, 'date -s "2020-03-22 19:06:26"'
      #            --> Solution 2: enable NTP by following step 3-(5) below
      
      yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
      yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm

   5. Install dkms

      yum -y install dkms

   6. Disable the display driver nouveau

      1. Add the following line to the end of GRUB_CMDLINE_LINUX in /etc/default/grub

         rd.driver.blacklist=nouveau nouveau.modeset=0

         Alternatively, one can simply copy it from

         cp /work1/shared/eureka/etc/grub /etc/default/grub

      2. Execute the following

         grub2-mkconfig -o /boot/grub2/grub.cfg
         
         echo "blacklist nouveau"         >> /etc/modprobe.d/blacklist.conf
         echo "options nouveau modeset=0" >> /etc/modprobe.d/blacklist.conf
         
         mv /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r)-nouveau.img
         dracut /boot/initramfs-$(uname -r).img $(uname -r)
         grub2-mkconfig -o /boot/grub2/grub.cfg

      3. Reboot

      4. Verify that the nouveau driver is not loaded

         lsmod | grep nouveau # It should print nothing

   7. Install the CUDA driver

      1. [Optional] Disable persistence and MPS daemons if you are upgrading an existing driver

         nvidia-smi -pm 0
         echo quit | nvidia-cuda-mps-control

      2. Install with

         sh /work1/shared/eureka/package/cuda/cuda_11.3.1_465.19.01_linux.run --silent --driver

      3. Validate with

         cat /proc/driver/nvidia/version # NVIDIA driver: 465.19.01; gcc: 4.8.5

      4. Copy initialization scripts

         cp /work1/shared/eureka/init_script/*.sh  /etc/profile.d/
         cp /work1/shared/eureka/init_script/*.csh /etc/profile.d/
         
         cp /work1/shared/eureka/etc/rc.local /etc/rc.d/
         chmod +x /etc/rc.d/rc.local

      5. [Optional] Comment out the line nvidia-cuda-mps-control -d in /etc/rc.d/rc.local to disable MPS on nodes running tensorflow on pre-Volta GPUs (e.g., eureka32 and eureka33). It is because stream callbacks are not supported on pre-Volta MPS clients; see https://docs.nvidia.com/deploy/pdf/CUDA_Multi_Process_Service_Overview.pdf.

      6. [Optional] Edit /etc/rc.d/rc.local to make cuda double GPU node could identify both GPUs
         

         export CUDA_VISIBLE_DEVICE=0,1
         nvidia-smi -i 0,1 -c EXELUSIVE_PROCESS
         

      7. Reboot

      8. Validate the following

         nvidia-smi -q | grep Persistence      # Enabled
         nvidia-smi -q | grep "Driver Version" # 440.33.01
         nvidia-smi -q | grep "CUDA Version"   # 10.2
         nvidia-smi -q | grep "Compute Mode"   # Exclusive_Process
         
         ps -ef | grep mps # nvidia-cuda-mps-control -d

4. NIS client

   1. Install

      yum -y install ypbind

   2. Configure

      setup # [Authentication configuration] -> [Use NIS]

         Domain: tumaz.gpucluster.calab
         IP: tumaz
      

      systemctl is-enabled ypbind # enabled

   3. Do not create the perl5 directory by setting PERL_HOMEDIR=0 in

      /etc/profile.d/perl-homedir.sh

   4. Reboot

   5. Validate the following

      yptest         # "1 tests failed"
                     # "Test 9: yp_all" should list all accounts
                     # "Test 3" may fail with the message "WARNING: No such key in map (Map passwd.byname, key nobody)"
      ypwhich        # tumaz.gpucluster.calab
      
      ls -l /home    # Show correct user and group names instead of just UID and GID
      
      # Log in using an existing user account

5. NTP client
   

   yum -y install ntp
   systemctl start ntpd
   systemctl enable ntpd
   systemctl is-enabled ntpd # enabled

6. TORQUE

   1. Install the required packages

      yum -y install tcl-devel tk-devel

   2. Install torque

      cd /work1/shared/eureka/package/torque/src/torque-3.0.6
      # WARNING: do NOT run "Fish_Install.sh" in parallel (i.e., install one node at a time)
      sh Fish_Install.sh >& log.eurekaXX
      cd ../../etc
      cp pbs /etc/init.d/
      # Login node only: edit "pbs.conf" to set "start_server=1" and "start_mom=0"
      cp pbs.conf /etc/
      cp nodes /var/spool/TORQUE/server_priv/nodes
      systemctl enable pbs
      source /etc/profile.d/torque.sh
      cd ../src/torque-3.0.6/
      ./torque.setup root
      killall pbs_server
      systemctl start pbs # This error message is fine: "LOG_ERROR::No such file or directory (2) in read_config, fstat: config"
      systemctl status pbs

   3. Check

      cat /var/spool/TORQUE/pbs_environment # LANG=en_US.utf-8

7. InfiniBand

   1. Install driver

      Follow Eureka Installation: InfiniBand --> ADAPTOR --> Install Driver
      

   2. [Ref] Hardware list

      Adaptor: MCX555A-ECAT
      ConnectX®-5 VPI adapter card, EDR IB (100Gb/s) and 100GbE,
      single-port QSFP28, PCIe3.0 x16, tall bracket
      

      Switch: MSB7800-ES2F
      Switch-IB(TM) 2 based EDR InfiniBand 1U Switch, 36 QSFP28
      ports, 2 Power Supplies (AC), x86 dual core, standard depth, P2C
      airflow, Rail Kit
      

8. Miscellaneous setup

   1. sensors

      yum -y install lm_sensors
      
      # sensors-detect --auto # skip this since it could be dangerous; see WARNING in “man sensors-detect”
      
      sensors # Verify that the reading of "temp1" under "k10temp-pci-XXXX" is normal (~50C)
              # Ref: https://hwmon.wiki.kernel.org/device_support_status_a_c
              #      https://www.guru3d.com/articles-pages/amd-ryzen-threadripper-2950x-review,8.html
              # I **guess** temp1 is Tctl and thus Tdie=temp1-offset, where offset=27C, is the correct temperature
              # to be compared with the CPU critical temperature

   2. gnuplot, htop

      yum -y install gnuplot htop

   3. ssh without password for the root

      cd /work1/shared/eureka/ssh_root/
      cp authorized_keys id_rsa* /root/.ssh/
      
      # Verification
      ssh eureka00   # "yes" to "continue connecting"
      ssh eurekaXX   # "yes" to "continue connecting"
      exit
      exit

   4. Intel compiler

      cd /opt
      ln -s /software/intel
      rm -rf ./rh # not related to Intel, actually

   5. ffmpeg

      rpm --import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro
      rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-5.el7.nux.noarch.rpm
      yum -y install ffmpeg ffmpeg-devel

   6. Time stamp of command history

      1. su
      2. Add export HISTTIMEFORMAT='%d/%m/%y %T ' to the end of file /etc/profile
      3. source /etc/profile
      4. Check by history

9. Python2 (Choose)
   *. [For current system] sh /work1/xuanshan/install-python-packages.sh *. [Original steps] (Not usable due to pip version is too old to be update.)

     # install packages
     sh /work1/shared/eureka/package/python/install-packages.sh
     # [Optional] If the installation of mpi4py fails (which somehow freezes the screen and causes failure of yt after reboot),
     # follow the steps below:
     pip uninstall mpi4py
     cd /usr/lib64/python2.7/site-packages
     rm -rf yt yt-3.5.1.dist-info
     pip install yt
     pip install mpi4py
   
     # check dependency
     pip check ipython numpy cython jupyter h5py scipy astropy matplotlib yt mpi4py # No broken requirements found.
   
     # check MPI
     su YOUR_ACCOUNT
     mpirun -n 16 python -m mpi4py.bench helloworld # Hello, World! I am process ? of 16 on eurekaXX.
   
     # check yt
     cd /tmp
     cp -r /work1/shared/eureka/test/yt ./test-yt
     cd test-yt
     sh test.sh # It should generate two PNG images Data_000000_Projection_z_density.png & Data_000001_Projection_z_density.png
     cd ..
     rm -rf test-yt
   
     # check HDF5
     python -c 'import h5py; print( "%s"%h5py.version.hdf5_version )' # 1.0.6
   

10. Python3

    1. cd /work1/shared/eureka/package/python3/Python3.9.10
    2. make clean
    3. ./configure --enable-optimizations --enable-shared
    4. make altinstall
    5. cp libpython3.9.so.1.0 /usr/lib64/
    6. Check python3.9
    7. Make link python3 to python3.9: rm /usr/bin/python3 ln -s /usr/local/bin/python3.9 /usr/bin/python3
    8. Update pip pip3 install -U pip
    9. Install necessary packages

       pip3 install --ignore-installed pyparsing
       pip3 install --upgrade dnspython
       pip3 install --upgrade pyudev
       pip3 install --ignore-installed --upgrade python-ldap
       pip3 install --upgrade ipython
       pip3 install --upgrade numpy
       pip3 install --upgrade cython
       pip3 install --upgrade six
       pip3 install --upgrade urllib3
       pip3 install idna==2.10
       pip3 install --upgrade certifi
       pip3 install chardet==4.0.0
       pip3 install --upgrade jupyter
       export HDF5_DIR=/software/hdf5/default
       pip3 install --no-binary=h5py h5py
       pip3 install --upgrade scipy
       pip3 install --upgrade astropy
       pip3 install --upgrade matplotlib
       pip3 install --upgrade gitpython
       pip3 install --upgrade pandas
       

    10. Install yt [Choose one]

        If option 1 failed, remove yt package and try option 2

        • [Option 1]
          1. Install: pip3 intsall -U yt
          2. Check: python3 -c "import yt"
        • [Option 2]
          1. cd /work1/shared/eureka/package/python3/yt
          2. pip3 install -e .
          3. Check: python3 -c "import yt"
        • Final check: yt version
    11. Install remain packages

        pip3 install mpi4py
        pip3 install dulwich
        pip3 install girder-client
        

    12. Install cmake

        yum -y install cmake3
        

11. Check

    1. NUMA

       lscpu | grep NUMA # It should show **2** NUMA nodes

    2. Memory frequency

       dmidecode -t memory | grep Speed # It should show **Speed: 2933 MT/s; Configured Memory Speed: 2933 MT/s**

    3. Queuing system and GAMER performance

       # log in to eureka00
       su your_account
       qsub -I -lnodes=eurekaXX:ppn=16
       cd /tmp
       cp -r /work1/shared/eureka/test/gamer/bin/template/ tmp-gamer
       cd tmp-gamer
       mpirun -map-by ppr:4:socket:pe=4 --report-bindings ./gamer
       awk 'NR>1 {print $7}' Record__Performance # Performance should be ~4.2e7
       cd ..
       rm -rf ./tmp-gamer/
       exit

    4. [Optional] CPU/GPU temperature

       /work1/shared/eureka/benchmark/temperature

    5. Malware scanner ClamAV

       1. Install ClamVA: yum install -y clamav clamd clamav-update
       2. Download and update ClamAV’s official virus signature databases: freshclam
       3. Scan directories: clamscan /etc/* /opt/* /boot/* /usr/* /tmp/* /sys/* /run/* /root/* /var/*
       4. Check result:

          ----------- SCAN SUMMARY -----------
          Engine version: 0.103.5
          Scanned directories: 5
          Scanned files: 2774
          Infected files: 0
          Data scanned: 565.18 MB
          Data read: 668.04 MB (ratio 0.85:1) 
          Time: 50.516 sec (0 m 50 s)
          Start Date: 2022:05:12 10:21:31
          End Date:   2022:05:12 10:22:22
          

          If Infected files is 0, then pass.

Ref:

• https://docs.clamav.net/Introduction.html
• https://openfind.freshdesk.com/support/solutions/articles/35000185768-centos-7-%E5%AE%89%E8%A3%9D-clamav-sop

Boot this machine with Ubuntu Live USB

1. Show all the disk found on this machine
   lsblk

2. Make the mount point

   mkdir /mnt/slash
   mkdir /mnt/boot
   mkdir /mnt/boot_efi #if the successful installed centos 7 has
   mkdir /mnt/usb

3. Mount centos 7 on the /mnt in Ubuntu Live USB

4. Mount the root , since it is LV(logical volume) , required more cared

   lvscan 			# scan lv are to be fount 
   vgchange -ay	# activate this lvm volume (if not activated) , so that we can mount
   lvscan
   mount /dev/centos/root /mnt/slash

5. Mount Centos 7 /boot and /boot/efi (if has) , on the /mnt in Ubuntu Live USB

   mount /dev/nvme0n1p1 /mnt/boot_efi
   mount /dev/nvme0n1p2 /mnt/boot

6. Clone each partition inside the mounted USB

   cd /mnt/usb
   mkdir slash
   mkdir boot
   mkdir boot_efi
   rsync -avHX /mnt/slash/ /mnt/usb/slash/		    		# Clone the root, may take a while
   rsync -avHX /mnt/boot_efi/ /mnt/usb/boot_efi/			# Clone the /boot/efi (if has)
   rsync -avHX /mnt/boot/ /mnt/usb/boot/					# Clone the /boot

7. Unmount the usb

   umount /mnt/usb			# Might take a while
   umount /mnt/slash
   umount /mnt/boot
   umount /mnt/boot_efi

(Boot with Ubuntu Live USB), NOT TESTED YET

1. Using GParted, to partition the (target) disk, should be the same as (source) disk

   boot
   boot efi
   centos
   swap
   

2. Mount the ssd to Ubuntu Live USB

   mkdir /mnt/slash
   mkdir /mnt/boot
   mkdir /mnt/boot_efi (if the successful installed centos 7 has)
   mkdir /mnt/usb
   
   mount ... # same steps as mounting the source , mount (target)ssd and the usb

3. Clone the OS for each partition from the usb to ssd

   rsync -avHX /mnt/usb/slash/ /mnt/slash/
   rsync -avHX /mnt/usb/boot/ /mnt/boot/
   rsync -avHX /mnt/usb/boot_efi/ /mnt/boot_efi/

4. Change the UUID in /etc/fstab file on the target machine

   ls -l /dev/disk/by-uuid				# ways to view the disk UUID
   blkid /dev/sdaX						# ways to view the disk UUID, but I think should change to 
    								"blkid /dev/centos/root" , since that's how we 
   mount it.
   vim /mnt/slash/etc/fstab			# Since we mount the centos root (ssd) at /mnt/slash/

5. Reinstall Grub

   mount --bind /proc /mnt/slash/proc   # Bind mount some system directories needed by grub
   mount --bind /sys  /mnt/slash/sys
   mount --bind /dev  /mnt/slash/dev
   mount --bind /run  /mnt/slash/run
   
   chroot /mnt/slash                    # Change the root environment, so that we can call grub install 
   inside                               # the migrated system (now, the target system)
   
   grub-install /dev/sda                # Probably should change to "grub-install /dev/centos/root" , since that's how we mount it.
   update-grub

6. Done, reboot the target machine!

• Further reference: https://www.wikihow.com/Create-a-Disk-Image-from-a-Linux-System-Using-Systemback

• Home Page
• System Installation