Network Summary - theunissenlab/lab-documentation GitHub Wiki

Network summary

Network diagram

What are tdrive and zdrive?

These are our remote data servers. They are both virtually mounted at /auto/tdrive/ on individual workstations (through autofs and ldap). tdrive has 13TB of space. zdrive was installed in late 2019 and has 48TB of space (upgradeable to 112TB if 4 more 16TB drives are added).

These servers are both physically located remotely in the colocation datacenter ("colo") located in Earl Warren Hall. Access and configuration notes

What is fhome?

Users home directories, physically stored on nashome and virtually mounted on individual workstations at /auto/fhome/. Don't store data in fhome because there isn't much space! Store data in tdrive instead!

How are the remote drives (tdrive, nashome/fhome, and zdrive) mapped?

zebra looks like it has the configuration (in /home/ubuntu/ldap/) that set up the automounting of these network drives. Each workstation in /etc/ldap.conf has configuration that lists uri ldap://ldap. Since ldap is an alias for zebra, it looks like it is somehow using the files on zebra, for example /home/ubuntu/ldap/change_fhome.ldif.

Setting up a new computer

  1. Assign the desired hostname to the computer
  2. Plug the computer into a ethernet port on our internal network (one of the 8 ports in LKS). finch assigns it an IP address.
  3. Copy the script activate-netlogin.sh from zebra (located at home/ubuntu/activate-netlogin.sh) onto the new computer and run it. Just take the default responses when prompted.
  4. Set up ssh by installing it with sudo apt-get install ssh and start it by sudo service sshd start
  5. Write this to the file /etc/lightdm/lightdm.conf.d/50-manual-login.conf:
[Seat:*]
greeter-show-manual-login=true
allow-guest=false

Software that you should probably install on all workstations

  • ssh

  • screen

  • htop

  • git

  • anaconda (if you use it)

  • cuda, cudnn if GPU (see below)

  • nvidia driver (if theres a graphics card)

  • python-tk

Nvidia driver, cuda, cudnn

sudo apt purge nvidia-*
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update
sudo apt install nvidia-384
  • Add LD_LIBRARY_PATH and CUDA_HOME to path (set in bashrc)
export CUDA_HOME=/usr/local/cuda
export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH
export PATH=$CUDA_HOME:$PATH

  • Install cuda and cudnn. This involves running a cuda install script and then adding cudnn files to /usr/local/cuda/lib64 and /usr/local/cuda/include

  • Install a working version of libgpuarray (http://deeplearning.net/software/libgpuarray/installation.html). I had to build it from source (clone and build) but you might be able to just use conda

Setting up a new user

New users are added by modifying the ldap entries on our ldap server zebra as ubuntu (super user).

  1. Add the new user. First login to zebra:

ssh zebra -l ubuntu

Second add the user, here newuser is going to be the user name and theunissen is the group:

sudo ldapadduser newuser theunissen

  1. Modify the user ldap entry so that home points to the correct place:
sudo ldapmodifyuser newuser
   changetype: modify
   replace: homeDirectory
   homeDirectory: /auto/fhome/newuser
   Ctrl D

  1. In the output of the previous step, find the uid and gid numbers (the uid should be unique to the user, and the gid is 702 for theunissen). These will be used in the next step.

  2. Login into nashome as root to make the new home directory and change its owner and group:

ssh root@nashome (ask "frederic" for password)

cd /raid/data/_NAS_NFS_Exports_/homes/fethome

mkdir newuser

chown uid newuser (uis is a number you got from ldap in 2 above)

chgrp gid newuser (gid is a number you got from ldap in 2 above - probably 702)

  1. Change the user password. Logout from nashome and log in as ubuntu user on zebra:

sudo ldapsetpasswd newuser

  1. Login from one of the desktops to see if it all works. You will also want to check that .bashrc file from one of the older users (e.g. /auto/fhome/fet/.bashrc) and take what you like from there and edit to make it your own.

Remote mounted drives (tdrive, zdrive, and fhome)

Read this section if you are debugging why you can't access a drive or if you are trying to add a new drive.

The remote mounting is configured on zebra. When connected to zebra, you can see a list of the drives configured for automount with the command ldapsearch.

To add a new entry, modify this template and create a new ldif file. Add the ldif entry using ldapadd. To do this, bind as root and use -W for a password prompt. The password is finchfinchfin. Use ldapsearch again to verify that the new entry is there.

Troubleshooting

Login loop after rebooting or /auto/fhome not accessible

  • Press Ctrl-Alt-F1 (or enter a shell). As the "ubuntu" user run the command sudo service autofs restart. This resolves a timing bug between when fhome is mounted and when the network becomes accessible. Ctrl-Alt-F7 returns you to the GUI.

  • You should be able to see fhome and tdrive now. (Check with ls /auto/fhome and ls /auto/tdrive). If not, you may need to rerun the activate-netlogin.sh script from above, in particular the line pam-auth-update. Then do sudo service autofs restart again.

  • If automount (autofs) is not working at all, in a pinch you can manually mount the necessary drives. If you might need to mkdir /auto/zdrive /auto/tdrive and /auto/fhome.

sudo mount -t nfs zdrive:/volume1/zdrive /auto/zdrive

sudo mount -t nfs tdrive:/mnt/NAS/tdrive/tdrive /auto/tdrive

sudo mount -t nfs nashome:/raid/data/_NAS_NFS_Exports_/homes/fethome /auto/fhome