Lab Network architecture

Important features of our lab network:

1. Centralized user accounts, so your lab login can be used on any workstation

2. Centralized home directories /auto/fhome/, so they are accessible from any workstation

3. Shared storage drives with large capacity, tdrive (13TB) and zdrive (48TB), that are automatically mounted on each workstation at (/auto/tdrive and /auto/zdrive)

4. A VPN server so that you can access lab computers and data while you are working from home

5. An internal DNS service so that workstations and servers on the lab network can be addressed by their hostnames (e.g. zdrive, tdrive, zebra, finch, strfinator, corvus, aquila, tucana, ...)

Instructions for how to do administrative tasks or troubleshooting can be found on the Network Administration page.

I. Network Diagram

To update this diagram (lucid chart) please go under lucid

II. All Lab Computers and Locations

Host	WS*	Physical location	External IP/address	Internal IP	Description
zebra		data center	169.229.219.171	10.0.1.11	Connected directly to tdrive and zdriveVirtually connected to finch with IPsecOpenVPN serverLDAP Server
finch		LKS	169.229.146.178fet.neuro.berkeley.edu	10.0.2.1	Runs dhcp (assigns ip addresses to device mac addresses)Runs dns (maps host names to ip addresses)Virtually connected to zebra with IPsecComputers can look up other computers on the network by hostname using finch
tdrivezdrive		data center		10.0.1.20410.0.1.203	Data storage (tdrive 13TB, zdrive 48TB)
nashome		LKS		10.0.2.3	Raid0 SSD fileserveruser = ubuntuNetwork attached storage (NAS)Stores /auto/fhome home directories
aquila, corvus, tucana	x	LKS		10.0.2.xhostname.fet.berkeley.edu	64GB RAM, 1080Ti GPU
goats, strfinator	x	LKS		10.0.2.x	32GB RAM
sirius	x	LKS		10.0.2.x	8GB RAM
della17ec9		LKS			Is a printer, settings here.Connect by USBWe once had wireless printing configured but it hasn't worked in a while.
plump	x	125 NAF			Connected to the rest of the network via ssh Runs pecking test
kiwi	x	131 NAF			Connected to the rest of the network via vpn Runs OperantEphys
chubbyninja	x	125 NAF			Not connected to the rest of the network Runs pecking test
pumpkin	x	125 NAF			Not connected to the rest of the network Runs pecking test

* WS: workstation

III. Detailed Server Information

finch (10.0.2.1)

VPN Server (openvpn)

zebra runs an OpenVPN server. The primary configuration file is located at /etc/openvpn/outside.conf - it sets the certificates and keys used for VPN authentication and encryption, as well as defining finch (fet.neuro.berkeley.edu) as the DNS server address for client connections (push "dhcp-option DNS 10.0.2.1").

LDAP Server (slapd)

zebra runs the LDAP server as well which manages user account credentials, as well as automount information for shared drives. The LDAP commands are stored in .ldif files located at /home/ubuntu/ldap/*.ldif. slapd is the name of the actual program running the ldap server.

Each computer on the network (including zebra itself) has a configuration file at /etc/ldap/ldap.conf, which defines their ldap server as uri ldap://ldap - its a little confusing, but the second ldap in that line refers to the server zebra (somehow finch, the DNS server, maps the hostname ldap to zebra's IP).

Run ldapsearch -x to view LDAP rules for users and automounts. (The -x option avoids the password prompt. I don't know whose password it refers to and why.)

To add a new ldap entry, learn how to create an ldif file or base it off of existing ones in /home/ubuntu/ldap/. Use ldapadd to add this new entry (you may need to use the options -x or -W for authentication purposes).

DHCP Server (isc-dhcp-server)

DHCP server (maps the mac addresses of computers connected to the network to their IP addresses)

DNS Server (Bind9)

DNS server (maps IP addresses of computers to their hostnames, routes requests to those hostnames to the correct IP).

Connecting to finch

Connect to finch by SSH with user:ubuntu. Your ssh key should be added to finch's authorized keys. As of Nov 2020 I don't know whose keys are authorized, so its possible we don't know how to connect.

nashome (10.0.2.3)

NAS server storing all our home directories. Runs Thecus NAS software.

Browser interface

There is a management interface in the browser (just go to 10.0.2.3 or nashome.fet.berkeley.edu in your browser when on the lab network). But I don't know the password! Its possible the password can be reset on the physical machine in LKS.

Connect

Connect to nashome using ssh with ssh root@nashome. pass finchfinchfinch.

tdrive (10.0.1.203)

Connect in browser

I believe that tdrive is running Synology software but I am not sure how to connect to it. It is possible you need to go to the colo and physically connect to it with ethernet, then use the Synology Assistant software to open it in the browser.

zdrive (10.0.1.204)

NAS server for our data with 48 TB storage capacity. Runs Synology NAS software.

Connect in browser

You can connect to the Synology Assistant NAS software running on zdrive when connected to the VPN at https://zdrive.fet.berkeley.edu:5001. User theunissen, pass finchfinchfinch.

zdrive's configuration (Synology)

Here are all the notes on how zdrive is configured for remote access. These are just the things that we actively changed from the default settings. Note that we didn't really understand a lot of what these settings meant so they could be wrong or missing important things.

* Storage Manager > Storage Pool
  * Created storage pool (Storage Pool 1) with RAID type 5.

* Storage Manager > Volume
  * Created Volume 1 on Storage Pool 1 with **btrfs** file system

* Control Panel > Connectivity > Network > General
  * Server Name: zdrive
  * Default Gateway: 10.0.1.1 (LAN 1)
  * Preferred DNS Server: 10.0.2.1

* Control Panel > Connectivity > Network > Network Interface > LAN 1 > IPv4
  *  Use manual configuration:
     * IP Address: 10.0.1.204
     * Subnet mask: 255.255.0.0
     * Gateway: 10.0.1.1

* Control Panel > Connectivity > Network > Static Route
  * Created static route with following settings (this is needed so that all workstations can access the server)
    * Network Destination: 10.0.2.0
    * Netmask: 255.255.255.0
    * Gateway: 10.0.1.1
    * Interface: LAN 1

* Control Panel > File Sharing > SMB/AFP/NFS
  * Enable NFS is checked as well as NFSv4.1 support
  * In advanced settings, default UNIX permissions is checked

* Control Panel > File Sharing > Domain/LDAP > LDAP
  * Enable LDAP Client (Lets the system know about our network's users and groups)
    * LDAP Server Address: 10.0.1.11 (this is zebra)
    * Base DN: dc=fet,dc=local

* Control Panel > File Sharing > Shared Folder
  * Created shared drive called "zdrive" on Volume 1
  * Under the folder's NFS Permissions, created the following access rule:
    * Client: *
    * Privilege: Read/Write
    * Squash: No mapping (don't map users to root, did that by accident at first)
    * Asynchronous: Yes
    * Cross-mount: Allowed

* File Station > zdrive/zdrive > Permission
  * Created two rules at the top level directory (/volume1/zdrive) that allow the owner (creator) of subdirectories to have full ownership and read/write access to their own folder, and read access on all other user's folders.
    * Read access for `[email protected]` (everyone can read data from anyone else's directories)
    * Full Control for `Owner` (users can do anything they want on their own subdirectories)

Raid Arrays in LKS (condor, osprey)

Two large raid arrays exist in LKS, accessible on the 10.0.2.X subnetwork, can be mounted using sudo mount -t nfs condor:/mnt/nfs_share /mnt/condor. They were setup according to Condor and Osprey Setup

Other workstations (aquila, corvus, tucana, strfinator, goats, ...) (10.0.2.x)

Shared drive automounting (autofs-ldap)

Our workstations automatically mount directories by looking to the configured ldap server (defined in /etc/ldap.conf). The ldap server ldap currently points to 10.0.1.11 which is zebra. See Troubleshooting for how to manually mount these drives.

Connecting to a workstation

Connect to a workstation while on the lab network with SSH using its hostname, e.g. ssh USERNAME@corvus, or with its full domain name e.g. ssh [email protected].

IV. Things To Do

• Upgrade servers and workstations to Ubuntu 20.04 LTS from 16.04 LTS (Long term support (LTS) for 16.04 ends in April 2021) (-kevin)

• Create images of zebra and finch so that we can recover their setup and configuration if something goes wrong (-kevin)

• Fill in links in this wiki page (-kevin)

• Try editing the zdrive permission so users can create and write to anyone's directories? (-kevin)

V. History

The lab network was set up by James Gao (Gallant Lab) back in 2017 during our transition from Tolman Hall to Li Ka Shing. aquila, corvus, and tucana were built in 2017 after the move to LKS. zdrive was added to expand our storage capacity in 2019 when tdrive was running out of space.

Here is the lab meeting by James Gao for a thorough explanation of Gallant lab infrastructure

James on GLab network infrastructure (on bdrive)

VI. Fail2ban

On finch, fail2ban is setup to prevent unauthorized access. If someone gets banned, first confirm their IP is banned by checking: sudo fail2ban-client status sshd then unban the specific IP with sudo fail2ban-client set sshd unbanip IP_ADDRESS