NOMAD Server Infrastructure Notes - aichemy-hub/docs GitHub Wiki

NOMAD Server Infrastructure Notes

General Setup

Each client computer has to be running the NOMAD client, found on https://github.com/nomad-nmr/nomad-spect-client. The NOMAD client is responsible for getting instructions from the server and writing input files which iconNMR then executes.

When an NMR experiment is executed, the data will be copied onto the NOMAD server and available to users on the college network via http://aichemy-nmr.ch.ic.ac.uk.

Network Connectivity

The computers in the lab are disconnected from the college network but are connected to each other through a local lab network. There is one computer in the lab serv-10 which is both on the lab network and on the college network and internet.

The NOMAD system consists of the following components:

In order for NOMAD to work the client computers must be able to reach the server via the college network. This is achieved by creating a HTTP proxy on serv-10, via caddy <https://caddyserver.com/docs>. When the client needs to talk to the NOMAD server, it talks sends an HTTP request to serv-10. serv-10 has caddy running on it (on port 80) and it will send this request to the server. It will then send the response back to the client. Check the debug guide <./nomad-debug-guide.rst> for more information on how to check the caddy proxy is working properly.

nomad-client service

In /etc/systemd/system/nomad-client.service:

[Unit]
Description=Nomad Client

[Service]
ExecStart=/home/nmrsu/bin/nomad-client

[Install]
WantedBy=multi-user.target

Then in ~/bin/nomad-client:

#!/usr/bin/env bash

npm run verbose --prefix /opt/nomad-spect-client/

To start, stop, view status:

sudo systemctl start nomad-client
sudo systemctl stop nomad-client
sudo systemctl status nomad-client

To make sure it will run when the system reboots

sudo systemctl enable nomad-client

[!IMPORTANT] There is no need to manually run nomad-client from the command line. This is now all handled through the service set up above.

To view the logs, run:

sudo journalctl -u nomad-client

unofficial node builds

We're running CentOS7 on instrument machines, which isn't compatible with official recent builds of node (>v18) and npm. In order to install later version of nomad-client (>3.4.0) we had to use an unofficial build of node. See this issue for details: https://github.com/aichemy-hub/docs/issues/30

Doku disk monitoring

Doku is running at http://aichemy-nmr.ch.ic.ac.uk:9090/. It can be used to monitor disk usage of Docker volumes and bind mounts.

The Dockerfile is on the server at /doku-dashboard/.

TIG (Telegraf/InfluxDB/Grafana) stack

A TIG stack has been set up to monitor disk usage over time. There is a public Grafana dashboard (internal to Imperial) showing disk usage, RAM and CPU over time.

Disk usage of over 85% should trigger a Slack message in the #alerts channel of the aichemy Slack workspace.

The Dockerfile and other config files are on the server at /tig-stack/.