JupyterHub Configuration - wanglab-neuro/jupyterlab_server GitHub Wiki

Example deployments

Simplest deployment for a small amount of users (0-100) and single server
https://github.com/jupyterhub/the-littlest-jupyterhub
MIT CSAIL (with MIT OIDC OAuth - MIT's OAuth2/OpenID service https://oidc.mit.edu/) (2016)
https://github.com/gifford-lab/jupyterhub
Dockerized JupyterHub server setup (used for MIT 16.32 Principles of Optimal Control and Estimation - with Hash authentication)
https://gitlab.com/mit-acl/lab/server-jupyterhub
Medium-scale deployment with Docker Compose (2018)
https://opendreamkit.org/2018/10/17/jupyterhub-docker/
Deploying JupyterHub for Education (2015)
https://docs.rackspace.com/blog/deploying-jupyterhub-for-education/
KWARC (with discussion about why they didn't integrate SSO)
https://kwarc.info/teaching/jupyter-documentation.pdf

Request hostname and static IP

Request a Hostname/IP Address reservation
Obtain an SSL certificate for a web server http://kb.mit.edu/confluence/x/x487 Create req on Ubuntu sudo openssl genrsa -out /etc/ssl/private/neuro-wang-serv1.mit.edu.key 2048 sudo openssl req -new -key /etc/ssl/private/neuro-wang-serv1.mit.edu.key -out /etc/ssl/reqs/neuro-wang-serv1-2021.req cd /etc/ssl/reqs cp neuro-wang-serv1-2021.req /mnt/d/

If Jupyterhub is running on a container, the recommended method is to mount the directory where the keys are as a volume, or use a proxy. Alternatively, the key and certificate may be copied to the container before adding to the config file:

   docker cp servername.key jhub_wl:/etc/ssl/private/
   docker cp servername.cer jhub_wl:/etc/ssl/certs/
   cd /etc/jupyterhub   
   nano jupyterhub_config.py 
   c.JupyterHub.ssl_key = '/etc/ssl/private/servername.key'
   c.JupyterHub.ssl_cert = '/etc/ssl/certs/servername.cer'

To avoid having to copy and change config after each "run" call, modify the Dockerfile (e.g., VOLUME /uploads), or pass key as arguments when starting jupyterhub: jupyterhub --ip 10.0.1.2 --port 443 --ssl-key my_ssl.key --ssl-cert my_ssl.cert.
Just make sure never to include keys or passwords in shared repositories or images (public or not). Use secret files / folders.

On Windows, create Inbound rule on OS's firewall to allow HTTPS connection:
Windows > Advanced Settings > Create new Inbound rule: HTTPS / port 443

Finally add https:// to the server's address. Once SSL Connection is enable, the unsecure address will not work.
http://servername:8000/hub/login should now be: https://servername:8000/hub/login.

Mount a volume

Use -v or -mount. See caveats: https://docs.docker.com/desktop/windows/wsl/#best-practices For example docker run -d --name jhub_wl -v J:\shared:/home/admin/files -p 8000:8000 jhub_wl:ssl docker run -d --name jhub_conda -v /etc/ssl/:/etc/ssl/ -p 8000:8000 jhub_conda:latest Here the files will be accessible in Jupyterlab in /user/username/lab/tree/files

Bootstrapping users

https://github.com/jupyterhub/jupyterhub/tree/main/examples/bootstrap-script Be aware that if running Jupyterhub on a container, pre spawn hook commands are executed on that container, not the host system (e.g., new user, directory, etc).

New user work directories will not be writable if mounted from existing host directories as volume as root. See this comment: https://github.com/jupyterhub/dockerspawner/issues/160#issuecomment-330162308 Here's how to fix it. Assuming the "work" folder is mounted as the notebook directory in with c.DockerSpawner.notebook_dir = notebook_dir c.DockerSpawner.volumes = { '/home/{username}/work': notebook_dir } On host machine: NBDIR=/home/newuser/work/ sudo chown :100 "$NBDIR" sudo chmod g+rws "$NBDIR" sudo setfacl -d -m g::rwx "$NBDIR"

Instead of doing this manually, set the spawner environment. e.g.: c.DockerSpawner.environment = { "CHOWN_HOME": "yes", "CHOWN_EXTRA": "/home/jovyan", "CHOWN_HOME_OPTS": "-R", "NB_UID": 1000, "NB_GID": 1000, }

See 3.3.2 Docker Options https://readthedocs.org/projects/jupyter-docker-stacks/downloads/pdf/latest/ https://discourse.jupyter.org/t/dockerspawner-and-volumes-from-host/7008/6 How to configure the NB_UID using DockerSpawner with jupyterhub-singleuser image: https://groups.google.com/g/jupyter/c/-VJXHy5hnfM

Share data with your users

https://tljh.jupyter.org/en/latest/howto/content/share-data.html

Warning: docker comes along with root it doesnt require sudo. So if sudo is needed, install it.
Add it to Dockerfile or login (e.g., docker exec -it jhub_wl bash) and run:

	apt-get update && \
	      apt-get -y install sudo

If files mounted in e.g., /home/admin/files, create symlink
ln -s /home/admin/files /srv/data/shared else, mkdir -p /srv/shared to just create that directory.
Then go to user directory and link that folder:

	cd /etc/skel/
	ln -s /srv/shared shared

This works right away for new users. Existing users don't see the folder until rebooting server.

How do I share links to notebooks?

https://jupyterhub.readthedocs.io/en/stable/getting-started/faq.html#how-do-i-share-links-to-notebooks

Group utilities

creating groups: https://www.techrepublic.com/article/how-to-create-users-and-groups-in-linux-from-the-command-line/ https://github.com/gutow/tljh_grp_utils listing groups all groups: getent group specific group: getent group | grep hub_jhub_users deleting group: groupdel hub_jhub_users

Then:
create group: sudo ./mkhubgrp jhub_users
populate: sudo gpasswd -M jupyter-userA,jupyter-userB,jupyter-userC users
create a link: sudo ./lnhubgrp users

Or, for DockerSpawner:
on host, create shared folder, create user group, and change permissions accordingly mkdir -p /data/shared
sudo groupadd jhub_users
sudo chown root:jhub_users /data/shared

sudo chmod g+s /data/shared
Then add each user to group and mount directory as volume in c.DockerSpawner.volumes
'/data/shared': home_dir + '/shared'

Conda environments

Create environment, then add kernel.
https://softwarejargon.com/jupyterlab-and-conda-environment-installation-and-setup/
https://www.python-engineer.com/posts/setup-jupyter-notebook-in-conda-environment/
pip install ipykernel or conda isntall ipykernel For env called jmatlab:
-> simple way to add kernel: python kernel install --name=jmatlab
-> official ipython doc: python -m ipykernel install --user --name jmatlab --display-name "Python (jmatlab)"

Single user multiple containers

See DockerSpawner doc / https://github.com/jupyterhub/dockerspawner/issues/257 / phockett_jupyterhub-docker - jupyter branch
c.JupyterHub.allow_named_servers=True
c.JupyterHub.spawner_class = 'dockerspawner.DockerSpawner'
c.SystemUserSpawner.name_template = '{prefix}-{username}-{servername}'
c.SystemUserSpawner.image_whitelist = ['jupyter/scipy-notebook', 'jupyter/tensorflow-notebook']

Using GPU

https://docs.docker.com/compose/gpu-support/
https://www.dlm.med.fau.de/setting-jupyterhub-deep-learning/
https://tustunkok.github.io/tutorial/notes-to-myself/vps/2020/05/16/how-to-create-a-gpu-powered-containerized-multi-user-jupyterhub-research-server.html

For containers running on WSL, see WSL wiki page for GPU configuration.
DockerSpawner requires extra methods equivalent to the docker run --gpus all
https://github.com/jupyterhub/dockerspawner/issues/244

Running Matlab in Jupyterlab

The Matlab install should be in the container.

Install MATLAB Engine API for Python:
https://www.mathworks.com/help/matlab/matlab_external/install-the-matlab-engine-for-python.html

On container: install Matlab kernel for Jupyter (https://github.com/Calysto/matlab_kernel)
Create environment (see e.g., http://www.jmlilly.net/jupyter-matlab)
conda create -vv -n jmatlab python=3.9 jupyter, activate, then pip install matlab_kernel

Updates and backups

https://jupyterhub.readthedocs.io/en/stable/admin/upgrading.html https://jupyterhub.readthedocs.io/en/0.7.2/upgrading.html https://blog.jupyter.org/jupyterhub-0-8-19b00b75e3df https://github.com/jupyterhub/jupyterhub-deploy-docker#how-can-i-backup-a-users-notebook-directory

Backup db
From host terminal, root directory (create backups folder if needed):
docker cp jupyterhub:/srv/jupyterhub/jupyterhub.sqlite ./backups/jupyterhub-backup-$(date +%Y-%m-%d).sqlite
Can be done also from container itself, but if jupyterhub_jupyterhub_data volume is deleted, that backup will go too.
docker exec -it jupyterhub bash #Enter container
cp -v jupyterhub.sqlite jupyterhub-backup-$(date +%Y-%m-%d).sqlite # Make copy

Reset db
If jupyterhub is already running: docker stop jupyterhub
Otherwise: 'docker-compose up --no-start', if using docker compose, or docker create jupyterhub
docker cp ./backups/jupyterhub-backup-...-.sqlite jupyterhub:/srv/jupyterhub/jupyterhub.sqlite
docker start jupyterhub