Running Jupyterlab on Poseidon - USGS-CMG/usgs-cmg-portal GitHub Wiki

1. Login to Denali

   ssh denali.whoi.edu
   

2. Get off the master node:

   srun -p compute --ntasks-per-node=1 --mem=8G --time=1:00:00 --pty bash
   

3. Install Miniconda:

   wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
   bash Miniconda3-latest-Linux-x86_64.sh -b
   export PATH=$HOME/miniconda3/bin:$PATH
   conda activate base
   

4. Use conda-forge channel for all packages:

   conda config --add channels conda-forge --force
   conda config --remove channels defaults --force 
   

5. Install Mamba into base environment (conda equivalent written in C++, much faster at solving package requirements) and update base packages:

   conda install mamba
   mamba update --all
   

6. Install packages required to jupyterlab into base environment

   mamba install jupyterlab jupyter-server-proxy dask-labextension ipywidgets nb_conda_kernels nodejs
   

7. create password to login to jupyterlab

   jupyter server --generate-config
   jupyter server password
   

8. Create a custom science environment including ipykernel (include extra packages here):

   mamba create -n pangeo python=3.8 xarray h5netcdf dask dask-jobqueue ipywidgets hvplot geoviews datashader ipykernel \
                   pyepsg rioxarray metpy zarr rechunker intake intake-xarray
   

When you want to use Jupyterlab on Poseidon:

1. Login to Denali

   ssh denali.whoi.edu
   

2. Get off the master node:

   srun -p compute --ntasks-per-node=4 --mem=32G --time=3:00:00 --pty bash
   

3. Start jupyter with a start_jupyter scrip something like this:

   conda activate base
   cd /vortexfs1/usgs/rsignell
   JPORT=$(shuf -i 8400-9400 -n 1)
   echo "ssh -N -L 8889:`hostname`:$JPORT [email protected]"
   jupyter lab --no-browser --ip=`hostname` --port=$JPORT
   

4. Copy the printed out line starting with ssh -N -L 8889: to your clipboard and paste into a new terminal
5. Open a local browser to http://localhost:8889, and login to jupyterlab using the password you created above

Continue with the instructions here: https://pangeo.io/setup_guides/hpc.html#launch-dask-with-dask-jobqueue