Our course tutorials are available as docker images with pre-installed software and data. There are two interfaces you can launch and work with the exercises: JupyterLab or command terminal.

Login to the cloud server

You will be provided with the username and address to connect to a cloud server for running the exercises. To get on the server, you need to open up a shell command terminal on your computer and use ssh to connect to the server.

On MacOS and Linux, this is done using the terminal program and running

ssh student_##@agm#.statgen.us

where the # are the numbers provided to you for your account. After running this command, you'll be prompted for your password.

On Windows, using putty, connect to the host agm#.statgen.us, again where # have been provided; when you connect, you will then be prompted for your username and then your password.

You will then land in a shell environment ready to launch the exercises.

You can run the exercise either through JuptyerLab or through a command interface. Both options are discussed below. We recommend Option 1 when possible, but you can use Option 2 as plan B if the JupyterLab interface does not work as expected (typically due to low available memory in the cloud server).

Alternative to cloud server: use your own computer

We use docker to configure the tutorial software environment, and a script called statgen-setup, written in SoS language to initialize the docker containers for running the tutorials. If you use your own computer, e.g. a Mac or Linux laptop, you need to install docker and SoS, as well as the statgen-setup script.

Instructions below are tested on Mac and Linux computers. Currently we do not offer support to running under Windows, although it will still work for a Windows running the Windows Subsystem for Linux (WSL). Windows users can follow the WSL installation instructions to configure their Windows system, before returning to the installation instructions below.

To open up JupyterLab server on Windows, it is recommended that a modern web browser be used, eg Edge instead of Internet Explorer.

Install Docker

Please download the docker installer for your operating system, and follow the instructions from the download page to install Docker.

A few notes for Mac users:

1. After Docker is installed, please click on the Docker icon (found in your Applications) to turn on the Docker Engine. You will need to use your user account at https://hub.docker.com to login to your Docker client in order to access imaged released on dockerhub. If you do not have an account please register one and use it.
2. Every time you restart your machine you may need to restart Docker Engine manually if you would like to run the tutorials. You should see a whale-like icon on your task bar indicating that docker service is running on the background of your computer.
3. To test if Docker is installed properly and is ready to use, please open up a command terminal (found in your Applications) and type docker run hello-world. You should see greeting messages output on the screen, indicating successful installation.

Install SoS

SoS requires Python 3. If your computer already has Python 3 and has the pip module installed with Python 3, you may skip the next section on miniconda3.

Please open a command terminal on your computer. For Mac users please type "terminal" in your Application home page and click on the app that shows up.

Install Python 3 environment via miniconda3

We suggest installing Python 3 through the command line version of miniconda3. After you download the installation package, please open up your command terminal, navigate to the folder the installer is located (via cd command, eg cd ~/Downloads). You should find the installer in the folder under the name like Miniconda3-latest-MacOSX-x86_64.sh for Mac OS. The installation commands below assume a Mac installer but it should work similarly for Linux or Windows WSL users:

chmod +x Miniconda3-latest-MacOSX-x86_64.sh
./Miniconda3-latest-MacOSX-x86_64.sh

Follow the prompts to install. After installation is complete, you need to open up a new terminal window for the new conda environment to take effect. You can type in the command terminal which conda to verify the installation. You should see an output of the path at which miniconda3 is installed.

Install SoS via pip

The pip module for Python 3 should be available if you have installed miniconda3 or other equivalent packages. To install SoS, please type in command terminal pip install sos to download the software from the Internet and install it. To verify the installation please type sos -h you should see valid output showing the command interface of the sos program.

Install statgen-setup script

Please download this statgen-set script, save it to your computer with filename statgen-setup. Then please open your command terminal, use cd command to navigate to where the file is downloaded and saved to (on Mac OS it should be ~/Downloads by default), and run

chmod +x statgen-setup

to make this script executable. You should now be able to run this script in the command terminal as ./statgen-setup from the directory it is downloaded to. Please test it by typing ./statgen-setup -h to output the help information for this script.

You can also move this script to specific folders in your system (bash PATH) such that you will be able to run it simply as statgen-setup without having to type in the path e.g. ./. One possibility is to install it to where sos program is installed. To do so, first type which sos to see the path where sos is installed to. Then you can move statgen-setup script to that same path (either via mv command on the terminal, or cut and paste it through the file manager in your operating system).

Option 1: Launch exercise in JupyterLab

This is the only option for exercises written in IPython Notebooks (although JupyterLab also offers shell terminals that you can use to run command based tutorials). Currently available tutorials based on Ipython Notebooks are:

• finemap
• ldpred2
• vat
• pseq
• regenie

To launch, for example finemap exercise in JupyterLab, run from the command terminal:

statgen-setup launch --tutorial finemap

Or, e.g., ~/Downloads/statgen-setup launch --tutorial finemap if statgen-setup script is located in ~/Dowlnoads and not is installed to the system's PATH (and thus cannot be accessed via simply running statgen-setup).

When it's done, you should see a line printed on the screen that contains a URL:

In this example it is http://207.246.125.18/student_12/finemap. Please copy that URL to your web browser. Your JupyterLab server should start like this:

The left panel of the JupyterLab displays all files available. If you see IPython notebooks (.ipynb files) displayed, you can click on them to launch and run the tutorial directly from these notebooks. Otherwise, please click on the PDF documents (.pdf files) to show them in the browser, and follow the instructions to run them in the terminal console that you can launch from JupyterLab. In some tutorials you will see a README file that prompts you to open a terminal console from JupyterLab and run get-data to download and deploy the tutorial material. It may take a while for get-data command to load the data. Please wait to start the tutorials until after the command is complete.

How to launch terminal console from JupyterLab:

Click on File--> New --> Terminal

Then your Terminal console should be launched:

Convert & Save tutorials to MS Word format

We have the server configured to make it possible to export a notebook as docx file (MS Word format) to your local computer. To do this, simply click on File button as shown above, then Export Notebook As -> Export Notebook to Docx. Notice that PDF format export is currently not supported (requires xelatex which we don't install in our server).

A note for course users on our cloud server

We terminate the JupyterLab container instances periodically to maintain the cloud server at a reasonable load. Currently this is configured to happen daily at midnight. The files generated from the tutorials will still be saved to the hard drive, but data in the memory will be lost. This includes the bash environments and R variables for relevant exercises. You can always resume your previous analysis with a new container instance, but you'll have to reset some of these variables. For example for ldpred2 exercise, the variable:

work_dir="height_results"

at the beginning of the exercise needs to be reset every time the JupyterLab instance restarts.

Option 2: run exercise from command shell

In addition to tutorials available from JupyterLab as described above, the following tutorials are available from command terminal:

• annovar
• epistasis
• fastlmm-gcta
• gemini
• plink
• mlink
• pleiotropy
• nps
• popgen
• regression
• slink

To run, for example finemap exercise from command terminal, please run:

statgen-setup login --tutorial finemap

You will be in a command environment. Please navigate to work directory by typing

cd work

If you do not see data files under this directory, that means the data is too large to warrant preloading them to the docker image. Please run get-data command to download them (it may take a while). You can then start working on the tutorials now by copy-paste text from handouts (PDF, HTML or Ipython notebook format), or by typing in the commands directly.

When you are done with the tutorial just type exit to exit. If you log in again, your session should resume unless the container has been terminated externally. If you want to fresh restart the container you can add --restart switch to the command above.

Notice that work folder is mounted to the physical hard drive of the computer system as a folder under your HOME directory. Only the files under this work folder will be saved to your computer. Files outside work will be temporarily saved to the docker container instance and will be lost if you terminate or restart the docker container.

For shared Linux account

If you share a Linux account with other students, you need to add --my-name to the launch commands:

statgen-setup launch --tutorial <tutorial> --my-name <my-name>
statgen-setup login --tutorial <tutorial> --my-name <my-name>

where <tutorial> is one of the available tutorials. <my-name> is any (unique) identification of your choice that should not conflict with the choice of another user if you share a computer. If you have your unique Linux user accounts, you can leave it unspecified.

Video illustrations

To facilitate users setting up their computer for the course exercises, we have prepared video instructions available on YouTube.

Install WSL2, Docker, Conda and SoS on Windows

Install Docker, Conda and SoS on MacOS

Launch tutorials

Get help

Please open a ticket in our issue tracker if you have any difficulty setting up your system: https://github.com/statgenetics/statgen-courses/issues We will help you trouble-shoot.