Module 2.1 project set up, conda, fastqc, multiqc - mestato/EPP622_2026 GitHub Wiki

Groups of Students

Each group has 10 files and a fun easter egg courtesy of Zane.

Log in to sphinx and navigate to our class project directory. Project directories have much more space than home directories.

ssh <yourusername>@sphinx.ag.utk.edu
cd /lustre/isaac24/proj/UTK0505/

# If you do not have access to the course directory, but have access to sphinx, you can access the data here:
cd /pickett_sphinx/teaching/EPP622_2026

Create a folder to hold your lab work and lets start building some organizational structure.

mkdir <yourusername>
cd <yourusername>
mkdir raw_data
mkdir analyses
cd raw_data

Copy the ten files you have been assigned into your raw_data folder.

for file in /lustre/isaac24/proj/UTK0505/module2_data/lab_data/*fastq.gz
do
	echo $file
	cp $file .
done

Make your raw data read only

Learn about permissions

chmod u-w *gz
cd ..
chmod u-w raw_data

Now try to delete. It should fail.

OR better yet, symbolically link the files. Important points about symbolic links:

• Use the -s flag every time or bad things can happen
• Use ls to determine both which "files" are actually symbolic links and where they are pointing

Open up permissions, delete the copies, make symbolic links.

cd raw_data
cp copy_data.sh link_data.sh

Modify link_data.sh

for file in /lustre/isaac24/proj/UTK0505/module2_data/lab_data/*fastq.gz
do
	echo $file
	ln -s $file .
done

Now run it

bash link_data.sh
ls -la

If you already have a working anaconda, miniconda, or mamba installation, skip the install

First, make sure your SCRATCHDIR variable is set to the isaac24 scratch location in your ~/.bashrc. Using nano or the text editor of your choice, add these lines to your ~/.bashrc file (at the end or anywhere is fine):

# add to your ~/.bashrc
export SCRATCHDIR="/lustre/isaac24/scratch/<your_netid>"

Then run this on the command line (do NOT put in your .bashrc):

source ~/.bashrc
echo $SCRATCHDIR

Don't try to move on with this lab until that last command - echo $SCRATCHDIR - returns /lustre/isaac24/scratch/<your username here>

Now install miniconda with its files living in scratch instead of your home directory.

mkdir -p $SCRATCHDIR/miniconda3
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O $SCRATCHDIR/miniconda3/miniconda.sh
bash $SCRATCHDIR/miniconda3/miniconda.sh -b -u -p $SCRATCHDIR/miniconda3
rm $SCRATCHDIR/miniconda3/miniconda.sh
source $SCRATCHDIR/miniconda3/bin/activate
conda config --add channels defaults
conda config --add channels bioconda
conda config --add channels conda-forge
$SCRATCHDIR/miniconda3/bin/conda init
source ~/.bashrc

Close and reopen a new terminal. Check that everything is working with

conda list

Getting Started with Conda

Create a new environment (may need to answer "a" for accept and "y" for yes to a few things)

conda create --name snps 

List all environments:

conda env list

Activate an environment

conda activate snps

Deactivate the current environment (then reactivate so we can do installs)

conda deactivate
conda activate snps

Install packages (will need to answer "y" for yes to the install and this will take a few minutes)

conda install bioconda::fastqc bioconda::multiqc bioconda::bwa-mem2 bioconda::fastp bioconda::samtools 

List installed packages in the active environment

conda list

It is important to look at the quality of our reads. Lets use Fastqc for this.

cd /lustre/isaac24/proj/UTK0505/<yourusername>/analyses
mkdir 01_fastqc
cd 01_fastqc

Lets pick one file name to run this on

which fastqc
mkdir SAMPLENAME
fastqc ../../raw_data/YOURFILE.fastq.gz -o SAMPLENAME/

The output is html, which provides nice interactive features, but also needs to be copied to your laptop to see in a web browser.

scp [email protected]:/lustre/isaac24/proj/UTK0505/mstaton1/analyses/01_fastqc/SAMPLENAME/\*html .

Now time to do the rest - with a for loops. First, lets get symbolic links, which will also mean we don't need to specify an output directory.

ln -s ../../raw_data/*fastq.gz .
ls -l

And our bash script. We'll ask fastqc to use two threads to make it slightly faster.

for file in *fastq.gz
do
    fastqc -t 2 $file
done

We could transfer all these over and look at them one by one. But a better option is...

Its so easy!

multiqc .

Now lets take a look at the summary stats by scp'ing them to our laptops.

scp [email protected]:/lustre/isaac24/proj/UTK0505/mstaton1/analyses/01_fastqc/multiqc_report.html .

1. First, conda is written in python, which is comparatively slow to lower-level languages like C++. Luckily, someone wrote a nice C++ module to help speed up conda called: mamba.

To install mamba:

conda install mamba

2. Loading conda on login can drastically slow down server login speed. To avoid this, you should comment out the text from your initial conda installation in your .bashrc file by adding # in front of each line. This is what I commented out:

## >>> conda initialize >>>
## !! Contents within this block are managed by 'conda init' !!
#__conda_setup="$('/lustre/isaac24/scratch/mstaton1/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
#if [ $? -eq 0 ]; then
#    eval "$__conda_setup"
#else
#    if [ -f "/lustre/isaac24/scratch/mstaton1/miniconda3/etc/profile.d/conda.sh" ]; then
#        . "/lustre/isaac24/scratch/mstaton1/miniconda3/etc/profile.d/conda.sh"
#    else
#        export PATH="/lustre/isaac24/scratch/mstaton1/miniconda3/bin:$PATH"
#    fi
#fi
#unset __conda_setup
## <<< conda initialize <<<

Then add these lines:

# Lazy Conda/Mamba: init only when first invoked, keeps logins fast
export MAMBA_ROOT_PREFIX="/lustre/isaac24/scratch/$USER/miniconda3"

__lazy_conda_init() {
    __conda_setup="$('/lustre/isaac24/scratch/$USER/miniconda3/bin/conda' 'shell.bash' 'hook' 2>/dev/null)"
    if [ $? -eq 0 ]; then
        eval "$__conda_setup"
    else
        if [ -f "/lustre/isaac24/scratch/$USER/miniconda3/etc/profile.d/conda.sh" ]; then
            . "/lustre/isaac24/scratch/$USER/miniconda3/etc/profile.d/conda.sh"
        fi
    fi
    unset __conda_setup

    if [ -f "/lustre/isaac24/scratch/$USER/miniconda3/etc/profile.d/mamba.sh" ]; then
        . "/lustre/isaac24/scratch/$USER/miniconda3/etc/profile.d/mamba.sh"
    fi
}

conda() {
    __lazy_conda_init
    conda "$@"
}

mamba() {
    __lazy_conda_init
    mamba "$@"
}

Now, you will no longer see "(base)" when you log on, and you will run conda activate or conda activate <environment_name> to load conda when you need it.

3. Next you can configure your .condarc file, which you can think of as your "settings" for conda. (Note: .condarc lives in your home directory, even if you install miniconda in your scratch directory!) Your conda settings automatically extend to mamba.

To access your .condarc file:

nano ~/.condarc

4. Now, paste the following in your .condarc, then save and exit.

channels:
  - conda-forge
  - bioconda
channel_priority: strict
allow_non_channel_urls: true
solver: libmamba
auto_activate: true

• channels: \ - conda-forge \ - bioconda: Adds the download channels: conda-forge (for most default dependencies) and bioconda (most biology dependencies/specialized software). You can add - defaults below the other two channels if your conda cannot find certain dependencies, but this is unlikely.
• channel_priority: strict : Speeds up solve time by pulling base dependencies from conda-forge first, then bioconda.
• allow_non-channel_urls: true : Allows installation from custom sources.
• solver: libmamba : Drastically reduces environment solve (installation) time.
• auto_activate: true : Setting this to true will enable conda automatically when logging into ISAAC, BUT it will drastically slow your log-in time once you have downloaded a lot of software. I suggest setting this to false--and running conda activate only when you need to use conda--if your login time becomes sluggish.

5. As a sanity check, reload your conda and check that the changes to your channels were applied:

conda deactivate
conda activate
conda config --show | grep -A4 -e "\bchannels"