Workflow Management and Reproducibility - iffatAGheyas/bioinformatics-tutorial-wiki GitHub Wiki

4.8 Workflow Management & Reproducibility

Complex RNA-Seq analyses involve many steps and tools. Automating and containerizing your pipeline ensures reproducibility, ease of maintenance, and portability.

---

4.8.1 Snakemake

• Why Snakemake?
  – Python-based “Make”-style workflow engine
  – Declarative rules with input/output files
  – Built-in support for cluster / cloud execution

Example Snakefile:

# Snakefile

# 1. Config
configfile: "config.yaml"

# 2. Samples
SAMPLES = config["samples"]

# 3. Rules
rule all:
    input:
        expand("counts/{sample}.counts.txt", sample=SAMPLES)

rule fastqc:
    input:
        r1="raw_data/{sample}_R1.fastq.gz",
        r2="raw_data/{sample}_R2.fastq.gz"
    output:
        "qc/fastqc/{sample}_R1_fastqc.html",
        "qc/fastqc/{sample}_R2_fastqc.html"
    shell:
        "fastqc -o qc/fastqc {input.r1} {input.r2}"

rule trim:
    input:
        r1="raw_data/{sample}_R1.fastq.gz",
        r2="raw_data/{sample}_R2.fastq.gz"
    output:
        r1="trimmed/{sample}_R1.trimmed.fastq.gz",
        r2="trimmed/{sample}_R2.trimmed.fastq.gz"
    shell:
        "cutadapt -q 20 -o {output.r1} -p {output.r2} {input.r1} {input.r2}"

# … additional rules for alignment, quantification, etc. …

Example config.yaml:

samples:
  - SampleA
  - SampleB
  - SampleC
reference: "ref/genome.fa"
gtf:       "ref/annotations.gtf"

• Run the workflow:

conda activate rna_seq_env
snakemake --cores 8 --use-conda

4.8.2 Nextflow

• Why Nextflow?
  • Groovy-based DSL, seamless cluster/cloud support
  • Built-in Docker / Singularity integration
  • Versioned pipelines via GitHub integration

Example main.nf:

// Your Nextflow pipeline code goes here
process hello {
  """
  echo 'Hello, Nextflow!'
  """
}
workflow {
  hello()
}

Run the pipeline:

nextflow run main.nf -profile docker -resume -with-report report.html

4.8.3 Containerization

Encapsulate your software stack for portability:

• Docker (Linux / Mac / Windows)

# Dockerfile
FROM continuumio/miniconda3
RUN conda install -c bioconda fastqc multiqc star salmon subread snakemake nextflow
COPY . /pipeline
WORKDIR /pipeline
ENTRYPOINT ["snakemake"]
CMD ["--cores", "4"]

docker build -t rna_seq_pipeline:latest .
docker run --rm -v $PWD:/work rna_seq_pipeline:latest

• Singularity (HPC environments)

singularity build rna_seq_pipeline.sif docker://rna_seq_pipeline:latest
singularity exec rna_seq_pipeline.sif snakemake --cores 8

4.8.4 Best Practices

• Version control your workflow scripts (Snakefile, nextflow.config) and environment.yml or Dockerfile.
• Bind your metadata (samplesheet.tsv, config.yaml) to the pipeline—never hard-code sample names.
• Use explicit software versions in Conda environments or containers for full reproducibility.
• Test on a small subset of data before scaling to full datasets.
• Log execution and resource usage (snakemake --report, Nextflow -with-trace) for performance tuning and provenance.

With automated, containerized workflows in place, your RNA-Seq pipeline becomes modular, reproducible, and easily shareable—ideal for collaboration and production analyses.