Quick Start

Create pipeline script

Create a script to submit all jobs we need. Here I assume we have 2 jobs and each one has its own script(written in python)

#!/bin/bash
qsub -N Job1 -q pipe.q -S /usr/bin/python -cwd job1.py
qsub -N Job2 -q pipe.q -S /usr/bin/python -cwd job2.py

Define the job by inherit class Queue

Name Jobs.py

import Queue
import time
class job1(Queue.Queue):
    def __init__(self):
        super(job1, self).__init__()

    def run(self):
        print "I am ",self.step_name
        time.sleep(5)

class job2(Queue.Queue):
    def __init__(self):
        super(job1, self).__init__()

    def run(self):
        print "I am ",self.step_name
        time.sleep(5)

Create job1.py and job2.py

#!/usr/bin/python
import Jobs
job1 = Jobs.job1()
# set job name for this step
job1.setName("tophat");
# status log file for previous step
job1.setPrevStateLog('0.hipipe')
# status log file of this step
job1.setCurrentStateLog('tophat.hipipe')
# log file of this step
job1.setTaskLog('tophat.out')
# run your job
job1.go()
# or you can call external program
job1.go_by_cmd(['ls','-al'])

#!/usr/bin/python
import Jobs
job2 = Jobs.job2()
# set job name for this step
job2.setName("gatk");
# status log file for previous step
job2.setPrevStateLog('tophat.hipipe')
# status log file of this step
job2.setCurrentStateLog('gatk.hipipe')
# log file of this step
job2.setTaskLog('gatk.out')
# run your job
job2.go()
# or you can call external program
job2.go_by_cmd(['ls','-al'])

What happen during the runtime

First, job1.py will check 0.hipipe to see if there is an error message as below.

[hipipe] error
[hipipe] stop

If such error message exist, the program will end directly without running. Because job1 is my first job and I assign a non-exist file to it, it will pass the check and run normally and output the file below.

tophat.hipipe

[hipipe] start
[hipipe] end

tophat.out

I am Tophat

Then the job2.py execute and it also check if there is error message in tophat.hipipe. Because the formal job end normally so there will not error message show in tophat.hipipe, so the job2 named gatk will start to run and output the gatk.hipipe and gatk.out.