Skip to end of metadata
Go to start of metadata

This page shows you how to run a regular bash script as a pipeline. The runAsPipeline script, accessible through the rcbio/1.0 module, converts an input bash script to a pipeline that easily submits jobs to the Slurm scheduler for you.

Features of the new pipeline:

  • Submit each step as a cluster job using sbatch.
  • Automatically arrange dependencies among jobs.
  • Email notifications are sent when each job fails or succeeds.
  • If a job fails, all its downstream jobs automatically are killed.
  • When re-running the pipeline on the same data folder, if there are any unfinished jobs, the user is asked to kill them or not.
  • When re-running the pipeline on the same data folder, the user is asked to confirm to re-run or not if a step was done successfully earlier.

Please read below for an example.

Log on to O2

If you need help connecting to O2, please review the Using Slurm Basic wiki page.

From Windows, use the graphical PuTTY program to connect to o2.hms.harvard.edu and make sure the port is set to the default value of 22.

From a Mac Terminal, use the ssh command, inserting your eCommons ID instead of user123:

$ ssh user123@o2.hms.harvard.edu


Start interactive job, and create working folder

mfk8@login03:~$ srun --pty -p interactive -t 0-12:0:0 --mem 2000MB -n 1 /bin/bash
mfk8@compute-a-16-79:~$ mkdir -p /n/scratch2/$USER/testRunBashScriptAsSlurmPipeline && cd /n/scratch2/$USER/testRunBashScriptAsSlurmPipeline


Load the pipeline related modules

# This will setup the path and environmental variables for the pipeline
mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ module load rcbio/1.0


Build some testing data in the current folder

mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ echo -e "John Paul\nMike Smith\nNick Will\nJulia Johnson\nTom Jones"  >> universityA.txt
mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ echo -e "John Paul\nMike Smith\nNick Will\nJulia Johnson\nTom Jones"  >> universityB.txt


Take a look at the example files

mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ cat universityA.txt
John Paul
Mike Smith
Nick Will
Julia Johnson
Tom Jones
mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ cat universityB.txt
John Paul
Mike Smith
Nick Will
Julia Johnson
Tom Jones


The original bash script

mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ cp /n/app/rcbio/1.0/bin/bash_script_v1.sh .
mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ cat bash_script_v1.sh
#!/bin/sh

for i in A B; do            


    u=university$i.txt   
  
    #step1, find John and Mike    
    grep -H John $u >>  John.txt; grep -H Mike $u >>  Mike.txt       
  
    #step2, find Nick and Julia
    grep -H Nick $u >>  Nick.txt; grep -H Julia $u >>  Julia.txt
   
done
   
# step3 merge          
cat John.txt Mike.txt Nick.txt Julia.txt > all.txt

exit

# command to create testing data
# echo -e "John Paul\nMike Smith\nNick Will\nJulia Johnson\nTom Jones"  >> universityA.txt
# echo -e "John Paul\nMike Smith\nNick Will\nJulia Johnson\nTom Jones"  >> universityB.txt


How does this bash script work?

There is a loop that goes through the two university text files to search for John and Mike, and then searches for Nick and Julia.  After all searching is finished, then the results are merged into a text single file. This means that the merge step has to wait until the earlier two steps are finished. However, the runAsPipeline workflow builder can't read this script directly. We will need to create a modified bash script that adds parts that explicitly tell the workflow builder the order in which the jobs need to run, among other things. 

The modified bash script

mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ cp /n/app/rcbio/1.0/bin/bash_script_v2.sh .
mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ cat bash_script_v2.sh
#!/bin/sh

#the following is added, meaning: starting a loop and use university as looping variable    
#loopStart,i	
for i in A B; do            
 
    u=university$i.txt    
  
    #the following is added, meaning: start step1, depends on nothing, step name is "find1", want to copy u to /tmp        
    #@1,0,find1,u:     
    grep -H John $u >>  John.txt; grep -H Mike $u >>  Mike.txt        
 
    #the following is added, meaning: start step2, depends on nothing, step name is "find2", want to copy u to /tmp        
    #@2,0,find2,u,sbatch -p short -n 1 -t 50:0
    grep -H Nick $u >>  Nick.txt; grep -H Julia $u >>  Julia.txt

#the following is added, meaning: loop end here    
#loopEnd                     
done

  
#the following is added, meaning: start step3, depends on step1 and 2, step name is "merge"
#@3,1.2,merge:           
cat John.txt Mike.txt Nick.txt Julia.txt > all.txt


# command to create testing data
# echo -e "John Paul\nMike Smith\nNick Will\nJulia Johnson\nTom Jones"  >> universityA.txt
# echo -e "John Paul\nMike Smith\nNick Will\nJulia Johnson\nTom Jones"  >> universityB.txt


Notice that there are a few things added to the script here:

  • before the loop starts, #loopStart,i was added. Here the variable i is looping variable, which will be recognized by the converter.
  • before the loop ends, #loopEnd was added. This will be recognized by the converter.
  • before each step starts: #@step_id,depend_on_id,step_name,sbatch_options was added. Notice that sbatch_options is optional; only step 2 has it in this example.
  • Step 1 is denoted by #@1,0,find1,u:, which means this is step 1 that depends on no other step, is named find1, and the u database needs to be copied to the /tmp directory.  
  • Step 2 is denoted by #@2,0,find2,usbatch -p short -n 1 -t 50:0, which means this is step 2 that depends on no other step, is named find2, and this step needs copy a u database to /tmp directory with given sbatch options.
  • Step 3 is denoted by #@3,1.2,merge:. This means that this is step3 that depends on step 1 and step 2, and the step is named merge.

Build a pipeline based on the modified bash script

mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ runAsPipeline bash_script_v2.sh "sbatch -p short -t 10:0 -n 1" useTmp


This command will generate new bash script named slurmPipeLine.201801100946.sh in flag folder (201801100946 is the timestamp that runAsPipeline was invoked at). Then test run it, meaning does not really submit jobs, but only create a fake job id, 123 for each step. If you were to append run at the end of the command, the pipeline would actually be submitted to the Slurm scheduler.

Ideally, the software should run faster using local /tmp disk space than the network storage. For this small query, the difference is small, or even slower if you use local /tmp.

Sample output from the converter

Note that only step 2 used -t 50:0, and all other steps used the default -t 10:0. The default walltime limit was set in the runAsPipeline command, and the walltime parameter for step 2 was set in the bash_script_v2.sh script.

mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ runAsPipeline bash_script_v2.sh "sbatch -p short -t 10:0 -n 1" useTmp
converting bash_script_v2.sh to flag/slurmPipeLine.201801161424.sh

find loopStart: #loopStart,i	

find job marker:
#@1,0,find1,u:     

find job:
grep -H John $u >>  John.txt; grep -H Mike $u >>  Mike.txt        


find job marker:
#@2,0,find2,u,sbatch -p short -n 1 -t 50:0
sbatch options: sbatch -p short -n 1 -t 50:0

find job:
grep -H Nick $u >>  Nick.txt; grep -H Julia $u >>  Julia.txt
find loopend: #loopEnd                     

find job marker:
#@3,1.2,merge:           


find job:
cat John.txt Mike.txt Nick.txt Julia.txt > all.txt
flag/slurmPipeLine.201801161424.sh bash_script_v2.sh is ready to run. Starting to run ...
Running flag/slurmPipeLine.201801161424.sh bash_script_v2.sh
---------------------------------------------------------

step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u
depend on no job
sbatch -p short -t 10:0 -n 1 --nodes=1  -J 1.0.find1.A -o /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out -e /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.sh 
# Submitted batch job 123

step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u
depend on no job
sbatch -p short -n 1 -t 50:0 --nodes=1  -J 2.0.find2.A -o /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out -e /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.sh 
# Submitted batch job 123

step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u
depend on no job
sbatch -p short -t 10:0 -n 1 --nodes=1  -J 1.0.find1.B -o /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh 
# Submitted batch job 123

step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u
depend on no job
sbatch -p short -n 1 -t 50:0 --nodes=1  -J 2.0.find2.B -o /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out -e /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.sh 
# Submitted batch job 123

step: 3, depends on: 1.2, job name: merge, flag: merge reference:
depend on multiple jobs
sbatch -p short -t 10:0 -n 1 --nodes=1 --dependency=afterok:123:123:123:123 -J 3.1.2.merge -o /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /n/scratch2/kmk34/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh 
# Submitted batch job 123

all submitted jobs:
job_id       depend_on              job_flag  
123         null                  1.0.find1.A
123         null                  2.0.find2.A
123         null                  1.0.find1.B
123         null                  2.0.find2.B
123         ..123.123..123.123    3.1.2.merge
---------------------------------------------------------


Run the pipeline

Thus far in the example, we have not actually submitted any jobs to the scheduler. To submit the pipeline, you will need to append the run parameter to the command. If run is not specified, test mode will be used, which does not submit jobs and gives theplaceholder of 123 for jobids in the command's output. 

mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ runAsPipeline bash_script_v2.sh "sbatch -p short -t 10:0 -n 1" useTmp run
converting bash_script_v2.sh to flag/slurmPipeLine.201801101002.run.sh


find loopStart: #loopStart,i


find job marker:
#@1,0,find1,u:


find job:
grep -H John $u >> John.txt; grep -H Mike $u >> Mike.txt


find job marker:
#@2,0,find2,u,sbatch -p short -n 1 -t 50:0
sbatch options: sbatch -p short -n 1 -t 50:0


find job:
grep -H Nick $u >> Nick.txt; grep -H Julia $u >> Julia.txt
find loopend: #loopEnd


find job marker:
#@3,1.2,merge:


find job:
cat John.txt Mike.txt Nick.txt Julia.txt > all.txt
flag/slurmPipeLine.201801101002.run.sh is ready to run. Starting to run ...
Running flag/slurmPipeLine.201801101002.run.sh
---------------------------------------------------------


step: 1, depends on: 0, job name: find1, flag: find1.A reference: .u
depend on no job
sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 -J 1.0.find1.A -o /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out -e /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.out /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.A.sh
# Submitted batch job 8091045


step: 2, depends on: 0, job name: find2, flag: find2.A reference: .u
depend on no job
sbatch -p short -n 1 -t 50:0 --kill-on-invalid-dep=yes --nodes=1 -J 2.0.find2.A -o /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out -e /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.out /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.A.sh
# Submitted batch job 8091046


step: 1, depends on: 0, job name: find1, flag: find1.B reference: .u
depend on no job
sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 -J 1.0.find1.B -o /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out -e /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.out /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/1.0.find1.B.sh
# Submitted batch job 8091047


step: 2, depends on: 0, job name: find2, flag: find2.B reference: .u
depend on no job
sbatch -p short -n 1 -t 50:0 --kill-on-invalid-dep=yes --nodes=1 -J 2.0.find2.B -o /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out -e /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.out /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/2.0.find2.B.sh
# Submitted batch job 8091048


step: 3, depends on: 1.2, job name: merge, flag: merge reference:
depend on multiple jobs
sbatch -p short -t 10:0 -n 1 --kill-on-invalid-dep=yes --nodes=1 --dependency=afterok:8091045:8091047:8091046:8091048 -J 3.1.2.merge -o /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out -e /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.out /n/scratch2/mfk8/testRunBashScriptAsSlurmPipeline/flag/3.1.2.merge.sh
# Submitted batch job 8091049
all submitted jobs:
job_id depend_on job_flag
8091045 null 1.0.find1.A
8091046 null 2.0.find2.A
8091047 null 1.0.find1.B
8091048 null 2.0.find2.B
8091049 ..8091045.8091047..8091046.8091048 3.1.2.merge
---------------------------------------------------------


Monitoring the jobs

You can use the command:


mfk8@compute-a-16-79:testRunBashScriptAsSlurmPipeline$ squeue -u $USER


To see the job status (running, pending, etc.). You also get two emails for each step, one at the start of the step, one at the end of the step.


To run your own script as Slurm pipeline

If you have a bash script with multiple steps and you wish to run it as Slurm pipeline, modify your old script and add the notation to mark the start and end of any loops, and the start of any step for which you want to submit as an sbatch job. Then you can use runAsPipeline with your modified bash script, as detailed above. 

Let us know if you have any questions. Please include your working folder and commands used in your email. Any comment and suggestion are welcome!


  • No labels