Scripting and Submitting Jobs on Alpine¶
Running jobs on Alpine¶
- Some analysis can take a while (especially with large datasets), so we'll submit it as a "job".
- So far we’ve been working on the interactive node, for jobs we’ll be using the computing node
- Batch jobs: are resource provisions that run applications on compute nodes and do not require supervision or interaction.
- Job script: is a set of Linux commands paired with the resource requirements for your batch job
Create a test job and run it¶
1. Go to OnDemand and create a new file in your slurm directory, call it "test.sh"
2. click edit to open and edit the file.
This can be pasted into your example script. Note, to submit jobs, the .sh file must ALWAYS have the #!/bin/bash line as the first line. the rest are job parameters and are also required, but the order does not matter.
#!/bin/bash
#SBATCH --job-name=test
#SBATCH --nodes=1
#SBATCH --ntasks=2
#SBATCH --partition=amilan
#SBATCH --time=01:00:00
#SBATCH --mail-type=ALL
#SBATCH --mail-user=YOUR_USERNAME@colostate.edu
#SBATCH --output=slurm-%j.out
#SBATCH --qos=normal
#Activate qiime
module purge
module load qiime2/2024.10_amplicon
# command goes here
OUTFILE="message_${SLURM_JOB_ID}.txt"
echo "status report" > $OUTFILE
echo "Job ID: $SLURM_JOB_ID" >> $OUTFILE
echo "Node: $(hostname)" >> $OUTFILE
echo "Timestamp: $(date)" >> $OUTFILE
echo "You ran your first job!" >> $OUTFILE
Submitting jobs for windows users
There could be some odd spaces/characters that windows copied into the script, to be safe run dos2unix on the script before submitting it. Below is an example.
To submit the job to the compute cluster, use:
You will then get a job ID as an output. It might be good to note this job ID in case you need to kill the job or check its status. We can also use the On-Demand portal to look at our job status. You should receive an email as well with your job status!
If you get a "failed" email, you can check your slurm directory for the slurm output file (looks like slurm-JOB-ID.out) to see what went wrong. This is how you troubleshoot failed code from a job.
How do I pick what parameters to use for my job?¶
QIIME2 16S Workflow on Alpine: Typical SLURM Job Requirements¶
| Step | What it does | Time | Nodes | Tasks | Notes |
|---|---|---|---|---|---|
| Import / Convert to QZA | Packages raw data into QIIME2 format | 0.5–2 hours | 1 | 6 | light compute |
| Demultiplex / Summary (demux) | Splits reads by sample + generates quality plots | 1–3 hours | 1 | 6 | Runtime depends on dataset size |
| DADA2 denoising | Error correction + ASV generation + chimera removal + read merging | 3–12+ hours | 1 | 8-12 | requires more compute and time than other steps |
| Phylogenetic tree (SEPP insertion) | Places ASVs into an existing reference phylogeny | up to 12 hours | 1 | 23 | most computationaly heavy part of the workflow |