Summary

• Objective
• Launch jobs with Slurm
• Resources supervision with Slurm
• Liens
• License

Objectives

Know how to launch different type of jobs with Slurm.

Jobs monitoring with Slurm

Launch jobs with Slurm:

Launch commands from the master node

The following command allocate computing resources ( nodes, memory, cores) and immediately launch the command on each allocate resource.

$ srun + command

Example:

$ srun hostname

Allow to obtain the name of the computing resource used

Reserve computing resources to launch Slurm commands

We use:

$ salloc

This sommand allows to reserve one or several computing resources and work on the master node at the same time.

Commands on the computing resources can be launched later with the command srun + arguments.

When you use this command, it is important to precise a reservation time with the option –time

Example:We reserve 2 nodes( option -N) at the same time for 5 minutes and later we run the hostname command thanks to srun

$ salloc --time=05:00 -N 2 -p short
$ srun hostname

We obtain:

$[tando@master0 ~]$ srun hostname
node21.alineos.net
node14.alineos.net
 

Connect to a node in interactive mode and launch commands:

To connect to a node in interactive mode for X minutes , use the following command:

$ srun -p short --time=X:00 --pty bash -i

Then you can launch on this node without using the srun prefix

Connect to a node in interactive mode with x11 support:

The x11 support allows you to launch graphical software within a node.

You first have to connect to the bioinfo-master.ird.fr with the -X option:

$ ssh -X login@bioinfo-master.ird.fr 

Then you can launch this command with the --x11 option

$ srun -p short --x11 --pty bash -i

Partitions available:

Depending on the type of jobs you want to launch you have the choice between several partitions.

The partitions can be considered job queues, each of which has an assortment of constraints such as job size limit, job time limit, users permitted to use it, etc.

Priority-ordered jobs are allocated nodes within a partition until the resources (nodes, processors, memory, etc.) within that partition are exhausted.

Here are the available partitions:

Note that the gpu node access is restricted, a request access should be done here: request access to gpu

Main options for Slurm:

salloc, srun or sbatch can be used with the following options:

Launch jobs via a script

The batch mode allows to launch an analysis by following the steps described into a script.

Slurm allows to use different types of scripts such as bash, perl or python.

Slurm allocates the desired computing resources and launch analyses on these resources in background

To be interpreted by Slurm, the script should contain a specific header with all the keyword #BATCH to precise the Slurm options.

Slurm script example:

#!/bin/bash
## Define the job name
#SBATCH --job-name=test
## Define the output file
#SBATCH --output=res.txt
## Define the number of tasks
#SBATCH --ntasks=1
## Define the execution time limit
#SBATCH --time=10:00
## Define 100Mo of memory per cpu
#SBATCH --mem-per-cpu=100
sleep 180 #launch a 3 minutes sleep 

To launch an analysis use the following command:

$ sbatch script.sh

With script.sh the name of the script to use

Submit a array job

#!/bin/bash
#SBATCH --partition=short      ### Partition
#SBATCH --job-name=ArrayJob    ### Job Name
#SBATCH --time=00:10:00        ### WallTime
#SBATCH --nodes=1              ### Number of Nodes
#SBATCH --ntasks=1             ### Number of tasks per array job
#SBATCH --array=0-19%4           ### Array index from 0  to 19 with 4 runnings jobs

 
echo "I am Slurm job ${SLURM_JOB_ID}, array job ${SLURM_ARRAY_JOB_ID}, and array task ${SLURM_ARRAY_TASK_ID}."

You have to use the $SBATCH --arrayoption to define the range

The variable ${SLURM_JOB_ID} precise the job id

${SLURM_ARRAY_JOB_ID}precise the id of the job array

${SLURM_ARRAY_TASK_ID} precise the number of the job array task.

The script should give a answer like:

$ sbatch array.srun
Submitted batch job 20303
$ cat slurm-20303_1.out
I am Slurm job 20305, array job 20303, and array task 1.
$ cat slurm-20303_19.out
I am Slurm job 20323, array job 20303, and array task 19.

Submit a R job

You can use the same syntax than before for Slurm. You just have to launch your R script with the R script command

#!/bin/bash
## Define the job name
#SBATCH --job-name=test
## Define the output file
#SBATCH --output=res.txt
## Define the number of tasks
#SBATCH --ntasks=1
## Define the execution time limit
#SBATCH --time=10:00
## Define 100Mo of memory per cpu
#SBATCH --mem-per-cpu=100
Rscript script.R #launch the R script script.R 

Submit a job with several command in parallel at the same time

You have to use the options --ntasks and --cpus-per-task

Example:

#!/bin/bash

#SBATCH --ntasks=2
#SBATCH --cpu-per-task=2

srun --ntasks=1 sleep 10 & 
srun --ntasks=1 sleep 12 &
wait

In this example, we use 2 tasks with 2 cpus allocated per task that is to say 4 cpus allocated for this job.

For each task a sleep is launched at the same time.

Notice the use of srun to launch a parallelised command and the & to launch the command in background

The wait is needed here to ask the job to wait for the end of each command before stopping

Submit an OpenMP job:

A OpenMP job is a job using several cpus on the same single node. Therefore the number of nodes will always be one.

This will work with a program compiled with openMP

#!/bin/bash
#SBATCH --partition=short   ### Partition
#SBATCH --job-name=HelloOMP ### Job Name
#SBATCH --time=00:10:00     ### WallTime
#SBATCH --nodes=1           ### Number of Nodes
#SBATCH --ntasks-per-node=1 ### Number of tasks (MPI processes)
#SBATCH --cpus-per-task=28  ### Number of threads per task (OMP threads)
#SBATCH --account=hpcrcf    ### Account used for job submission
 
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
 
./hello_omp

Environment variables:

   SLURM_JOB_ID		The ID of the job allocation.
   SLURM_JOB_NAME		The name of the job.
   SLURM_JOB_NODELIST	List of nodes allocated to the job.
   SLURM_JOB_NUM_NODES	Number of nodes allocated to the job.
   SLURM_NTASKS		Number of CPU tasks in this job.
   SLURM_SUBMIT_DIR	The directory from which sbatch was invoked.

Delete a job

$ scancel <job_id>

With <job_id>: the job number

Monitor resources:

Get jobs infos :

use the command

$ squeue 

To refresh the infos every 5 seconds

$ squeue -i 5 

Infos on a particular job:

$ scontrol show job <job_id>

With <job_id>: the job number

Infos on the jobs of a particular user

$ squeue -u <user> 

With <user>: the user login

More infos on jobs:

$ sacct --format=JobID,elapsed,ncpus,ntasks,state,node

Infos on resources used by a finished job

$ seff <job_id>  

With <job_id>: the job number

You can add the following command at the end of your script to get infos of the jobs in your output file.

$ seff $SLURM_JOB_ID

Get infos on partition:

Type the following command:

$ sinfo 

It gives infos on partitions and nodes

To get more informations:

$ scontrol show partitions 

scontrol show can be used with nodes, user, account etc…

Konow the time limit for each partition:

$sinfo -o "%10P %.11L %.11l"

Get infos on nodes

Type the command:

$ sinfo -N -l 

Several states are possible:

• alloc : The node is fully uses

• mix : The node is partially used

• idle : No job is runing on the node

• drain : node is finishing the received jobs but it doesn’t accept new ones ( when the node will be stopped for maintenance)

To obtain more informations :

$ scontrol show nodes 

Liens

• Cours liés : HPC Trainings

License

The resource material is licensed under the Creative Commons Attribution 4.0 International License (here).