{"id":456,"date":"2020-10-02T12:46:21","date_gmt":"2020-10-02T17:46:21","guid":{"rendered":"https:\/\/computing.fnal.gov\/wilsoncluster\/?page_id=456"},"modified":"2024-02-07T15:45:17","modified_gmt":"2024-02-07T21:45:17","slug":"slurm-job-scheduler","status":"publish","type":"page","link":"https:\/\/computing.fnal.gov\/wilsoncluster\/slurm-job-scheduler\/","title":{"rendered":"SLURM job scheduler"},"content":{"rendered":"\n

Slurm workload manager, formerly known as Simple Linux Utility For Resource Management (SLURM), is an open source, fault-tolerant, and highly scalable resource manager and job scheduling system of high availability currently developed by SchedMD<\/a>. Initially developed for large Linux Clusters at the Lawrence Livermore National Laboratory, Slurm is used extensively on most Top 500 supercomputers around the globe.<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

Caveat concerning batch job submission<\/mark><\/h3>\n\n\n\n

Please note that Slurm batch jobs will NOT run from your home directory on the Wilson cluster!<\/mark><\/strong><\/p>\n\n\n\n

Your Wilson home directory, $HOME<\/code>, is your lab-wide “nashome” directory. The \/nashome<\/code> filesystem is mounted with Kerberos authentication, and unfortunately, Kerberos is not compatible<\/mark><\/strong> with Slurm. Start batch jobs from your area under either \/work1<\/code> or \/wclustre<\/code>.<\/p>\n\n\n\n

Common Slurm commands<\/h4>\n\n\n\n
command<\/strong><\/td>brief<\/strong> description<\/strong><\/td><\/tr>
squeue<\/a><\/td>reports the state of queued jobs<\/td><\/tr>
sbatch<\/a><\/td>submit a job script for later execution<\/td><\/tr>
scancel<\/a><\/td>cancel a pending or running job<\/td><\/tr>
scontrol<\/a><\/td>view or modify Slurm configuration and state<\/td><\/tr>
salloc<\/a><\/td>allocate resources and spawn a shell which is then used to execute srun commands<\/td><\/tr>
srun<\/a><\/td>submit a job for execution or initiate job steps in real time<\/td><\/tr>
sacctmgr<\/a><\/td>view and modify Slurm account information<\/td><\/tr>
sinfo<\/a><\/td>worker node and partition information<\/td><\/tr><\/tbody><\/table>
Table 1 – Common Slurm Commands<\/figcaption><\/figure>\n\n\n\n

See the downloadable<\/a> PDF cheatsheet for a summary of the commands. SchedMD has a quick start guide<\/a> for Slurm users.<\/p>\n\n\n\n

squeue examples: checking the status of batch jobs<\/h4>\n\n\n\n

The command below checks the status of your jobs in the queue.<\/p>\n\n\n\n

$ squeue -u $USER<\/code><\/pre>\n\n\n\n
The following command will show jobs assigned to the wc_cpu (CPU) partition.<\/p>\n\n\n\n
$ squeue -p wc_cpu<\/code><\/pre>\n\n\n\n
The squeue<\/code> command with the options below will tell you the status of GPU batch jobs including which workers are in use, how many GPUs a job is using, the number of cores, maximum memory, how long the job has been running, and the time limit for the job.<\/p>\n\n\n\n
$ squeue -p wc_gpu --Format=Account:.10,UserName:.10,NodeList:.10,tres-alloc:.64,State:.8,TimeUsed:.10,TimeLimit:.12<\/code><\/pre>\n\n\n\n
Details about your Slurm account(s)<\/h4>\n\n\n\n
Every Slurm user has a default account. You can find your default with the command:<\/p>\n\n\n\n
$ sacctmgr list user name=$USER\n      User   Def Acct     Admin\n---------- ---------- ---------\n     smith    wc_test      None<\/code><\/pre>\n\n\n\n
Users on Wilson may have multiple Slurm accounts with different quality of service (QOS) levels. You can find all your associated accounts along with available QOS levels using the following command:<\/p>\n\n\n\n
$ sacctmgr list user WithAssoc Format=User,Account,QOS,DefaultQOS name=$USER\n      User    Account                  QOS   Def QOS\n---------- ---------- -------------------- ---------\n     smith    hpcsoft             opp,test       opp\n     smith   scd_devs     opp,regular,test       opp\n     smith  spack4hpc     opp,regular,test       opp\n     smith    wc_test     opp,regular,test       opp<\/code><\/pre>\n\n\n\n
NOTE:<\/strong> If you do not specify an account name during your job submission (using --account<\/code>), your default account will be used.<\/p>\n\n\n\n
Quality of service (QoS) levels<\/h4>\n\n\n\n
Jobs submitted to Slurm are associated with an appropriate QoS (or Quality of Service) configuration. Admins assign parameters to a QoS that are used to manage dispatch priority and resource use limits.<\/p>\n\n\n\n
Name<\/strong><\/td>Description<\/strong><\/td>Priority<\/strong> (higher is better)<\/strong><\/td>Global Resource Constraint<\/strong>s<\/td>Max Wall<\/strong>time<\/strong><\/td>Per Account Constraints<\/strong><\/td>Per User Constraints<\/strong><\/td><\/tr>
admin<\/td>admin testing<\/td>100<\/td>None<\/td>None<\/td>None<\/td>None<\/td><\/tr>
test<\/td>for quick user testing<\/td>75<\/td>Max nodes = 5
Max GPUs = 4<\/td>		04:00:00<\/td>None<\/td>Max running jobs = 1
Max queued jobs = 3<\/td><\/tr>
regular<\/td>regular QoS for approved accounts<\/td>25<\/td>None<\/td>1-00:00:00<\/td>None<\/td>None<\/td><\/tr>
walltime7d<\/td>only available to certain approved accounts<\/td>25<\/td>Max nodes = 50<\/td>7-00:00:00<\/td>None<\/td>None<\/td><\/tr>
opp<\/td>available to all accounts for opportunistic usage<\/td>0<\/td>None<\/td>08:00:00<\/td>None<\/td>Max queued jobs = 50<\/td><\/tr><\/tbody><\/table>
Table 2 – Wilson cluster QoS overview<\/figcaption><\/figure>\n\n\n\n
<\/p>\n\n\n\n
A few notes about the resource constraints:<\/p>\n\n\n\n