{"id":193,"date":"2021-02-23T14:52:54","date_gmt":"2021-02-23T20:52:54","guid":{"rendered":"http:\/\/computing.fnal.gov\/lqcd\/?page_id=193"},"modified":"2026-02-23T11:31:00","modified_gmt":"2026-02-23T17:31:00","slug":"slurm","status":"publish","type":"page","link":"https:\/\/computing.fnal.gov\/lqcd\/slurm\/","title":{"rendered":"Slurm on Fermilab USQCD Clusters"},"content":{"rendered":"\n<p>Slurm workload manager, formerly known as Simple Linux Utility For Resource Management (SLURM), is an open source, fault-tolerant, and highly scalable resource manager and job scheduling system of high availability currently developed by&nbsp;<a rel=\"noreferrer noopener\" href=\"https:\/\/slurm.schedmd.com\/\" target=\"_blank\">SchedMD<\/a>. Initially developed for large Linux Clusters at the Lawrence Livermore National Laboratory, Slurm is used extensively on most Top 500 supercomputers around the globe.<\/p>\n\n\n\n<p>If you have questions about job dispatch priorities on the Fermilab LQCD clusters then please&nbsp;<a href=\"https:\/\/computing.fnal.gov\/lqcd\/job-dispatch-explained\/\" data-type=\"page\" data-id=\"517\">visit this page<\/a>&nbsp;or send us an email with your question to&nbsp;<a href=\"mailto:hpc-admin@fnal.gov\">hpc-admin@fnal.gov<\/a>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"https:\/\/computing.fnal.gov\/lqcd\/wp-content\/uploads\/2021\/02\/layout-lq1.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"564\" src=\"https:\/\/computing.fnal.gov\/lqcd\/wp-content\/uploads\/2021\/02\/layout-lq1-1024x564.png\" alt=\"\" class=\"wp-image-202\" srcset=\"https:\/\/computing.fnal.gov\/lqcd\/wp-content\/uploads\/2021\/02\/layout-lq1-1024x564.png 1024w, https:\/\/computing.fnal.gov\/lqcd\/wp-content\/uploads\/2021\/02\/layout-lq1-300x165.png 300w, https:\/\/computing.fnal.gov\/lqcd\/wp-content\/uploads\/2021\/02\/layout-lq1-768x423.png 768w, https:\/\/computing.fnal.gov\/lqcd\/wp-content\/uploads\/2021\/02\/layout-lq1-1536x847.png 1536w, https:\/\/computing.fnal.gov\/lqcd\/wp-content\/uploads\/2021\/02\/layout-lq1.png 1970w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Slurm Commands<\/h3>\n\n\n\n<p>One must log in to the appropriate submit host (see&nbsp;<span class=\"has-inline-color has-vivid-red-color\">Start Here<\/span>&nbsp;in the graphics above) in order to run Slurm commands for the appropriate accounts and resources.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>scontrol<\/code>&nbsp;and&nbsp;<code>squeue<\/code>: Job control and monitoring.&nbsp;<\/li>\n\n\n\n<li><code>sbatch<\/code>: Batch jobs submission.&nbsp;<\/li>\n\n\n\n<li><code>salloc<\/code>: Interactive job sessions are request.&nbsp;<\/li>\n\n\n\n<li><code>srun<\/code>: Command to launch a job.&nbsp;<\/li>\n\n\n\n<li><code>sinfo<\/code>: Nodes info and cluster status.<\/li>\n\n\n\n<li><code>sacct<\/code>: Job and job steps accounting data.<\/li>\n\n\n\n<li>Useful environment variables are&nbsp;<code>$SLURM_NODELIST<\/code>&nbsp;and&nbsp;<code>$SLURM_JOBID<\/code>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Slurm User Accounts<\/h3>\n\n\n\n<p>In order to check your &#8220;default&#8221; Slurm account use the following command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[@lq ~]$ sacctmgr list user name=johndoe\n      User   Def Acct       Admin  \n----------  ---------- ----------\n   johndoe    project       None <\/pre>\n\n\n\n<p>To check \u201call\u201d the Slurm accounts you are associated with use the following command:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[@lq ~]$ sacctmgr list user name=johndoe withassoc       \nUser   Def Acct     Admin    Cluster    Account  Partition     Share   Priority MaxJobs MaxNodes  MaxCPUs MaxSubmit     MaxWall  MaxCPUMins                  QOS   Def QOS \n---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ------- -------- -------- --------- ----------- ----------- -------------------- ---------    \njohndoe   projectx      None     lq   projecta                    1                                                                            opp,regular,test       opp <\/pre>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Slurm Resource Types<\/h3>\n\n\n\n<p>For details on available resource types and their associated constraints, please <a href=\"https:\/\/computing.fnal.gov\/lqcd\/job-dispatch-explained\/\">visit this page<\/a>. In summary, resources are available across two partitions and through three different Slurm QoS as shown in table below.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>Partition name<\/strong><\/td><td><strong>Resource type<\/strong><\/td><td><strong>Number of resources<\/strong><\/td><td><strong>Available QoS<\/strong><\/td><td><strong>Slurm directives for access<\/strong><\/td><\/tr><tr><td>lq1_cpu<\/td><td>2.50GHz Intel Xeon Gold 6248 \u201c<em>Cascade Lake<\/em>\u201c, 196GB memory per node (4.9GB\/core), EDR Omni-Path<\/td><td>179 worker nodes with 40 CPU cores each<\/td><td>normal (for approved allocations)<br>opp (for opportunistic usage)<br>test (for quick testing)<\/td><td>For partition selection:<br><code>--partition=lq1_cpu<\/code><br><br>Node allocation is done using the standard directives like <code>--nodes<\/code>, <code>--ntasks-per-node<\/code> etc.<\/td><\/tr><tr><td>lq2_gpu<\/td><td>NVIDIA A100 GPU nodes<\/td><td>18 worker nodes with 4 GPUs each<\/td><td>normal (for approved allocations)<br>opp (for opportunistic usage)<br>test (for quick testing)<\/td><td>For partition selection:<br><code>--partition=lq2_gpu<\/code><br><br>To request two GPUs:<br><code>--gres=gpu:2<\/code><br>OR<br><code>--gres=gpu:a100:2<\/code><br>OR<br><code>--gpus=a100:2<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Using Slurm: examples<\/h3>\n\n\n\n<p><span style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-orange-color\">Submit an interactive job requesting 12 worker nodes<\/span> :<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[@q:~]$ srun --pty --nodes=12 --ntasks-per-node=40 --partition lq1_cpu bash\n[user@lq1wn001:~]$ env | grep NTASKS\nSLURM_NTASKS_PER_NODE=40\nSLURM_NTASKS=480\n[user@lq1wn001:~]$ exit<\/pre>\n\n\n\n<p><span style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-orange-color\">Submit a batch job requesting 4 worker nodes<\/span> :<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[@lq ~]$ cat myscript.sh\n#!\/bin\/sh\n#SBATCH --job-name=test\n#SBATCH --partition=lq1_cpu\n#SBATCH --nodes=4\n&nbsp;\n# print hostname of worker node\nhostname\nsleep 5\nexit\n&nbsp;\n[@q ~]$ sbatch myscript.sh\nSubmitted batch job 46\n<\/pre>\n\n\n\n<p>Once the batch job completes the output is available as follows :<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[@lq ~]$ cat slurm-46.out\n lq1wn053.fnal.gov<\/pre>\n\n\n\n<p><span style=\"background-color:rgba(0, 0, 0, 0)\" class=\"has-inline-color has-luminous-vivid-orange-color\">Submit a batch job requesting 2 GPUs<\/span> :<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&#91;@lq ~]$ cat myscript.sh\n#!\/bin\/sh\n#SBATCH --account=scd_csi.lq2_gpu\n#SBATCH --qos=normal\n#SBATCH --partition=lq2_gpu\n#SBATCH --gpus=a100:2\n\n# print hostname of worker node\nhostname\nsleep 5\nexit<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Slurm Reporting<\/h3>\n\n\n\n<p>The&nbsp;<code>lquota<\/code>&nbsp;command run on&nbsp;<code>lq.fnal.gov<\/code>&nbsp;will provide allocation usage reporting as shown below.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">[amitoj@lq ~]$ lquota\n Last Updated on: Tue Feb 23 15:00:01 CST 2021\n |--------------- |--------------------- |-------------- |--------------- |--------\n | Account        | Used Sky-ch on LQ1   | Pace          | Allocation     | % Used\n |                | since Jul-1-2020     | MM-DD-YYYY    | in Sky-ch      |\n |--------------- |--------------------- |-------------- |--------------- |--------\n | chiqcd         | 7,825,317            | Jun-29-2021   | 11,992,366     | 65%\n | gluonpdf       | 18,525               | Jan-4-2038    | 500,000        | 4%\n | hadstruc       | 3,269,152            | Apr-26-2021   | 4,137,963      | 79%\n | hiq2ff         | 36                   | Dec-15-3822   | 100,000        | 0%\n | lattsusy       | 452,441              | May-11-2021   | 600,000        | 75%\n | lp3            | 6,178,070            | Oct-23-2021   | 12,500,000     | 49%\n&lt;---snip---&gt;<\/pre>\n\n\n\n<p class=\"has-text-align-center\">lq1-ch=lq1-core-hour , Sky-ch=Sky-core-hour ,1 lq1-ch=1.05 Sky-ch<\/p>\n\n\n\n<p>For questions regarding the reports or should you notice discrepancies in data please email us at&nbsp;<a href=\"mailto:hpc-admin@fnal.gov\">hpc-admin@fnal.gov<\/a> <\/p>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Slurm Environment Variables<\/h3>\n\n\n\n<figure class=\"wp-block-table\"><table><tbody><tr><td><strong>Variable Name<\/strong><\/td><td><strong>Description<\/strong><\/td><td><strong>Example Value<\/strong><\/td><td><strong>PBS\/Torque analog<\/strong><\/td><\/tr><tr><td>$SLURM_JOB_ID<\/td><td>Job ID<\/td><td>5741192<\/td><td>$PBS_JOBID<\/td><\/tr><tr><td>$SLURM_JOBID<\/td><td><strong>Deprecated.<\/strong> Same as SLURM_JOB_ID<\/td><td><\/td><td><\/td><\/tr><tr><td>$SLURM_JOB_NAME<\/td><td>Job Name<\/td><td>myjob<\/td><td>$PBS_JOBNAME<\/td><\/tr><tr><td>$SLURM_SUBMIT_DIR<\/td><td>Submit Directory<\/td><td>\/project\/charmonium<\/td><td>$PBS_O_WORKDIR<\/td><\/tr><tr><td>$SLURM_JOB_NODELIST<\/td><td>Nodes assigned to job<\/td><td>lq1wn00[1-5]<\/td><td>cat $PBS_NODEFILE<\/td><\/tr><tr><td>$SLURM_SUBMIT_HOST<\/td><td>Host submitted from<\/td><td>lq.fnal.gov<\/td><td>$PBS_O_HOST<\/td><\/tr><tr><td>$SLURM_JOB_NUM_NODES<\/td><td>Number of nodes allocated to job<\/td><td>2<\/td><td>$PBS_NUM_NODES<\/td><\/tr><tr><td>$SLURM_CPUS_ON_NODE<\/td><td>Number of cores\/node<\/td><td>8,3<\/td><td>$PBS_NUM_PPN<\/td><\/tr><tr><td>$SLURM_NTASKS<\/td><td>Total number of cores for job<\/td><td>11<\/td><td>$PBS_NP<\/td><\/tr><tr><td>$SLURM_NODEID<\/td><td>Index to node running on relative to nodes assigned to job<\/td><td>0<\/td><td>$PBS_O_NODENUM<\/td><\/tr><tr><td>$SLURM_LOCALID<\/td><td>Index to core running on within node<\/td><td>4<\/td><td><br>$PBS_O_VNODENUM<br><\/td><\/tr><tr><td>$SLURM_PROCID<\/td><td>Index to task relative to job<\/td><td>0<\/td><td>$PBS_O_TASKNUM &#8211; 1<\/td><\/tr><tr><td>$SLURM_ARRAY_TASK_ID<\/td><td>Job Array Index<\/td><td>0<\/td><td>$PBS_ARRAYID<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Binding and Distribution of tasks<\/h3>\n\n\n\n<p>There&#8217;s a good description of MPI process affinity binding for srun&nbsp;here:&nbsp;<a href=\"https:\/\/doc.zih.tu-dresden.de\/jobs_and_resources\/binding_and_distribution_of_tasks\/\" target=\"_blank\" rel=\"noreferrer noopener\">Click here<\/a> <\/p>\n\n\n\n<p>A reasonable distribution \/ affinity choice for the lq1_cpu partition of the Fermilab LQCD clusters is:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">srun --distribution=cyclic:cyclic --cpu_bind=sockets --mem_bind=no<\/pre>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Launching MPI processes<\/h3>\n\n\n\n<p>Please refer to the&nbsp;<a href=\"https:\/\/computing.fnal.gov\/lqcd\/mpi-and-binding\/\" data-type=\"page\">following page<\/a>&nbsp;for recommended MPI launch options.<\/p>\n\n\n\n<h3 class=\"wp-block-heading has-text-align-left\">Additional useful information<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a rel=\"noreferrer noopener\" href=\"http:\/\/slurm.schedmd.com\/pdfs\/summary.pdf\" target=\"_blank\">A quick two page summary of Slurm Commands<\/a><\/li>\n\n\n\n<li><a rel=\"noreferrer noopener\" href=\"https:\/\/slurm.schedmd.com\/quickstart.html\" target=\"_blank\">Quick Start Slurm User Guide<\/a><\/li>\n\n\n\n<li><a rel=\"noreferrer noopener\" href=\"https:\/\/slurm.schedmd.com\/rosetta.pdf\" target=\"_blank\">Comparison between Slurm and other popular batch schedulers<\/a><\/li>\n\n\n\n<li><a rel=\"noreferrer noopener\" href=\"https:\/\/slurm.schedmd.com\/\" target=\"_blank\">Official Slurm documentation<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Slurm workload manager, formerly known as Simple Linux Utility For Resource Management (SLURM), is an open source, fault-tolerant, and highly scalable resource manager and job scheduling system of high availability currently developed by&nbsp;SchedMD. Initially developed for large Linux Clusters at the Lawrence Livermore National Laboratory, Slurm is used extensively on most Top 500 supercomputers around&#8230; <a class=\"more-link\" href=\"https:\/\/computing.fnal.gov\/lqcd\/slurm\/\"> More &#187;<\/a><\/p>\n","protected":false},"author":52,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-193","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/193","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/users\/52"}],"replies":[{"embeddable":true,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/comments?post=193"}],"version-history":[{"count":45,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/193\/revisions"}],"predecessor-version":[{"id":7918,"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/pages\/193\/revisions\/7918"}],"wp:attachment":[{"href":"https:\/\/computing.fnal.gov\/lqcd\/wp-json\/wp\/v2\/media?parent=193"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}