{"id":4999,"date":"2023-10-06T16:08:59","date_gmt":"2023-10-06T21:08:59","guid":{"rendered":"https:\/\/computing.fnal.gov\/lqcd\/?page_id=4999"},"modified":"2023-10-10T16:00:32","modified_gmt":"2023-10-10T21:00:32","slug":"mpi-and-binding","status":"publish","type":"page","link":"https:\/\/computing.fnal.gov\/lqcd\/mpi-and-binding\/","title":{"rendered":"MPI: affinity and binding"},"content":{"rendered":"\n

Launching MPI processes with srun<\/h2>\n\n\n\n

MPI implementations Open MPI, MVAPICH, and Intel MPI are slurm \u201caware\u201d. They will detect slurm and use its services to distribute and start MPI binaries. The slurm srun<\/a> command must be told which API to use for MPI. The command<\/p>\n\n\n\n

$ srun --mpi=list\nMPI plugin types are...\n\tpmix\n\tcray_shasta\n\tnone\n\tpmi2\nspecific pmix plugin versions available: pmix_v4,pmix_v5<\/pre>\n\n\n\n
lists the supported APIs.<\/p>\n\n\n\n
The table below lists recommended launchers for the different MPI implementations. These combinations have been proved to work. Combinations that are not listed either fail, or do not properly launch MPI.<\/p>\n\n\n\n
MPI<\/strong><\/td>command<\/strong><\/td><\/tr>
Open MPI<\/td>srun --mpi=pmix<\/code><\/td><\/tr>
Intel MPI<\/td>srun --mpi=pmi2<\/code><\/td><\/tr>
nvhpc (Open MPI)<\/td>srun --mpi=pmix<\/code><\/td><\/tr>
MVAPICH<\/td>srun --mpi=pmi2<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Binding and distribution of tasks<\/h3>\n\n\n\n
The srun command provides command line options<\/a> to specify the distribution and binding of MPI ranks to CPU cores and local memory. Careful specification of the distribution and affinities is especially important when running MPI in the hybrid approach combining MPI with thread parallelism. TU Dresden has a nice compendium<\/a> illustrating different CPU MPI rank+threads distribution and binding options for MPI.<\/p>\n\n\n\n
LQ2 GPU workers<\/h3>\n\n\n\n
Each LQ2 worker is equipped with four NVIDIA A100-80<\/a> GPU devices interconnected by an NVLink mesh. The system is a dual socket with 3rd Gen. AMD EPYC 7543<\/a> 32-Core Processors (64 codes total). Each worker has two InfiniBand adapters. The figure below shows the topology reported by the hwloc-ls<\/code> command.<\/p>\n\n\n\n