Fermilab Lattice QCD Facility

OpenHPC on LQ1 cluster

The HPC software on the Fermilab LQ1 cluster is packaged and distributed by the OpenHPC project. We are currently providing OpenHPC release v1.3 for Red Hat Enterprise 7.7

Installed software components

Available software components are easily configured using the Lua lmod system which modifies the PATH and LD_LIBRARY_PATH (bash) shell environment variables and sets any other needed variables. Lmod also guards against a user having conflicting software packages (e.g. gnu8 compiler and an MPI built with clang) in your shell environment. More information on using lmod is available in the Introduction to lmod.

Quick Introduction to using Lmod

You can list all of the software components available using the spider option:

$ module spider # (spyder is also acceptable spelling)
[output suppressed]

There is also an avail option, however, this option, unlike spider will only show available modules that do not conflict with your currently configured options. Assuming we do not have anything already configured with lmod, the avail option will display mainly the available compilers and other utilities that do not depend upon a particular compiler. Here is an (edited) example of the listing you will see:

$ module avail
---------------------------- /opt/ohpc/pub/modulefiles ------------------------------
EasyBuild/3.9.2 cmake/3.14.3 gnu8/8.3.0 llvm5/5.0.1 autotools
autotools gnu/5.4.0 hwloc/2.0.3 papi/5.7.0 singularity/3.2.1
charliecloud/0.9.7 gnu7/7.3.0 intel/19.0.4.243 pmix/2.2.2 valgrind/3.15.0

The load command will enable a software package within your shell environment. If there is only a single package version available, it suffices to use the package name, e.g. gnu8, without specifying the particular version: gnu8/8.3.0. The following loads autotools, cmake, and gnu8:

$ module load autotools cmake gnu8

Currently loaded modules can be displayed with list:

$ module list

Currently Loaded Modules:

1) autotools 2) cmake/3.14.3 3) gnu8/8.3.0

If a package is no longer needed, it can be unloaded:

$ module unload autotools
$ module list

Currently Loaded Modules:

1) cmake/3.14.3 2) gnu8/8.3.0

Running avail again will now show available modules that have gnu8 as a dependency:

$ module avail
 
--------------- /opt/ohpc/pub/moduledeps/gnu8 -----------------
R/3.5.3 impi/2019.4.243 mpich/3.3 openblas/0.3.5 plasma/2.8.0 scotch/6.0.6
gsl/2.5 likwid/4.3.4 mvapich2/2.3.1 openmpi3/3.1.4 py2-numpy/1.15.3 superlu/5.2.1
hdf5/1.10.5 metis/5.1.0 ocr/1.0.1 pdtoolkit/3.25 py3-numpy/1.15.3

We can now load an MPI package:

$ module load openmpi3
$ module list

Currently Loaded Modules:

1) cmake/3.14.3 2) gnu8/8.3.0 3) openmpi3/3.1.4

We can use swap to change to another MPI implementation:

$ module swap openmpi3 mvapich2
$ module list

Currently Loaded Modules:

1) cmake/3.14.3 2) gnu8/8.3.0 3) mvapich2/2.3.1

The purge command will unload all current modules:

$ module purge
$ module list

No modules loaded

This is useful at the beginning of batch scripts to prevent the batch shell from unintentionally inheriting a module environment from the submission shell.

Python3, and Python2 conda environments

Due to licensing issues Fermilab users are now banned from using software provided by the anaconda project. We recommend that users who build and support their own python environments use packages provided by conda forge. For convenience, we provide a conda forge environment configured with many commonly used packages. This Python 3.9 environment is enabled via modules.

$ module load condaforge/py39
$ conda activate

To see a list of available system installed environments:

$ module load condaforge/py39
$ conda env list
# conda environments:
#
base                  *  /srv/software/condaforge/py39
py27                     /srv/software/condaforge/py39/envs/py27

Note that we provide a minimal Python 2.7 environment since v2.7 is now deprecated. Users are encouraged to upgrade to Python 3. You are able to create your own local (to you) custom environments by either creating a new environment or starting by cloning and then customizing and existing environment.

Launching MPI processes

Near the beginning of your batch script, prior to launching an MPI process you should ensure only software modules required by the batch script have been loaded, for example if using gnu8 and openmpi:

$ module purge
$ module load gnu8 openmpi3

There are two common mechanisms for starting up MPI either use mpirun or srun. An mpirun command is provided by each MPI implementation, and is specific to that implementation. The commands provided by openmpi, mvapich, and Intel impi are all slurm “aware”. They will attempt to use slurm interfaces to distribute and start MPI binaries. In addition, slurm has the srun command that is able to start up MPI. The following command will list APIs srun supports for MPI.

$ srun --mpi=list
srun: MPI types are...
srun: pmi2
srun: pmix_v3
srun: pmix
srun: none
srun: openmpi
srun: pmix_v2

The table below lists recommended launchers for the different MPI implementations. These combinations have been proved to work. Combinations that are not listed either fail, or do not properly launch MPI.

We recommend using srun to launch all three MPI implementations since it is then possible to use command line options to specify the distribution and binding of MPI ranks to CPU cores and local memory. The options are described in the srun documentation. Careful specification of the process affinities is especially important when running MPI in the hybrid approach combining MPI with thread parallelism. TU Dresden has a nice compendium illustrating different srun distribution and binding options for MPI.

Binding and distribution of tasks

There’s a good description of MPI process affinity binding and srun here: Click here

Reasonable affinity choices by partition types on the Fermilab LQCD clusters are:

Intel (lq1) --distribution=cyclic:cyclic --cpu_bind=sockets --mem_bind=no