OpenHPC on LQ1 cluster
The HPC software on the Fermilab LQ1 cluster is packaged and distributed by the OpenHPC project. We are currently providing OpenHPC release v1.3 for Red Hat Enterprise 7.7
Installed software components
Available software components are easily configured using the Lua lmod system which modifies the
LD_LIBRARY_PATH (bash) shell environment variables and sets any other needed variables. Lmod also guards against a user having conflicting software packages (e.g. gnu8 compiler and an MPI built with clang) in your shell environment. More information on using lmod is available in the Introduction to lmod.
Quick Introduction to using Lmod
You can list all of the software components available using the
$ module spider # (spyder is also acceptable spelling) [output suppressed]
There is also an
avail option, however, this option, unlike
spider will only show available modules that do not conflict with your currently configured options. Assuming we do not have anything already configured with
avail option will display mainly the available compilers and other utilities that do not depend upon a particular compiler. Here is an (edited) example of the listing you will see:
$ module avail ---------------------------- /opt/ohpc/pub/modulefiles ------------------------------ EasyBuild/3.9.2 cmake/3.14.3 gnu8/8.3.0 llvm5/5.0.1 autotools autotools gnu/5.4.0 hwloc/2.0.3 papi/5.7.0 singularity/3.2.1 charliecloud/0.9.7 gnu7/7.3.0 intel/184.108.40.206 pmix/2.2.2 valgrind/3.15.0
load command will enable a software package within your shell environment. If there is only a single package version available, it suffices to use the package name, e.g.
gnu8, without specifying the particular version:
gnu8/8.3.0. The following loads
$ module load autotools cmake gnu8
Currently loaded modules can be displayed with
$ module list Currently Loaded Modules: 1) autotools 2) cmake/3.14.3 3) gnu8/8.3.0
If a package is no longer needed, it can be unloaded:
$ module unload autotools $ module list Currently Loaded Modules: 1) cmake/3.14.3 2) gnu8/8.3.0
avail again will now show available modules that have
gnu8 as a dependency:
$ module avail --------------- /opt/ohpc/pub/moduledeps/gnu8 ----------------- R/3.5.3 impi/2019.4.243 mpich/3.3 openblas/0.3.5 plasma/2.8.0 scotch/6.0.6 gsl/2.5 likwid/4.3.4 mvapich2/2.3.1 openmpi3/3.1.4 py2-numpy/1.15.3 superlu/5.2.1 hdf5/1.10.5 metis/5.1.0 ocr/1.0.1 pdtoolkit/3.25 py3-numpy/1.15.3
We can now
load an MPI package:
$ module load openmpi3 $ module list Currently Loaded Modules: 1) cmake/3.14.3 2) gnu8/8.3.0 3) openmpi3/3.1.4
We can use
swap to change to another MPI implementation:
$ module swap openmpi3 mvapich2 $ module list Currently Loaded Modules: 1) cmake/3.14.3 2) gnu8/8.3.0 3) mvapich2/2.3.1
purge command will unload all current modules:
$ module purge $ module list No modules loaded
This is useful at the beginning of batch scripts to prevent the batch shell from unintentionally inheriting a module environment from the submission shell.
Python3, and Python2 conda environments
Due to licensing issues Fermilab users are now banned from using software provided by the anaconda project. We recommend that users who build and support their own python environments use packages provided by conda forge. For convenience, we provide a conda forge environment configured with many commonly used packages. This Python 3.9 environment is enabled via modules.
$ module load condaforge/py39 $ conda activate
To see a list of available system installed environments:
$ module load condaforge/py39 $ conda env list # conda environments: # base * /srv/software/condaforge/py39 py27 /srv/software/condaforge/py39/envs/py27
Note that we provide a minimal Python 2.7 environment since v2.7 is now deprecated. Users are encouraged to upgrade to Python 3. You are able to create your own local (to you) custom environments by either creating a new environment or starting by cloning and then customizing and existing environment.
Launching MPI processes
Near the beginning of your batch script, prior to launching an MPI process you should ensure only software modules required by the batch script have been loaded, for example if using
$ module purge $ module load gnu8 openmpi3
There are two common mechanisms for starting up MPI either use
mpirun command is provided by each MPI implementation, and is specific to that implementation. The commands provided by
mvapich, and Intel
impi are all slurm “aware”. They will attempt to use slurm interfaces to distribute and start MPI binaries. In addition, slurm has the
srun command that is able to start up MPI. The following command will list APIs
srun supports for MPI.
$ srun --mpi=list srun: MPI types are... srun: pmi2 srun: pmix_v3 srun: pmix srun: none srun: openmpi srun: pmix_v2
The table below lists recommended launchers for the different MPI implementations. These combinations have been proved to work. Combinations that are not listed either fail, or do not properly launch MPI.
|MPI||Launcher||+ any required flags|
We recommend using
srun to launch all three MPI implementations since it is then possible to use command line options to specify the distribution and binding of MPI ranks to CPU cores and local memory. The options are described in the srun documentation. Careful specification of the process affinities is especially important when running MPI in the hybrid approach combining MPI with thread parallelism. TU Dresden has a nice compendium illustrating different
srun distribution and binding options for MPI.
Binding and distribution of tasks
There’s a good description of MPI process affinity binding and srun here: Click here
Reasonable affinity choices by partition types on the Fermilab LQCD clusters are:
Intel (lq1) --distribution=cyclic:cyclic --cpu_bind=sockets --mem_bind=no