{"id":384,"date":"2020-10-02T12:35:09","date_gmt":"2020-10-02T17:35:09","guid":{"rendered":"https:\/\/computing.fnal.gov\/wilsoncluster\/?page_id=384"},"modified":"2024-02-15T13:53:00","modified_gmt":"2024-02-15T19:53:00","slug":"software","status":"publish","type":"page","link":"https:\/\/computing.fnal.gov\/wilsoncluster\/software\/","title":{"rendered":"WC HPC Software"},"content":{"rendered":"\n

HPC toolchains on the Wilson cluster<\/h2>\n\n\n\n

The HPC toolchains Wilson were built using a combination of EasyBuild<\/a> recipes in combination with additional hand-built packages. Under AlmaLinux 8 (el8) the deployments are found in the directory \/srv\/software\/el8\/x86_64<\/code>. The subdirectory .\/eb<\/code> contains packages built with EasyBuild while directory .\/hpc<\/code> contains the additional packages built by other means. We provide EasyBuild as a package for users wishing to build their own software stacks. Users are also encouraged to consider spack<\/a> to build custom toolchains and libraries not found among the provided Wilson software packages.<\/p>\n\n\n\n

Installed compiler toolchains<\/h2>\n\n\n\n

The supported compiler toolchains include<\/p>\n\n\n\n

toolchain name<\/strong><\/td>compilers<\/strong><\/td>MPI<\/strong><\/td><\/tr>
gompi (gnu compiler+MPI)<\/td>gnu compiler suite<\/a><\/td>Open MPI<\/a><\/td><\/tr>
intel<\/td>intel compilers (one API<\/a>)<\/td>Intel MPI<\/td><\/tr>
nvhpc<\/td>NVIDIA HPC toolkit<\/a><\/td>Open MPI<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Quickstart: gnu compilers and Open MPI<\/h3>\n\n\n\n

To load the latest installed toolchain for CPU-only compilations, from your shell run the command<\/p>\n\n\n\n

$ module load gompi<\/code><\/pre>\n\n\n\n
The above loads the latest package versions. A specific version is specified by appending a version string to the package name, e.g., gompi\/2023a<\/code>.<\/p>\n\n\n\n
To add MPI support for GPUs, instead use the command<\/p>\n\n\n\n
$ module load gompi ucx_cuda ucc_cuda<\/code><\/pre>\n\n\n\n
The ucx_cuda<\/code> and ucc_cuda<\/code> modules provide GPU direct RDMA<\/a> support and optimized MPI collective operations<\/a>. Typically, these options will need to be enabled in application codes when they are built.<\/p>\n\n\n\n
Quickstart: Intel compilers and Intel MPI<\/h3>\n\n\n\n
To load the latest toolchain, use the command<\/p>\n\n\n\n
$ module load intel<\/code><\/pre>\n\n\n\n
The load command also enables both Intel’s MKL linear algebra and FFT libraries as well as a corresponding newer gnu compiler suite. <\/p>\n\n\n\n
Quickstart: NVIDIA HPC toolkits<\/h3>\n\n\n\n
The NVIDIA HPC toolkits mainly targets applications to run on NVIDIA GPUs. The command,<\/p>\n\n\n\n
$ module avail nvhpc<\/code><\/pre>\n\n\n\n
will list the available toolkit options and versions. Please refer to the HPC toolkit documentation<\/a> for further information.<\/p>\n\n\n\n
Installed toolchain versions<\/h2>\n\n\n\n
The tables in this section list the versions of the toolchains installed on the Wilson cluster. We follow the EasyBuild toolchain version scheme<\/a> for the gompi<\/code> and intel<\/code> toolchains.<\/p>\n\n\n\n
gcc + OpenMPI: gompi toolchains<\/h4>\n\n\n\n
version<\/td>date<\/td>gcc<\/td>Open MPI<\/td>binutils<\/td>CUDA ver.<\/td><\/tr>
2023a<\/td>Jun’23<\/td>12.3.0<\/td>4.1.5<\/td>2.40<\/td>12.2<\/td><\/tr>
2022a<\/td>Jun’22<\/td>11.4.0<\/td>4.1.5<\/td>2.40<\/td>11.8<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Intel compiler + Intel MPI: intel toolchains<\/h4>\n\n\n\n
version<\/td>date<\/td>compilers<\/td>MPI<\/td>MKL<\/td>gcc<\/td>binutils<\/td><\/tr>
2023a<\/td>Jun’23<\/td>2023.1.0<\/td>2021.9.0<\/td>2023.1.0<\/td>12.3.0<\/td>2.40<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
NVIDIA HPC toolkits<\/h4>\n\n\n\n
Installed software components<\/h2>\n\n\n\n
version<\/td>date<\/td>CUDA<\/td><\/tr>
23.7<\/td>Jul’23<\/td>12.2<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Quick introduction to using Lmod<\/h2>\n\n\n\n
Available software components are easily configured using the Lua lmod<\/a> system which modifies the PATH<\/code> and LD_LIBRARY_PATH<\/code> (bash) shell environment variables and sets any other needed variables. More information on using lmod is available in the Introduction to lmod<\/a>.<\/p>\n\n\n\n
You can list all of the software components available using the spider<\/code> option<\/p>\n\n\n\n
$ module spider\n[output suppressed]<\/code><\/pre>\n\n\n\n
There is also an avail<\/code> option, however, this option, unlike spider<\/code> will only show available modules that do not conflict with your currently configured options. Assuming we do not have anything already configured with lmod<\/code>, the avial<\/code> option will display mainly the available compilers and other utilities that do not depend upon a particular compiler. Here is an (edited) example of the listing you will see:<\/p>\n\n\n\n
$ module avail\n---------------- \/srv\/software\/el8\/x86_64\/hpc\/lmod\/Core ------------------------------------------\n anaconda\/2023.07-2  cmake\/3.27.3    cuda\/12.2.1 (D)    julia\/1.6.7-lts        mambaforge\/23.1.0-4\n apptainer\/1.2.1     cuda\/11.8.0     git\/2.41.0         julia\/1.10.0    (D)\n\n--------------- \/srv\/software\/el8\/x86_64\/hpc\/nvhpc\/modulefiles -----------------------------------\n nvhpc-byo-compiler\/23.7    nvhpc-hpcx-cuda12\/23.7    nvhpc-nompi\/23.7\n nvhpc-hpcx-cuda11\/23.7     nvhpc-hpcx\/23.7           nvhpc\/23.7\n\n--------------- \/srv\/software\/el8\/x86_64\/eb\/lmod\/all ---------------------------------------------\n binutils\/2.40_gcccore_12.3.0  imkl\/2023.1.0                          openmpi\/4.1.5_gcc_12.3.0\n binutils\/2.40                 imkl_fftw\/2023.1.0_iimpi_2023a         pmix\/4.2.6\n easybuild\/4.8.0               impi\/2021.9.0_intel_compilers_2023.1.0 szip\/2.1.1_gcccore_12.3.0\n fftw\/3.3.10_gcc_12.3.0        intel\/2023a                            tbb\/2021.10.0_gcccore_12.3.0\n gcc\/12.3.0                    intel_compilers\/2023.1.0               ucc\/1.2.0\n gcccore\/12.3.0                lapack\/3.11.0_gcc_12.3.0               ucc_cuda\/1.2.0_cuda_12.2.1\n gdrcopy\/2.3.1                 libevent\/2.1.12                        ucx\/1.14.1\n gompi\/2023a                   libpciaccess\/0.17                      ucx_cuda\/1.14.1_cuda_12.2.1\n hdf5\/1.14.2_gompi_2023a       libxml2\/2.11.4                         valgrind\/3.21.0_gompi_2023a\n hdf5\/1.14.2_serial_gcc_12.3.0 nccl\/2.18.5_cuda_12.2.1                vtune\/2022.3.0\n hwloc\/2.9.2_cuda_12.2.1       numactl\/2.0.16                         xz\/5.4.2\n hwloc\/2.9.2                   openblas\/0.3.24_gcc_12.3.0             zlib\/1.2.13\n iimpi\/2023a <\/code><\/pre>\n\n\n\n
Currently loaded modules can be displayed with module list<\/code>. Below, git<\/code>, cmake<\/code>, and gcc are activated, and module list<\/code> is used to see which modules are in use.<\/p>\n\n\n\n
$ module load git cmake gcc\n$ module list\nCurrently Loaded Modules:\n 1) git\/2.41.0  2) cmake\/3.27.2 3) gcccore\/12.3.0 \n 4) zlib\/1.2.13 5) binutils\/2.40_gcccore_12.3.0 6) gcc\/12.3.0\n\nWhen a package is no longer needed, it can be unloaded<\/span><\/code><\/pre>\n\n\n\n
$ module unload gcc\n$ module list\nCurrently Loaded Modules:\n  1) git\/2.41.0   2) cmake\/3.27.2<\/code><\/pre>\n\n\n\n
The unload<\/code> above automatically unloaded the dependent modules of gcc<\/code>.<\/p>\n\n\n\n
The module purge<\/code> command will unload all current modules. This command is useful at the beginning of batch scripts to prevent the batch shell from unintentionally inheriting the loaded module environment from a batch submission shell.<\/p>\n\n\n\n
Software modules on the IBM Power9 worker <\/h3>\n\n\n\n
The IBM Power 9 worker’s ppc64le<\/code> binary format is not compatible with the x86_64<\/code> binary format used by Intel and AMD CPUs. Users must include a sequence of shell commands to access Power9 software when running jobs on the IBM server.<\/p>\n\n\n\n
# setup the ppc64le environment\nunset MODULESHOME MODULEPATH\n . \/etc\/profile.d\/modules.sh\n . \/etc\/profile.d\/z00_lmod.sh\nmodule purge\n<\/code><\/pre>\n\n\n\n
The commands above purges any remaining x86_64<\/code> modules from the shell environment and points the Lmod system to the ppc64le<\/code> packages.<\/p>\n\n\n\n
Python, Jupyter, R, and  conda environments<\/h3>\n\n\n\n
We provide both the community version of the anaconda<\/code> bundle from the anaconda project<\/a> and the open source mambaforge<\/code> package from conda forge<\/a>.<\/p>\n\n\n\n
$ module avail conda mamba\n------------ \/srv\/software\/el8\/x86_64\/hpc\/lmod\/Core ------------\n   anaconda\/2023.07-2    mambaforge\/23.1.0-4<\/code><\/pre>\n\n\n\n
 Users are storongly encouraged to use conda environments to organize and isolate their software projects<\/strong>. Either mambaforge<\/code> or anaconda<\/code> can be used to build and support customized python environments, see the custom environments<\/a> documentation about managing environments. Mambaforge provides the mamba<\/a> package manager a faster more reliable drop-in replacement for conda. Note that anaconda<\/code> comes packaged with a rich bundle<\/a> of python and Jupyter<\/a> modules preinstalled in the base<\/code> environment.<\/p>\n\n\n\n
Import note about HOME and conda on worker nodes<\/h4>\n\n\n\n
The conda<\/code> command is hardwired<\/strong> to search $HOME\/.conda<\/code> under your home directory. The search will fail from batch on worker nodes since your home directory under \/nashome<\/code> is not accessible. Conda will work as long as HOME<\/code> is set to another directory, even a directory that does not exist! A simple workaround to temporarily reset the directory when running conda. The command below was run from a login node. The same command run from a worker node fails due to the \/nashome<\/code> dependency.<\/p>\n\n\n\n
$ conda config --show envs_dirs pkgs_dirs\nenvs_dirs:\n  - \/nashome\/s\/myname\/.conda\/envs\n  - \/srv\/software\/el8\/x86_64\/hpc\/anaconda\/2023.07-2\/envs\npkgs_dirs:\n  - \/srv\/software\/el8\/x86_64\/hpc\/anaconda\/2023.07-2\/pkgs\n  - \/nashome\/s\/myname\/.conda\/pkgs\n<\/code><\/pre>\n\n\n\n
Resetting HOME<\/code> removes the dependency on \/nashome<\/code>,<\/p>\n\n\n\n
$ HOME=\/nowhere conda config --show envs_dirs pkgs_dirs\nenvs_dirs:\n  - \/srv\/software\/el8\/x86_64\/hpc\/anaconda\/2023.07-2\/envs\n  - \/nowhere\/.conda\/envs\npkgs_dirs:\n  - \/srv\/software\/el8\/x86_64\/hpc\/anaconda\/2023.07-2\/pkgs\n  - \/nowhere\/.conda\/pkgs<\/code><\/pre>\n\n\n\n
Note that you can add additional paths to envs_dirs<\/code> and pkgs_dirs<\/code> by respectively setting environment variables CONDA_ENVS_PATH<\/code> and CONDA_PKGS_DIRS<\/code>. You can use these variables to set your default conda env location to your project area in \/work1<\/code>.<\/p>\n\n\n\n
Using conda<\/h4>\n\n\n\n
In the following we temporarily reset HOME<\/code> when running conda<\/code> as explained in the section above. To activate the base<\/code> anaconda<\/code> environment,<\/p>\n\n\n\n
$ module load anaconda\n$ HOME=\/nowhere conda activate\n(base) $ python\nPython 3.11.4\n>> ^D\n(base) $<\/code><\/pre>\n\n\n\n
Use the commands below to deactivate and envionment and remove anaconda from your shell environment.<\/p>\n\n\n\n
(base) $ HOME=\/nowhere conda deactivate\n$ module unload anaconda<\/code><\/pre>\n\n\n\n
The anaconda package on Wilson has been extended to include a environment configured with the gnu R system<\/a> of statistical analysis software. Use the command conda activate r_env<\/code> to access R.<\/p>\n\n\n\n
Julia language<\/h2>\n\n\n\n
The Wilson HPC software includes distributions for the Julia language<\/a>. Julia provides the ease of programming, similar to python, and near C\/C++ execution speeds<\/a>. Like python, functionality is easily extended by a large number of libraries<\/a>. Installed versions include recent production builds and a recent long-term support (LTS) version. The latest production build is suitable for use in new projects.<\/p>\n","protected":false},"excerpt":{"rendered":"
HPC toolchains on the Wilson cluster The HPC toolchains Wilson were built using a combination of EasyBuild recipes in combination with additional hand-built packages. Under AlmaLinux 8 (el8) the deployments are found in the directory \/srv\/software\/el8\/x86_64. The subdirectory .\/eb contains packages built with EasyBuild while directory .\/hpc contains the additional packages built by other means….  More »<\/a><\/p>\n","protected":false},"author":15,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-384","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/computing.fnal.gov\/wilsoncluster\/wp-json\/wp\/v2\/pages\/384","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/computing.fnal.gov\/wilsoncluster\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/computing.fnal.gov\/wilsoncluster\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/computing.fnal.gov\/wilsoncluster\/wp-json\/wp\/v2\/users\/15"}],"replies":[{"embeddable":true,"href":"https:\/\/computing.fnal.gov\/wilsoncluster\/wp-json\/wp\/v2\/comments?post=384"}],"version-history":[{"count":44,"href":"https:\/\/computing.fnal.gov\/wilsoncluster\/wp-json\/wp\/v2\/pages\/384\/revisions"}],"predecessor-version":[{"id":7765,"href":"https:\/\/computing.fnal.gov\/wilsoncluster\/wp-json\/wp\/v2\/pages\/384\/revisions\/7765"}],"wp:attachment":[{"href":"https:\/\/computing.fnal.gov\/wilsoncluster\/wp-json\/wp\/v2\/media?parent=384"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}