Force Field Molecular Dynamics ============================== These are instructions for compiling a variety of atomistic codes. By atomistic codes, we include codes that simulate the behavior of particles such as LAMMPS, codes for Classical Molecular Dynamics (CMD) such as AMBER, GROMACS, and NAMD, Tight Binding codes such as DFTB+ and DFT codes such as ABINIT, OCTOPUS, VASP, and Quantum Espresso. For compiling these codes we need Fortran, C, and C++ compilers, we will use GCC 9.3 and Intel 2019 for most of these codes. A set of numerical libraries are used very often by these codes. Dense Linear Algebra routines such as BLAS and LAPACK are used. Optimized versions such as OpenBLAS and Intel MKL are preferable over the reference versions from NetLIB. All atomistic codes in our list take advantage of parallelization, either OpenMP, MPI, or support GPUs. OpenMP is implemented on modern compilers such as GCC, Intel, and NVIDIA. For MPI we will use MPICH 3.4.1, OpenMPI 3.1.6, and Intel MPI from 2019. Other libraries needed for compiling these codes include an FFT library such as FFTW 3.3.9 or the implementation in MKL. The libraries need to be compiled for single and double precision as some codes use both. An HDF5 and NetCDF libraries provide hierarchical data storage for numerical data. Finally, a Python implementation is often needed, because these codes include a Python interface or use it for building or testing. We will start with PLUMED, a library for sampling algorithms, free-energy calculations, and other high-level calculations on top of MD packages. W Plumed ------ PLUMED is an open-source library that provides a wide range of different methods that work on top of other Molecular Dynamics codes. Capabilities of the code include: * Enhanced-sampling algorithms * Free-energy methods * Tools to analyze the vast amounts of data produced by molecular dynamics (MD) simulations. We compile PLUMED with the following modules:: module load dev/git/2.29.1 dev/cmake/3.21.1 \ lang/gcc/9.3.0 parallel/mpich/3.4.1_gcc93 \ lang/python/cpython_3.10.5_gcc93 \ libs/openblas/0.3.20_gcc93 \ libs/fftw/3.3.9_gcc93 Download PLUMED from:: wget https://github.com/plumed/plumed2/releases/download/v2.8.2/plumed-2.8.2.tgz Uncompress and configure the build:: tar -zxvf plumed-2.8.2.tgz cd plumed-2.8.2 ./configure --prefix=/shared/software/atomistic/plumed/2.8.2_gcc93_mpic341 PYTHON_BIN=python3 --enable-external-blas --enable-external-lapack LDFLAGS=-L${MD_OPENBLAS}/lib LIBS=-lopenblas After configuring the build. Execute:: make -j 12 make install PLUMED includes a test suite and it is always a good practice to test the compilation. Call make to run the test suite:: make check The final lines of the tests show no failures:: ... + check file ves/rt-td-uniform/report.txt for more information + test ves/rt-td-vonmises/ NOT APPLICABLE + check file ves/rt-td-vonmises/report.txt for more information + test ves/rt-VesDeltaF/ NOT APPLICABLE + check file ves/rt-VesDeltaF/report.txt for more information + test ves/rt-VesDeltaF-mwalkers/ NOT APPLICABLE + check file ves/rt-VesDeltaF-mwalkers/report.txt for more information + test ves/rt-waveletgrid/ NOT APPLICABLE + check file ves/rt-waveletgrid/report.txt for more information +++++++++++++++++++++++++++++++++++++++++++++++++++++ + Final report: + 321 tests performed, 268 tests not applicable + 0 errors found + Well done!! +++++++++++++++++++++++++++++++++++++++++++++++++++++ After installation, PLUMED offers a very convenient prepared module file. The file can be copied along with the rest of modulefiles with a simple edit for loading the modules used during compilation:: cp /shared/software/atomistic/plumed/2.8.2_gcc93_mpic341/lib/plumed/modulefile /shared/modulefiles/tier2/atomistic/plumed/2.8.2_gcc93_mpic341 The only change needed for this modulefile is adding one line loading the other modules:: #%Module1.0############################################## # Manually add here dependencies and conflicts module load lang/gcc/9.3.0 parallel/mpich/3.4.1_gcc93 lang/python/cpython_3.10.5_gcc93 libs/openblas/0.3.20_gcc93 libs/fftw/3.3.9_gcc93 ... PLUMED is ready to used and can be integrated with some other MD codes as we will see below AMBER 22 -------- Amber is a suite of biomolecular simulation programs. It is used to simulate large and complex atomic compounds using a set of molecular mechanical force fields for the simulation of biomolecules. In this document, we provide instructions to compile Amber using GCC 9.3, and Intel oneAPI 2023 Amber can be compiled with a variety of options for parallelization. Amber can be compiled as pure serial code, using multithreading with OpenMP, distributed parallelism with MPI, and using GPUs with CUDA. We will be building Amber with each option and one final compilation enabling OpenMP, MPI, and CUDA. Amber as a package is composed of two pieces, Amber itself and AmberTools. Amber facilitates faster simulations (on parallel CPU or GPU hardware) and is distributed with a paid license. AmberTools is a free package that collect open-source code to be used in conjunction with Amber. The version used to produce these notes is Amber 20, the latest version available by mid-2021. We start with two files, ``Amber22.tar.bz2`` and ``AmberTools23.tar.bz2``. The first step is to uncompress both of them. The two compressed tars will uncompress on the same target folder ``amber22_src``:: tar -jxvf amber22.tar.bz2 tar -jxvf AmberTools23.tar.bz2 cd amber20_src We start with GCC 9.3 where most of the tools can be enabled and the compilation is easier. We start with a pure Serial build, we move after to enable OpenMP, after that, we disable OpenMP and enable MPI. We move the building process to a GPU node to compile a CUDA version disabling OpenMP and MPI and we finalize enabling all options for a final build. The reason for doing this is that keeping just one parallelization active will facilitate the usage for casual users, instead of dealing with complex combinations of threads, MPI processes, and GPU cards. GCC 9.3 (Serial) ~~~~~~~~~~~~~~~~ Decompressing ``Amber20.tar.bz2`` and ``AmberTools20.tar.bz2`` will create a folder ``amber20_src``. There is one file ``amber20_src/build/run_cmake.sample`` that we will use as a template changing it for each build from now on. It is always convenient to build the code on a folder separated from the sources. Create a folder for each build, for the serial case, we suggest:: cd amber20_src mkdir build_gcc93_mpic341 cd build_gcc93_mpic341 cp ../amber22_src/build/run_cmake.sample run_cmake The reason for the name is that in this folder we will compile all builds, including the MPI version using MPICH 3.4.1 Inside this folder copy the file ``amber22_src/build/run_cmake.sample`` The orginal content of this file is:: #!/bin/bash # This file gives some sample cmake invocations. You may wish to # edit some options that are chosen here. # For information on how to get cmake, visit this page: # https://ambermd.org/pmwiki/pmwiki.php/Main/CMake-Quick-Start # For information on common options for cmake, visit this page: # http://ambermd.org/pmwiki/pmwiki.php/Main/CMake-Common-Options # (Note that you can change the value of CMAKE_INSTALL_PREFIX from what # is suggested below, but it cannot coincide with the amber20_src # folder.) AMBER_PREFIX=$(dirname $(dirname `pwd`)) if [ `uname -s|awk '{print $1}'` = "Darwin" ]; then # For macOS: if [ -x /Applications/CMake.app/Contents/bin/cmake ]; then cmake=/Applications/CMake.app/Contents/bin/cmake else cmake=cmake fi $cmake $AMBER_PREFIX/amber22_src \ -DCMAKE_INSTALL_PREFIX=$AMBER_PREFIX/amber20 \ -DCOMPILER=CLANG -DBLA_VENDOR=Apple \ -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log else # Assume this is Linux: cmake $AMBER_PREFIX/amber22_src \ -DCMAKE_INSTALL_PREFIX=$AMBER_PREFIX/amber20 \ -DCOMPILER=GNU \ -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log fi if [ ! -s cmake.log ]; then echo "" echo "Error: No cmake.log file created: you may need to edit run_cmake" exit 1 fi echo "" echo "If the cmake build report looks OK, you should now do the following:" echo "" echo " make install" echo " source $AMBER_PREFIX/amber22/amber.sh" echo "" echo "Consider adding the last line to your login startup script, e.g. ~/.bashrc" echo "" We will only make modifications to this file in two locations. We add an extra variable for defining the PREFIX:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/22_gcc93_serial This is the PREFIX that we will use for the Serial compilation The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber22_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=GNU -DBZIP2_LIBRARIES=/shared/software/lang/gcc/9.3.0/lib/libbz2.a \ -DOPENMP=FALSE -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log The lines above are for a purely serial build of Amber. Now we need to load some modules for compiling the code. Amber uses CMAKE as a software builder. The version included with RedHat 7.x (2.18) is too old for most scientific codes. We will load modules for cmake 3.21.1 and GCC 9.3:: module load dev/cmake/3.21.1 lang/gcc/9.3.0 lang/python/cpython_3.10.5_gcc93 parallel/cuda/11.7 There is a bug in the script for Miniconda, that will prevent cmake of building a complete environment. On the line 153 of ``amber20_src/cmake/UseMiniconda.cmake`` add a line to install pip via conda:: execute_process(COMMAND ${CONDA} update conda -y) execute_process(COMMAND ${CONDA} install pip -y) execute_process(COMMAND ${MINICONDA_PYTHON} -m pip install pip --upgrade) If you do skip this step, conda will remove pip after the update and several other python packages that are installed with pip will fail. This is the only change in the sources. No other changes will be done directly to sources, if something fails, we simply disable the corresponding package. Run the ``run_cmake`` inside the corresponding build folder:: cd build_gcc93_mpic341 ./run_cmake When running the script, Miniconda will be downloaded and installed, this is the portion of the execution that requires internet access and cannot be executed from a GPU node. We will preserve this folder for all other builds, in particular those for CUDA that are executed inside GPU nodes with no internet accesss. After CMAKE have prepared the folder for compilation, execute:: make The code will compile in a few minutes. To install the build execute:: make install GCC 9.3 (OpenMP) ~~~~~~~~~~~~~~~~ The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/22_gcc93_openmp This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber22_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=GNU -DBZIP2_LIBRARIES=/shared/software/lang/gcc/9.3.0/lib/libbz2.a \ -DOPENMP=TRUE -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install The running of ``make`` followed by ``make install`` is not necessary, the later will run make and build the binaries before installing them. GCC 9.3 (MPICH 3.4.1) ~~~~~~~~~~~~~~~~~~~~~ We will add a new module:: module load parallel/mpich/3.4.1_gcc93 The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_gcc93_mpic341 This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=GNU -DBZIP2_LIBRARIES=/shared/software/lang/gcc/9.3.0/lib/libbz2.a \ -DOPENMP=FALSE -DMPI=TRUE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install GCC 9.3 (CUDA) ~~~~~~~~~~~~~~ This build is done on a GPU node, request an interactive execution:: qsub -I -l nodes=1:ppn=8:gpus=3 -q comm_gpu_inter Load all the modules used in the previous builds adding the module for CUDA:: module load dev/cmake/3.21.1 lang/gcc/9.3.0 lang/python/cpython_3.9.5_gcc93 parallel/mpich/3.4.1_gcc93 parallel/cuda/11.3 The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_gcc93_cuda113 This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=GNU -DBZIP2_LIBRARIES=/shared/software/lang/gcc/9.3.0/lib/libbz2.a \ -DOPENMP=FALSE -DMPI=FALSE -DCUDA=TRUE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install GCC 9.3 (OpenMP + MPI + CUDA) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This build is done on a GPU node, reuse the previous interactive session or request a new one:: qsub -I -l nodes=1:ppn=8:gpus=3 -q comm_gpu_inter Load all the modules used in the previous builds adding the module for CUDA:: module load dev/cmake/3.21.1 lang/gcc/9.3.0 lang/python/cpython_3.9.5_gcc93 parallel/mpich/3.4.1_gcc93 parallel/cuda/11.3 The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_gcc93_mpic341_cuda113 This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=GNU -DBZIP2_LIBRARIES=/shared/software/lang/gcc/9.3.0/lib/libbz2.a \ -DOPENMP=TRUE -DMPI=TRUE -DCUDA=TRUE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install Once AMBER is compiled and the modulefile created, use a GPU node to run the testsuite. There are two versions of it running AMBER the CUDA tests are either serial or parallel. Here are the results:: ==> /shared/software/atomistic/amber/20_gcc93_mpic341_cuda/logs/test_amber_cuda/2021-05-26_11-31-03.log <== diffing md_SC_NVT_MBAR_SC_2.o.DPFP with md_SC_NVT_MBAR_SC_2.o PASSED ============================================================== make[1]: Leaving directory '/gpfs20/shared/src/AMBER/amber20/amber20_src/test/cuda' Finished CUDA test suite for Amber 20 at Wed May 26 11:40:28 EDT 2021. 242 file comparisons passed 7 file comparisons failed (1 of which can be ignored) 0 tests experienced errors ==> /shared/software/atomistic/amber/20_gcc93_mpic341_cuda/logs/test_amber_cuda_parallel/2021-05-26_11-42-11.log <== Note: The following floating-point exceptions are signalling: IEEE_DENORMAL Note: The following floating-point exceptions are signalling: IEEE_DENORMAL Note: The following floating-point exceptions are signalling: IEEE_DENORMAL diffing mdout.pme.gamd3.GPU_DPFP with mdout.pme.gamd3 PASSED ============================================================== make[1]: Leaving directory '/gpfs20/shared/src/AMBER/amber20/amber20_src/test/cuda' 179 file comparisons passed 43 file comparisons failed (3 of which can be ignored) 2 tests experienced errors GCC 11.1 (Serial) ~~~~~~~~~~~~~~~~~ The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_gcc111_serial This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=GNU -DDISABLE_TOOLS="gbnsr6;cifparse;gbnsr6;sff" \ -DCMAKE_Fortran_FLAGS="-fallow-invalid-boz -fallow-argument-mismatch" \ -DBZIP2_LIBRARIES=/shared/software/lang/gcc/11.1.0/lib/libbz2.a \ -DOPENMP=FALSE -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install GCC 11.1 (OpenMP) ~~~~~~~~~~~~~~~~~ The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_gcc111_openmp This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=GNU -DDISABLE_TOOLS="gbnsr6;cifparse;gbnsr6;sff" \ -DCMAKE_Fortran_FLAGS="-fallow-invalid-boz -fallow-argument-mismatch" \ -DBZIP2_LIBRARIES=/shared/software/lang/gcc/11.1.0/lib/libbz2.a \ -DOPENMP=TRUE -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install GCC 11.1 (OpenMPI 4.1.1) ~~~~~~~~~~~~~~~~~~~~~~~~ Loading the modules:: module load lang/python/cpython_3.9.5_gcc111 parallel/openmpi/4.1.1_gcc111 The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_gcc111_ompi411 This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=GNU -DDISABLE_TOOLS="gbnsr6;cifparse;gbnsr6;sff" \ -DCMAKE_Fortran_FLAGS="-fallow-invalid-boz -fallow-argument-mismatch" \ -DBZIP2_LIBRARIES=/shared/software/lang/gcc/11.1.0/lib/libbz2.a \ -DOPENMP=FALSE -DMPI=TRUE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install Intel Compilers 2021 (Serial) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Loading the modules:: module load dev/cmake/3.21.1 compiler/2021.2.0 mpi/2021.2.0 mkl/2021.2.0 lang/gcc/9.3.0 The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_intel21_serial This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=INTEL -DDISABLE_TOOLS="reduce" \ -DBISON_EXECUTABLE=/shared/software/lang/gcc/9.3.0/bin/bison \ -DOPENMP=FALSE -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install Intel Compilers 2021 (OpenMP) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Loading the modules:: module load dev/cmake/3.21.1 compiler/2021.2.0 mpi/2021.2.0 mkl/2021.2.0 lang/gcc/9.3.0 The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_intel21_openmp This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=INTEL -DDISABLE_TOOLS="reduce" \ -DBISON_EXECUTABLE=/shared/software/lang/gcc/9.3.0/bin/bison \ -DOPENMP=TRUE -DMPI=FALSE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install Intel Compilers 2021 (Intel MPI 2021) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Loading the modules:: module load dev/cmake/3.21.1 compiler/2021.2.0 mpi/2021.2.0 mkl/2021.2.0 lang/gcc/9.3.0 The variable defining the PREFIX will be:: AMBER_PREFIX=$(dirname $(dirname `pwd`)) PREFIX=/shared/software/atomistic/amber/20_intel21_impi21 This is the PREFIX that we will use for the OpenMP compilation. The other change is inside the block for running cmake for the Linux build:: # Assume this is Linux: cmake $AMBER_PREFIX/amber20_src \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCOMPILER=INTEL -DDISABLE_TOOLS="reduce" \ -DBISON_EXECUTABLE=/shared/software/lang/gcc/9.3.0/bin/bison \ -DOPENMP=FALSE -DMPI=TRUE -DCUDA=FALSE -DINSTALL_TESTS=TRUE \ -DDOWNLOAD_MINICONDA=TRUE -DMINICONDA_USE_PY3=TRUE \ 2>&1 | tee cmake.log After this execute:: ./run_cmake make install Intel Compilers 2021 (CUDA) ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Nvidia CUDA compiler nvcc does not support Intel Compilers 2021 as the base compiler for CUDA builds. Running Tests ~~~~~~~~~~~~~ To run the testsuite, the modulefile needs to be created and loaded. The module must set the variable ``$AMBERHOME`` needed to run the tests. Go to the folder ``amber20_src/test`` that contains the tests. For the parallel tests, set the variable ``$DO_PARALLEL`` to the right command for running MPI executions, for example:: export DO_PARALLEL="mpirun -np 4" These are the results of several tests: GCC 9.3 Serial:: $> ./test_amber_serial.sh ... ... Finished serial test suite for Amber 20 at Sat Aug 21 17:52:10 EDT 2021. 196 file comparisons passed 0 file comparisons failed 0 tests experienced errors Test log file saved as /shared/software/atomistic/amber/20_gcc93_serial/logs/test_amber_serial/2021-08-21_17-45-18.log No test diffs to save! GCC 9.3 OpenMP:: $> ./test_amber_serial.sh ... ... Finished serial test suite for Amber 20 at Sat Aug 21 18:04:09 EDT 2021. 196 file comparisons passed 0 file comparisons failed 0 tests experienced errors Test log file saved as /shared/software/atomistic/amber/20_gcc93_openmp/logs/test_amber_serial/2021-08-21_17-57-23.log No test diffs to save! GCC 9.3 CUDA:: $> ./test_amber_cuda_serial.sh ... ... Finished CUDA test suite for Amber 20 at Sat Aug 21 18:15:52 EDT 2021. 243 file comparisons passed 6 file comparisons failed (1 of which can be ignored) 0 tests experienced errors Test log file saved as /shared/software/atomistic/amber/20_gcc93_cuda113/logs/test_amber_cuda/2021-08-21_18-12-26.log Test diffs file saved as /shared/software/atomistic/amber/20_gcc93_cuda113/logs/test_amber_cuda/2021-08-21_18-12-26.diff GCC 9.3 MPICH 3.4.1:: $> export DO_PARALLEL="mpirun -np 8" $> ./test_amber_parallel.sh ... ... Finished parallel test suite for Amber 20 on Sat Aug 21 18:31:15 EDT 2021. Some tests require 4 threads to run, while some will not run with more than 2. Please run further parallel tests with the appropriate number of processors. See /shared/software/atomistic/amber/20_gcc93_mpic341/test/README. 274 file comparisons passed 4 file comparisons failed (1 of which can be ignored) 0 tests experienced an error Test log file saved as /shared/software/atomistic/amber/20_gcc93_mpic341/logs/test_amber_parallel/2021-08-21_18-27-41.log Test diffs file saved as /shared/software/atomistic/amber/20_gcc93_mpic341/logs/test_amber_parallel/2021-08-21_18-27-41.diff GCC 9.3 OpenMP + MPI + CUDA:: $> cat $PBS_NODEFILE | uniq > NODEFILE $> export DO_PARALLEL="mpirun -np 2 -f /shared/src/AMBER/amber20/amber20_src/test/NODEFILE" $> ./test_amber_cuda_parallel.sh ... ... 135 file comparisons passed 13 file comparisons failed (1 of which can be ignored) 0 tests experienced errors Test log file saved as /shared/software/atomistic/amber/20_gcc93_mpic341_cuda113/logs/test_amber_cuda_parallel/2021-08-21_19-27-18.log Test diffs file saved as /shared/software/atomistic/amber/20_gcc93_mpic341_cuda113/logs/test_amber_cuda_parallel/2021-08-21_19-27-18.diff Gromacs ------- Gromacs is a Classical Molecular Dynamics code. The version compiled was 2021.2 Several versions were compiled using GCC 9.3 and 11.1 Gromacs 2021.2 on Thorny Flat ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The download page is:: https://manual.gromacs.org It is a good practice to compile from a separate folder instead of compiling directly alongside with the sources, create a folder ``build_gcc93_mpic341`` inside the sources:: wget https://ftp.gromacs.org/gromacs/gromacs-2021.2.tar.gz tar -zxvf gromacs-2021.2.tar.gz cd gromacs-2021.2/ mkdir build_gcc93_mpic341 cd build_gcc93_mpic341 Cmake is only used during configuration and it is not needed at runtime The modules can be loaded with this command line:: module purge module load lang/gcc/9.3.0 parallel/mpich/3.4.1_gcc93 dev/cmake/3.18.3 \ lang/python/cpython_3.9.5_gcc93 libs/openblas/0.3.13_gcc93 The first configuration is the standard one (Single Precision) The cmake configuration line was:: cmake -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_MPI=on \ -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx \ -DGMX_LAPACK_USER="-L${MD_OPENBLAS}/lib -lopenblas" \ -DGMX_BLAS_USER="-L${MD_OPENBLAS}/lib -lopenblas" \ -DCMAKE_INSTALL_PREFIX=/shared/software/atomistic/gromacs/2021.2_gcc93_mpic341 .. The results of the tests were:: ... ... 70/73 Test #70: regressiontests/complex ............... Passed 135.38 sec Start 71: regressiontests/freeenergy 71/73 Test #71: regressiontests/freeenergy ............ Passed 34.09 sec Start 72: regressiontests/rotation 72/73 Test #72: regressiontests/rotation .............. Passed 28.83 sec Start 73: regressiontests/essentialdynamics 73/73 Test #73: regressiontests/essentialdynamics ..... Passed 10.23 sec 100% tests passed, 0 tests failed out of 73 Label Time Summary: GTest = 108.89 sec*proc (67 tests) IntegrationTest = 32.96 sec*proc (20 tests) MpiTest = 52.50 sec*proc (10 tests) SlowTest = 57.26 sec*proc (8 tests) UnitTest = 18.67 sec*proc (39 tests) Total Test time (real) = 317.75 sec [100%] Built target run-ctest-nophys Scanning dependencies of target check [100%] Built target check The second configuration enables the double precision for gromacs: The cmake configuration line was:: cmake -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_MPI=on -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_INSTALL_PREFIX=/shared/software/atomistic/gromacs/2021.2_double_gcc93_mpic341 -DGMX_LAPACK_USER="-L${MD_OPENBLAS}/lib -lopenblas" -DGMX_BLAS_USER="-L${MD_OPENBLAS}/lib -lopenblas" -DGMX_DOUBLE=on .. The results of the tests were:: 98% tests passed, 1 tests failed out of 46 Label Time Summary: GTest = 117.18 sec*proc (40 tests) IntegrationTest = 13.81 sec*proc (5 tests) MpiTest = 2.60 sec*proc (3 tests) SlowTest = 12.88 sec*proc (1 test) UnitTest = 90.49 sec*proc (34 tests) Total Test time (real) = 2075.93 sec Gromacs 5.1.5 on Thorny Flat ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The modules used were:: module load lang/intel/2018 dev/cmake/3.18.3 libs/boost/1.73 Cmake is only used during configuration and it is not needed at runtime The configuration line for cmake is executed on a folder created to contain the compiled code:: mkdir build_intel18 cd build_intel18 cmake -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_MPI=on \ -DCMAKE_C_COMPILER=mpiicc -DCMAKE_CXX_COMPILER=mpiicpc \ -DCMAKE_INSTALL_PREFIX=/shared/software/atomistic/gromacs/5.1.5_intel18 .. make -j12 make check make install A similar compilation was done using Intel 2019 compilers. One test fails from the test suite:: 96% tests passed, 1 tests failed out of 26 Label Time Summary: GTest = 2.29 sec*proc (17 tests) IntegrationTest = 2.01 sec*proc (2 tests) MpiIntegrationTest = 0.56 sec*proc (1 test) UnitTest = 2.29 sec*proc (17 tests) Total Test time (real) = 120.90 secs. The following tests FAILED: 17 - SelectionUnitTests (Failed) Gromacs 2021.3 on Spruce Knob ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The modules used were:: $> module purge $> module load lang/gcc/9.3.0 parallel/mpich/3.3.2_gcc93 dev/cmake/3.15.2 lang/python/cpython_3.9.7_gcc93 libs/openblas/0.3.10_gcc93 Download and uncompress the sources:: $> wget https://ftp.gromacs.org/gromacs/gromacs-2021.3.tar.gz $> tar -zxvf gromacs-2021.3.tar.gz $> cd gromacs-2021.3 $> mkdir build_gcc93 $> cd build_gcc93 Cmake is only used during configuration and it is not needed at runtime The first configuration is the standard one (Single Precision) The cmake configuration line was:: cmake -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_MPI=on \ -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx \ -DGMX_LAPACK_USER="-L${MD_OPENBLAS}/lib -lopenblas" \ -DGMX_BLAS_USER="-L${MD_OPENBLAS}/lib -lopenblas" \ -DCMAKE_INSTALL_PREFIX=/shared/software/atomistic/gromacs/2021.3_gcc93_mpic332 .. After configuration compile and install the code:: $> make $> make install The results of the tests were:: $> make check 67/73 Test #67: GmxapiMpiTests ........................ Passed 2.46 sec Start 68: GmxapiInternalInterfaceTests 68/73 Test #68: GmxapiInternalInterfaceTests .......... Passed 0.76 sec Start 69: GmxapiInternalsMpiTests 69/73 Test #69: GmxapiInternalsMpiTests ............... Passed 0.86 sec Start 70: regressiontests/complex 70/73 Test #70: regressiontests/complex ............... Passed 45.75 sec Start 71: regressiontests/freeenergy 71/73 Test #71: regressiontests/freeenergy ............ Passed 14.99 sec Start 72: regressiontests/rotation 72/73 Test #72: regressiontests/rotation .............. Passed 10.50 sec Start 73: regressiontests/essentialdynamics 73/73 Test #73: regressiontests/essentialdynamics ..... Passed 4.22 sec 100% tests passed, 0 tests failed out of 73 Label Time Summary: GTest = 105.49 sec*proc (67 tests) IntegrationTest = 22.65 sec*proc (20 tests) MpiTest = 68.16 sec*proc (10 tests) SlowTest = 79.18 sec*proc (8 tests) UnitTest = 3.66 sec*proc (39 tests) Total Test time (real) = 181.11 sec make[3]: Leaving directory '/gpfs/shared/src/gromacs-2021.3/build_gcc93' [100%] Built target run-ctest-nophys make[3]: Entering directory '/gpfs/shared/src/gromacs-2021.3/build_gcc93' Scanning dependencies of target check make[3]: Leaving directory '/gpfs/shared/src/gromacs-2021.3/build_gcc93' [100%] Built target check make[2]: Leaving directory '/gpfs/shared/src/gromacs-2021.3/build_gcc93' make[1]: Leaving directory '/gpfs/shared/src/gromacs-2021.3/build_gcc93' The second configuration enables double precision floating point numbers. The cmake configuration line was:: cmake -DGMX_BUILD_OWN_FFTW=ON -DREGRESSIONTEST_DOWNLOAD=ON -DGMX_MPI=on \ -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx \ -DGMX_LAPACK_USER="-L${MD_OPENBLAS}/lib -lopenblas" \ -DGMX_BLAS_USER="-L${MD_OPENBLAS}/lib -lopenblas" \ -DCMAKE_INSTALL_PREFIX=/shared/software/atomistic/gromacs/2021.3_double_gcc93_mpic332 -DGMX_DOUBLE=on .. The results of the tests were:: $> make check Start 65: MdrunSimulatorComparison 65/73 Test #65: MdrunSimulatorComparison .............. Passed 5.42 sec Start 66: GmxapiExternalInterfaceTests 66/73 Test #66: GmxapiExternalInterfaceTests .......... Passed 3.32 sec Start 67: GmxapiMpiTests 67/73 Test #67: GmxapiMpiTests ........................ Passed 3.19 sec Start 68: GmxapiInternalInterfaceTests 68/73 Test #68: GmxapiInternalInterfaceTests .......... Passed 0.98 sec Start 69: GmxapiInternalsMpiTests 69/73 Test #69: GmxapiInternalsMpiTests ............... Passed 1.42 sec Start 70: regressiontests/complex 70/73 Test #70: regressiontests/complex ............... Passed 49.69 sec Start 71: regressiontests/freeenergy 71/73 Test #71: regressiontests/freeenergy ............ Passed 16.65 sec Start 72: regressiontests/rotation 72/73 Test #72: regressiontests/rotation .............. Passed 12.70 sec Start 73: regressiontests/essentialdynamics 73/73 Test #73: regressiontests/essentialdynamics ..... Passed 4.17 sec 100% tests passed, 0 tests failed out of 73 Label Time Summary: GTest = 138.11 sec*proc (67 tests) IntegrationTest = 34.03 sec*proc (20 tests) MpiTest = 89.03 sec*proc (10 tests) SlowTest = 100.02 sec*proc (8 tests) UnitTest = 4.06 sec*proc (39 tests) Total Test time (real) = 221.48 sec make[3]: Leaving directory '/gpfs/shared/src/gromacs-2021.3/build_double_gcc93' [100%] Built target run-ctest-nophys make[3]: Entering directory '/gpfs/shared/src/gromacs-2021.3/build_double_gcc93' Scanning dependencies of target check make[3]: Leaving directory '/gpfs/shared/src/gromacs-2021.3/build_double_gcc93' [100%] Built target check make[2]: Leaving directory '/gpfs/shared/src/gromacs-2021.3/build_double_gcc93' make[1]: Leaving directory '/gpfs/shared/src/gromacs-2021.3/build_double_gcc93' LAMMPS ------ LAMMPS is a packages for atomistic and particle simulations. The latests stable version by the time (May 2021) is from October 29, 2020. LAMMPS was compiled using these modules:: lang/gcc/11.1.0 parallel/openmpi/3.1.6_gcc111 libs/fftw/3.3.9_gcc111 libs/hdf5/1.12.0_gcc111 LAMMPS was compiled using GCC 11.1, OpenMPI 3.4.1, FFTW 3.3.9, and HDF5 1.12 The first step is to download the code from:: wget https://lammps.sandia.gov/tars/lammps-29Oct20.tar.gz Uncompress the code:: tar -zxvf lammps-29Oct20.tar.gz Change to the src folder inside the uncompressed folder:: cd lammps-29Oct20/src You need a customized Makefile for compiling LAMMPS with the right compilers and libraries. The file is called Makefile.gcc111_ompi316 and must be located at ``src/MAKE``, the content of the file follows:: # mpi = MPI with its default compiler SHELL = /bin/sh # --------------------------------------------------------------------- # compiler/linker settings # specify flags and libraries needed for your compiler CC = mpicxx CCFLAGS = -g -O3 SHFLAGS = -fPIC DEPFLAGS = -M LINK = mpicxx LINKFLAGS = -g -O3 LIB = SIZE = size ARCHIVE = ar ARFLAGS = -rc SHLIBFLAGS = -shared # --------------------------------------------------------------------- # LAMMPS-specific settings, all OPTIONAL # specify settings for LAMMPS features you will use # if you change any -D setting, do full re-compile after "make clean" # LAMMPS ifdef settings # see possible settings in Section 3.5 of the manual LMP_INC = -DLAMMPS_GZIP -DLAMMPS_MEMALIGN=64 # -DLAMMPS_CXX98 # MPI library # see discussion in Section 3.4 of the manual # MPI wrapper compiler/linker can provide this info # can point to dummy MPI library in src/STUBS as in Makefile.serial # use -D MPICH and OMPI settings in INC to avoid C++ lib conflicts # INC = path for mpi.h, MPI compiler settings # PATH = path for MPI library # LIB = name of MPI library MPI_INC = -DMPICH_SKIP_MPICXX -DOMPI_SKIP_MPICXX=1 MPI_PATH = MPI_LIB = # FFT library # see discussion in Section 3.5.2 of manual # can be left blank to use provided KISS FFT library # INC = -DFFT setting, e.g. -DFFT_FFTW, FFT compiler settings # PATH = path for FFT library # LIB = name of FFT library FFT_INC = -DFFT_FFTW3 FFT_PATH = FFT_LIB = -L${MD_FFTW}/lib -lfftw3 # JPEG and/or PNG library # see discussion in Section 3.5.4 of manual # only needed if -DLAMMPS_JPEG or -DLAMMPS_PNG listed with LMP_INC # INC = path(s) for jpeglib.h and/or png.h # PATH = path(s) for JPEG library and/or PNG library # LIB = name(s) of JPEG library and/or PNG library JPG_INC = -I${MD_GCC}/include JPG_PATH = -L${MD_GCC}/lib JPG_LIB = -lpng -ljpeg -lz # --------------------------------------------------------------------- # build rules and dependencies # do not edit this section include Makefile.package.settings include Makefile.package EXTRA_INC = $(LMP_INC) $(PKG_INC) $(MPI_INC) $(FFT_INC) $(JPG_INC) $(PKG_SYSINC) EXTRA_PATH = $(PKG_PATH) $(MPI_PATH) $(FFT_PATH) $(JPG_PATH) $(PKG_SYSPATH) EXTRA_LIB = $(PKG_LIB) $(MPI_LIB) $(FFT_LIB) $(JPG_LIB) $(PKG_SYSLIB) EXTRA_CPP_DEPENDS = $(PKG_CPP_DEPENDS) EXTRA_LINK_DEPENDS = $(PKG_LINK_DEPENDS) # Path to src files vpath %.cpp .. vpath %.h .. # Link target $(EXE): main.o $(LMPLIB) $(EXTRA_LINK_DEPENDS) $(LINK) $(LINKFLAGS) main.o $(EXTRA_PATH) $(LMPLINK) $(EXTRA_LIB) $(LIB) -o $@ $(SIZE) $@ # Library targets $(ARLIB): $(OBJ) $(EXTRA_LINK_DEPENDS) @rm -f ../$(ARLIB) $(ARCHIVE) $(ARFLAGS) ../$(ARLIB) $(OBJ) @rm -f $(ARLIB) @ln -s ../$(ARLIB) $(ARLIB) $(SHLIB): $(OBJ) $(EXTRA_LINK_DEPENDS) $(CC) $(CCFLAGS) $(SHFLAGS) $(SHLIBFLAGS) $(EXTRA_PATH) -o ../$(SHLIB) \ $(OBJ) $(EXTRA_LIB) $(LIB) @rm -f $(SHLIB) @ln -s ../$(SHLIB) $(SHLIB) # Compilation rules %.o:%.cpp $(CC) $(CCFLAGS) $(SHFLAGS) $(EXTRA_INC) -c $< # Individual dependencies depend : fastdep.exe $(SRC) @./fastdep.exe $(EXTRA_INC) -- $^ > .depend || exit 1 fastdep.exe: ../DEPEND/fastdep.c cc -O -o $@ $< sinclude .depend The file should be located inside "src/MAKE" or "src/MAKE/MACHINES". Now inside the "src" folder, there is a Makefile that allow you to select which packages will be compiled along side LAMMPS. A good selection comes from adding all followed by removing those depend on libraries and after adding a few:: make yes-all make no-lib make yes-user-reaxc make yes-user-molfile A few external subpackages must be configured first. We want to add HDF5 and COLVARS with the following lines compiling the corresponding subpackages and enabling them for LAMMPS:: cd ../lib/h5md make -f Makefile.h5cc cd ../../src/ make yes-user-h5md make lib-colvars args="-m mpi" make yes-user-colvars And LAMMPS itself after that:: make gcc111_ompi316 After compiled the binary is called ``lmp_gcc82_ompi31`` For testing the build, you can use one of the benchmarks inside the `bench` folder. The benchmark runs on one of the compute nodes using 40 cores. The simulation involves more than 10 million atoms. The command line is:: mpirun -np 40 lmp_mpi -var x 4 -var y 8 -var z 10 -in in.rhodo.scaled This is the final output:: Loop time of 303.71 on 40 procs for 100 steps with 10240000 atoms Performance: 0.057 ns/day, 421.820 hours/ns, 0.329 timesteps/s 99.1% CPU use with 40 MPI tasks x no OpenMP threads MPI task timing breakdown: Section | min time | avg time | max time |%varavg| %total --------------------------------------------------------------- Pair | 201.28 | 205.41 | 210.7 | 15.7 | 67.63 Bond | 9.4133 | 9.5518 | 9.7418 | 2.2 | 3.15 Kspace | 31.517 | 36.816 | 40.964 | 36.8 | 12.12 Neigh | 36.55 | 36.561 | 36.571 | 0.1 | 12.04 Comm | 1.3529 | 1.5481 | 1.6872 | 7.0 | 0.51 Output | 0.0038965 | 0.0040967 | 0.0043541 | 0.1 | 0.00 Modify | 12.757 | 13.067 | 13.53 | 5.7 | 4.30 Other | | 0.7489 | | | 0.25 Nlocal: 256000.0 ave 256004 max 255996 min Histogram: 8 0 0 0 0 24 0 0 0 8 Nghost: 163342.0 ave 163347 max 163335 min Histogram: 8 0 8 0 0 0 8 0 0 16 Neighs: 9.62247e+07 ave 9.65195e+07 max 9.59192e+07 min Histogram: 4 4 0 8 0 8 8 0 4 4 Total # of neighbors = 3.8489892e+09 Ave neighs/atom = 375.87785 Ave special neighs/atom = 7.4318750 Neighbor list builds = 11 Dangerous builds = 0 Total wall time: 0:05:13 The table below shows the timings using the different builds created. +--------------------------------------------+--------------------------+ | Module | Total wall time | +--------------------------------------------+--------------------------+ | atomistic/lammps/2020.10.29_gcc111_impi19 | 0:04:54 | | atomistic/lammps/2020.10.29_gcc111_ompi316 | 0:05:13 | | atomistic/lammps/2020.10.29_gcc93_mpic341 | 0:05:11 | +--------------------------------------------+--------------------------+