Overview
NGA requires five libraries: LAPACK, SUNDIALS, FFTW, HYPRE, and some version of MPI. These libraries must be installed prior to compiling NGA. The MPI library only needs to be installed if it has not been on the machine you are using; note that all clusters already have an MPI library installed.
It is very very important that you use the same compiler version (and MPI version) when installing these libraries and NGA. There are many versions installed on the clusters, which can be viewed using the module avail command and loaded using the module load command. The recommended versions are as follows:
On mesabi in MSI : Intel/2018
If you want to use Open MPI, then
module load intel/2018
module load ompi/3.1.2/intel-2018-release-centos7
module load cmake
or just source the setting environment file
source NGA_modules
If you want to use Intel MPI, then
module load gcc/7.2.0
module load intel/2018
module load impi/2018/release_multithread
module load cmake
or just source the setting environment file
source NGA_modules_impi
On tigercpu (New Tiger) : Intel 18.0 (default)
module load intel
module load intel-mpi
On tiger (Old Tiger) : Intel 16.0
module load intel/16.0
module load openmpi/intel-16.0
On your workstation : installed version of Intel may vary, to load the default
module load intel
You can add the appropriate module load commands to your ~/.bashrc file so the appropriate modules are loaded every time you log in to the cluster or open a new terminal on your workstation. But this may prevent you to properly use MSI NICE/NX nodes for visualization. That’s why NGA_modules and NGA_modules_impi are provided in our enet shared folder R:\m-y-crfel\COMMON\NGA-installation\LibraryInstallation.
The libraries can either be installed through a single automated process, as described in the next section, or individually and manually, as described in subsequent sections.
Automated Installation
For convenience, the Bash script NGA-Library-Installer.sh (for Open MPI) and NGA-Library-Installer-impi.sh (for Intel MPI) have been provided in our enet shared folder R:\m-y-crfel\COMMON\NGA-installation\LibraryInstallation. They will automatically install the required libraries for NGA (including the serial and parallel versions of HYPRE, excluding Open MPI). Copy these scripts to your home directory on the machine where you are installing the libraries, and then enable executing the installation script with:
chmod 755 NGA-Library-Installer.sh
chmod 755 NGA-Library-Installer-impi.sh
Then execute the installation:
If Open MPI, then
./NGA-Library-Installer.sh msi
If Intel MPI, then
./NGA-Library-Installer-impi.sh msi
where “msi” can be replaced by “cluster” or “workstation”, depending on where you are installing the libraries. The script uses Intel version 18.0 on msi and Intel version 19.0 on cluster. Note that if you are installing on your workstation you may need to manually install a version of MPI (see below) prior to running the automated installation script for the other libraries.
This script updates your NGA_modules file (if Open MPI) or NGA_modules_impi file (if Intel MPI), and has already source it, so you don’t need to source it again.
to make sure these changes are reflected in your working terminal. If the install script completed with no errors, the libraries are now installed, and you can move on to installing NGA. If Open MPI, you can verify installation by checking in your /home/suo-yang/$USER/opt/ (if MSI) or /home/$USER/opt/ (if cluster) or /opt/ (if workstation) directory, which should have subdirectories where lapack, hypre, hypre (with openmp), fftw, and sundials were installed. The specific files that should be within these directories are listed at the end of each section of the manual installation instructions. If Intel MPI, just replace the opt above by opt-impi and check the same items.
The script can also be used to uninstall the libraries. To do this, add uninstall to the end of the command above. For example,
./NGA-Library-Installer.sh msi uninstall
An old version of the installation bash script NGA-Library-Installer-13.sh has also been provided, but its use is deprecated. It works with old versions of the compilers and libraries that may no longer be available. (intel-13.0, lapack-3.5.0, fftw-3.3.4, sundials-2.6.1, hypre-2.10.0b).
Manual Installation
Detailed instructions for manually installing these libraries are given below. The MPI library only needs to be installed if it has not been on the machine you are using; note that all clusters already have an MPI library installed.
The compiler flags in the instructions below assume the use of Intel compilers on Tiger or local CTRFL workstations. These instructions will need to be modified for other systems and software. Ask Prof. M or another knowledgeable individual.
Unless otherwise indicated, all commands should be performed as the normal user.
On Tiger, load the latest versions of the compiler and MPI libraries using the module load command as described above.
For the SUNDIALS installation, you will also want the latest version of Python 2.x (2.7.8 as of 08/17 on Tiger). On Old Tiger, execute module load python/2.7. On New Tiger (tigercpu), execute module load anaconda.
You can view the latest versions of the above through module avail.
Also, you may need to install g++ through yum install gcc-c++ if your personal workstation does not have this compiler.
LAPACK
LAPACK (Linear Algebra PACKage) is a library of various numerical linear algebra routines. The latest version of LAPACK (3.7.1 as of 08/17) can be downloaded from http://www.netlib.org/lapack.
Download and unpack the archive by executing the following:
cd /home/$USER/Downloads
wget <link address>
tar -zxf lapack-3.7.1.tgz
cd lapack-3.7.1
In the new directory, copy the file make.inc.example to make.inc. Open this file and change the Fortran compiler options to the following:
FORTRAN = ifort
OPTS = -O3 -xhost -ip
NOOPT =
LOADER = ifort
LOADOPTS = -O3 -xhost -ip
ARCH = xiar
In addition, the TIMER option will need to be changed. Comment out the current setting, TIMER = INT_ETIME, and re-enable TIMER = EXT_ETIME, which is compatible with the ifort compiler. There are additional options for a C compiler, but these do not need to be changed since NGA does not use the C interfaces to LAPACK.
NGA will need the BLAS (Basic Linear Algebra Subprograms) library for simple matrix/vector math such as multiplication, etc., and the LAPACK library. These libraries must be named, libblas.a and liblapack.a, respectively. Therefore, edit the following options in make.inc:
BLASLIB = ../../libblas.a
LAPACKLIB = liblapack.a
To speed up compiling, you should compile in parallel. Open Makefile and edit the instructions under the target blaslib to be $(MAKE) -C BLAS -j 4 to compile with 4 cores (if installing on Tiger you can specify 8 cores, as Tiger has 16 cores/node). Similarly, edit the instructions under the target lapacklib to be $(MAKE) -C SRC -j 4.
The compilation of the two libraries is done in two steps in the following order:
- To compile BLAS, execute
make blaslib - To compile LAPACK, execute
make lapacklib
To complete the installation of these libraries, copy libblas.a and liblapack.a to a convenient location through
sudo cp libblas.a liblapack.a /opt/lapack/lib
or
cp libblas.a liblapack.a /home/$USER/opt/lapack/lib
for installation on a local workstation/laptop (superuser access available) or on the Tiger cluster (no superuser access available), respectively.
FFTW
FFTW (Fastest Fourier Transform in the West) is a library for computing discrete Fourier transforms. The latest version of FFTW (3.3.6 as of 08/17) can be downloaded from http://www.fftw.org.
Download and unpack the archive. The compilers and compiler flags are set with environment variables. From the command line, execute the following commands:
export CC=icc
export CXX=icpc
export F77=ifort
export AR=xiar
export LD=xild
export CFLAGS="-O3 -xhost -ip"
export CXXFLAGS="-O3 -xhost -ip"
export FFLAGS="-O3 -xhost -ip"
To automatically generate a Makefile for FFTW, run the configure script. The option prefix will dictate where the code is installed. For example, depending on superuser access, execute either
./configure --prefix=/opt/fftw --enable-openmp
or
./configure --prefix=/home/$USER/opt/fftw --enable-openmp
To compile the FFTW library in parallel (on Tiger you can use -j 8), execute
make -j 4
and, to install the FFTW library, execute
sudo make install
or
make install
depending on the availability of superuser access.
Upon successful installation, the files fftw3.h and fftw3.f should appear in the directory /home/$USER/opt/fftw/include, and the files libfftw3.a and libfftw3_omp.a should appear in the directory /home/$USER/opt/fftw/lib.
CVODE (SUNDIALS)
CVODE, which is part of the SUNDIALS package, is a library for solving stiff systems of initial value ordinary differential equations and is used with detailed chemistry. The latest compatible version of SUNDIALS (2.7.0 as of 08/17) can be downloaded from http://computation.llnl.gov/projects/sundials/sundials-software. Note that SUNDIALS 3.0.0 removed the FCVDENSE function that is used in NGA, so all SUNDIALS versions starting with a 3 are incompatible without modifying the source code. If you do not want to deal with CMake, follow the instructions for SUNDIALS 2.5.0.
SUNDIALS 2.7.0
For installation on your local machine, you will need CMake 2.8.1 or higher (install locally using yum install cmake). Note that the Tiger cluster will already have CMake installed. Also, you will want a version of Python that is at least 2.7.x (see very top of this page).
Download and unpack the archive. If you are installing on Tiger, go to the website above and right-click on the SUNDIALS link. Select “Copy link address”. Then on Tiger execute:
cd /home/$USER/Downloads
wget <link address>
where you will paste the link address by clicking with your middle mouse button. Unpack the .tar.gz file:
tar -zxf sundials-2.7.0.tar.gz
Create a sundials directory along with a builddir subdirectory. Depending on superuser access, execute either:
sudo mkdir -p /opt/sundials/builddir
cd /opt/sundials/builddir
or
mkdir -p /home/$USER/opt/sundials/builddir
cd /home/$USER/opt/sundials/builddir
Then create bash variables to store the paths to the C and Fortran compilers. In the command-line, type
ccpath="$(which icc)"
fcpath="$(which ifort)"
Now you will use the CMake to generate the Makefile. Execute the following:
sudo cmake -DCMAKE_INSTALL_PREFIX=/opt/sundials \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$ccpath \
-DCMAKE_C_FLAGS_RELEASE=-O3 -xhost -ip \
-DCMAKE_Fortran_COMPILER=$fcpath \
-DCMAKE_Fortran_FLAGS_RELEASE=-O3 -xhost -ip \
-DEXAMPLES_ENABLE=OFF \
-DFCMIX_ENABLE=ON \
/home/$USER/Downloads/sundials-2.7.0
or
cmake -DCMAKE_INSTALL_PREFIX=/home/$USER/opt/sundials \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$ccpath \
-DCMAKE_C_FLAGS_RELEASE=-O3 -xhost -ip \
-DCMAKE_Fortran_COMPILER=$fcpath \
-DCMAKE_Fortran_FLAGS_RELEASE=-O3 -xhost -ip \
-DEXAMPLES_ENABLE=OFF \
-DFCMIX_ENABLE=ON \
/home/$USER/Downloads/sundials-2.7.0
or
cmake -DCMAKE_INSTALL_PREFIX=$HOME/opt/sundials \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$ccpath \
-DCMAKE_C_FLAGS_RELEASE=-O3 -xhost -ip \
-DCMAKE_Fortran_COMPILER=$fcpath \
-DCMAKE_Fortran_FLAGS_RELEASE=-O3 -xhost -ip \
-DEXAMPLES_ENABLE=OFF \
-DFCMIX_ENABLE=ON \
$HOME/Downloads/sundials-2.7.0
depending on superuser access.
Execute make install -j 4 to compile in parallel (on Tiger you can use -j 8) and install the library. If the installation directory requires superuser permissions for writing, the latter command must be executed as superuser.
Upon successful installation, the libraries libsundials_cvode.a, libsundials_fcvode.a, libsundials_fnvecserial.a, and libsundials_nvecserial.a should be in the lib sub-directory of the directory in which you installed SUNDIALS.
Lastly, you will need to add
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/$USER/opt/sundials/lib
to ~/.bashrc. Don’t forget to call source ~/.bashrc in your working terminal.
SUNDIALS 2.5.0
Download and unpack the archive. The compilers and compiler flags are set with environment variables. From the command line, execute the following commands:
export CC=icc
export CXX=icpc
export F77=ifort
export CFLAGS="-O3 -xhost -ip"
export CXXFLAGS="-O3 -xhost -ip"
export FFLAGS="-O3 -xhost -ip"
To automatically generate a Makefile for SUNDIALS, run the configure script. The option prefix will dictate where the code is installed. For example, depending on superuser access, execute either
./configure --prefix=/opt/sundials
or
./configure --prefix=/home/$USER/opt/sundials
To compile the SUNDIALS library, execute (on Tiger you can use -j 8):
make -j 4
and, to install the SUNDIALS library, execute
make install
If the installation directory requires superuser permissions for writing, the latter command must be executed as superuser.
Upon successful installation, the libraries libsundials_cvode.a, libsundials_fcvode.a, libsundials_fnvecserial.a, and libsundials_nvecserial.a should be in the lib sub-directory of the directory in which you installed SUNDIALS.
Open MPI
Message Passing Interface (MPI) is a standard for libraries that allow code parallelization on distributed memory systems. Only install an MPI implementation if one is not already installed on the machine you are using. All clusters, including Tiger, already have an MPI implementation installed. The MPI compilers are required to compile NGA (and the HYPRE library in the next step), so you should include the line module load openmpi/intel-16.0 in your ~/.bashrc file.
You will need to install an MPI library on CTRFL workstations if it is not already installed. Open MPI is the recommended distribution, and the latest version (2.1.1 as of 08/17) can be downloaded from http://open-mpi.org.
Download and unpack the archive. To automatically generate a Makefile for Open MPI, run the configure script. The option prefix will dictate where the code is installed. For example, depending on superuser access, execute either
./configure CC=icc CXX=icpc F77=ifort FC=ifort \
CFLAGS="-O3 -xhost -ip" CXXFLAGS="-O3 -xhost -ip" \
FFLAGS="-O3 -xhost -ip" FCFLAGS="-O3 -xhost -ip" \
--prefix=/opt/openmpi
or
./configure CC=icc CXX=icpc F77=ifort FC=ifort \
CFLAGS="-O3 -xhost -ip" CXXFLAGS="-O3 -xhost -ip" \
FFLAGS="-O3 -xhost -ip" FCFLAGS="-O3 -xhost -ip" \
--prefix=/home/$USER/opt/openmpi
To compile the Open MPI library, execute (on Tiger you can use -j 8):
make -j 4 all
and, to install the Open MPI library, execute
make install
If the installation directory requires superuser permissions for writing, the latter command must be executed as superuser.
Upon successful installation, the binaries mpicc, mpicxx, mpif77, mpif90, and mpifort should be in the bin sub-directory of the directory in which you installed Open MPI.
Note
If you are installing NGA on a workstation machine or you have installed your own copy of Open MPI, you’ll need to add Open MPI to your bash profile. Open the hidden ~/.bashrc file and add the following:
export PATH=$PATH:/home/$USER/opt/openmpi/bin
The directory for the bin path may not appear exactly as above; it depends on where Open MPI was installed.
Additionally, add the libraries in a similar fashion as above, using:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/$USER/opt/openmpi/lib
HYPRE
HYPRE (Scalable Linear Solvers and Multigrid Methods) is a library for solving linear systems in parallel with various iterative methods (Krylov and multigrid). The latest version of HYPRE (2.11.2 as of 08/17) can be downloaded from http://computation.llnl.gov/casc/hypre.
Download and unpack the archive. Note that an MPI library must be installed prior to the installation of HYPRE. Once the MPI library is installed, execute
source ~/.bashrc
from the home directory. The compilers and compiler flags are set with environment variables. From the command line, execute the following commands:
unset AR
unset LD
export CC=mpicc
export CXX=mpicxx
export FC=mpifort
export CFLAGS="-O3 -xhost -ip"
export CXXFLAGS="-O3 -xhost -ip"
export FCFLAGS="-O3 -xhost -ip"
Note that the first two lines ensure that AR is not set to xiar and LD is not set to xild, otherwise an error ar: illegal option -- H will prevent compilation. To automatically generate a Makefile for HYPRE, run the configure script while in the “src” directory. The option prefix will dictate where the code is installed. For example, depending on superuser access, execute either
cd /home/$USER/Downloads/hypre-2.11.2/src
./configure --prefix=/opt/hypre
or
cd /home/$USER/Downloads/hypre-2.11.2/src
./configure --prefix=/home/$USER/opt/hypre
To compile and install the HYPRE library, execute (on Tiger you can use -j 8):
make install -j 4
If the installation directory is something that requires superuser permissions for writing, the latter command must be executed as superuser.
Upon successful installation, the file libHYPRE.a should be in the lib sub-directory of the directory in which you installed HYPRE.
Note
For some problems on certain systems, when NGA is run in a hybrid parallel mode (MPI + OpenMP), higher performance can ‘sometimes‘ be achieved when HYPRE is allowed to use the OpenMP threads. To allow the latter, the same instructions as above should be followed. However, the compiler flags should be changed to
export CFLAGS="-O3 -qopenmp -xhost -ip"
export CXXFLAGS="-O3 -qopenmp -xhost -ip"
export FCFLAGS="-O3 -qopenmp -xhost -ip"
and an additional option is required for the configure script:
./configure --with-openmp --prefix=/opt/hypre-openmp
or
./configure --with-openmp --prefix=/home/$USER/opt/hypre-openmp
As a practical note, Prof. M generally keeps two working installations of HYPRE, one without OpenMP and one with OpenMP. This allows the fastest installation to be used for various problems without having to constantly recompile/reinstall HYPRE. HYPRE with OpenMP should also work for runs without OpenMP without errors/failure. Keeping both is just for 5-10% performance improvement, which may not be the case any more for new version of HYPRE. When installing the second HYPRE, remember to make clean first.
DMD
These instructions only apply to users using the DMD branch of NGA. New users do not need to do this unless you have been told to.
The DMD source code is stored in the NGA repository (NGA/src/tools/modred-2.0.1), so it does not need to be downloaded. The documentation is available at http://packages.python.org/modred.
Module needed on Tiger: python/2.7/mpi4py/intel-13.0/openmpi-1.6.3/1.3.1
To install, run: python setup.py install –user