CAMPARI Installation
Coding Style:
CAMPARI is - with the exception of the use of external libraries - written entirely in Fortran which utilizes some features of the 2008 standard but is otherwise compliant with the 2003 standard. Because it has evolved over many years and many modern features were not widely supported compilers 5-10 years ago, CAMPARI does not adhere to strict standards, for example regarding the use of interfaces or the use of ISO names for architecture abstraction in variable typing. A second reason is the somewhat incremental support model compilers offer for modern Fortran features, i.e., it is quite pointless to use 2008 features not yet supported by the compilers on the computers that the software needs to run on.CAMPARI uses - with very few exceptions - dynamic memory allocation and calculations on small systems usually do not require more than a few MB of memory. This is implemented exclusively via arrays which have the
ALLOCATABLE
attribute and not through pointers. However, the memory footprint for large systems may grow unfavorably
with system size both in terms of stack usage and in terms of total memory usage. The general use and layout of memory is not tighly controlled
at the moment, and this challenges cache use on modern multi-core chips. In case of stack overflows, CAMPARI
will typically exit with an otherwise undocumented segmentation fault (this behavior is compiler-dependent and can often be
remedied by manually increasing stack size for the shell from which CAMPARI is started [ulimit]). CAMPARI uses stack-allocated
variables in particular in the inner loops of dynamics-based calculations. For total memory exceptions,
disabling some of the calculation's optional features will be the only solution.The source code frequently contains comparisons of floating point numbers which entails all the possible pitfalls of intrinsically inaccurate floating point arithmetics. Only some sensitive operations are made safe (precision-tolerant). The code assumes double precision throughout so aggressive optimization strategies of modern compilers have to be evaluated carefully (not all algorithms may tolerate precision loss and will respond with warning messages that may or may not imply that results are compromised). Basically, in most cases CAMPARI silently relies on hardware and compiler to implement floating-point arithmetics standards such as IEEE 754-2008.
Lastly, for efficiency reasons, the code attempts to provide vectorizable loops in crucial execution paths as much as possible. This is the simplest form of parallelism and supported (often by default) by almost all modern compilers. Vectorization increases time required for compilation, but can very substantially reduce runtime. It is important as well to point out that it in general increases the accuracy of sums accumulating very many small floating-point numbers (as is almost universally the case in the types of applications CAMPARI supports). These loops are also the main reason why performance across different compilers can differ substantially, i.e., the various instructions in the SSE3, SSE4, AVX, AVX2, etc. instruction sets are not equally well identified as usable for a particular portion of the source code (the resultant machine codes differ substantially). CAMPARI does not call instruction set intrinsics directly because of limited personnel resources and the wide scope of functionality it supports.
In summary, CAMPARI expects the compiler to be able to handle the following:
- Allocatable arrays of variables of derived type which contain further allocatable arrays of built-in or derived types
- Preprocessing of C-style directives (
#ifdef ... #endifand so on) - Beyond standard math and built-in functions from the 2003 standards and before, support for built-in functions
erfanderfc(these are formally introduced as part of the 2008 standard but were often available in compiler-specific implementations before) - Full use of the language-intrinsic ISO_FORTRAN_ENV and ISO_C_BINDING modules
- Dynamic (implicit) allocation of deferred-length strings on assignment statements
- A way to set default floating point constants to double precision (this is crucial to avoid down-conversion of constants which may cause major problems in algorithms relying explicitly on double precision; this is a known issue with the PGI compiler)
- For the multi-threaded versions, support for the OpenMP 4.0 (or higher) standard (it is, however, easy to modify it toward the 3.1 standard as the only feature from the 4.0 standard are very few PROC_BIND clauses to PARALLEL regions)
Compiler Optimization:
The code was developed on UNIX systems with the Intel Fortran compiler and mostly run on Intel CPUs. This implies that executables generated with this particular combination of compiler and processor tend to run faster than alternatives (different processor or different compiler, notably we have used the Gnu, PGI, Cray, Absoft, and Oracle compilers, with Absoft and PGI producing at least partially unusable or unverified executables). The Cray compiler on a Cray supercomputer also produces very fast code in relative terms. Users should keep this in mind. Publicly available benchmarks of Fortran compilers (such as those available at polyhedron) can be useful but their overall picture may not be representative for any given specific code (like CAMPARI). As mentioned above, the validity of aggressive optimization options will have to be evaluated for each calculation.Installation Requirements:
- Preferably a Unix-type OS (note that CAMPARI has also been compiled on Windows and MacOS but not on any other OS such as Solaris or Irix) in conjunction with a modern multi-core chip (we have only used recent Intel and AMD processor lines but not PowerPC, Compaq (DEC) Alpha, ...)
- A Fortran03 compiler with the above attributes (we have used Intel, GNU, Oracle (formerly SunStudio), Cray, PGI, and Absoft compilers)
- For the MPI version: a Fortran08-compiled MPI implementation, against which to link the code (we recommend
OpenMPI or MPICH), at least one of which might be part of the default system
installation on Linux. Since V4 of CAMPARI, we encourage the Fortran 08 module ("
use mpi_f08" must work) but this can be disabled during compilation in favor of the previous module ("use mpi" must work). - For using particle-mesh Ewald summation (see here): a properly compiled FFTW (this may be available as part of your system installation and can then be linked by passing '-lfftw3' to the linker)
- For using the preconditioned SHAKE algorithm to maintain holonomic constraints (see here) or principal component analysis (see here): a properly compiled LAPACK -library to link against (for example version 3.2.2 with double precision routines → on UNIX, this often this will be part of the default system installation and may be activated by passing '-llapack' to the compiler)
- For writing and reading NetCDF-files: a properly compiled NetCDF-library (see here) to link against which includes Fortran 90 API (developed and maintained by Unidata; this may be available as part of your system installation and can then be linked by passing '-lnetcdf -lnetcdff' to the linker)
- For writing and reading from a PostgreSQL database: a properly compiled libpqxx-library (see here) to link against (developed and maintained by Jeroen T. Vermeulen; note that this has a large number of nested requirements since the development version/API of PostgreSQL itself is required, see elsewhere)
- For using the Python interface: a Python 3 installation (see here) to provide the necessary libraries (NumPy is strictly required, SciPy is required but can be manually masked); this relies on ForPy (developed by Elias Rabel, GitHub page), shipped with CAMPARI under identical license terms
- For using some graph-based analyses (see spectral decompositions and committor probabilities): a properly compiled version of the HSL sparse matrix linear library developed and maintained by the Science and Technology Facilities Council (STFC) in the UK.
- A tar-ball (archived) or similar full copy of the CAMPARI distribution
- Possibly a C- and C++-compiler to compile auxiliary libraries
Getting CAMPARI:
The obvious first step is to extract the archive to your favorite destination (${CAMPARI_HOME}) or to get a copy through CVS if you are connected to a current CVS repository. Unlike most current Linux/Unix software, CAMPARI does not employ an explicit build system or configuration tool (e.g., autoconf/automake or cmake) beyond a simple Makefile. The compilation task is so uniform that the required settings are small in number. The downside is that portability has to be ensured explicitly by the user by providing the correct flags to the compiler. This "localization" happens in a dedicated file edited by the user and the global "Makefile.manual" expects this file to be called "Makefile.local" and to be located in the source-subdirectory of the main distribution tree. Hence:
[user {subdir}]$ cvs checkout campari
Or:
[user {subdir}]$ tar -xjvf campari.tar.bz2
At this point, you have two alternative routes. The first route is the manual installation in which you might have to minimally edit the existing "Makefile.manual" and write the short configuration file ("Makefile.local"). Detailed instructions are provided below. In the second route, you can use the configure script to set some of the required information automatically. This can simplify installation, in particular on systems that have a relatively canonical Linux installation. It is not recommended to mix the two modes of installation in the same directory tree.
Manual Installation
Makefile.manual and Makefile.local:
The largely fixed input file for the manual installation mode is called "Makefile.manual". We will use a file containing some custom overrides called "Makefile.local":
[user {subdir}]$ cd campari/source
[user source]$ touch Makefile.local
A Makefile is nothing but an advanced shell script (using its own scripting language) containing a series of compilation and linking commands to produce an executable from a set of sources. One of its core features is to use time stamps on source files to allow for safe incremental compilation (i.e., to re-compile only those source files that have changed since the last compilation and to make sure all libraries and binaries are up-to-date). The global "Makefile.manual" is located in the source-directory and uses two auxiliary files: the aforementioned "Makefile.local" and a file called "DEPENDENCIES", which has to reside in the source-directory as well. The latter is automatically generated from the source files by executing:
[user source]$ ./make_dependencies.sh .The customization in "Makefile.local" requires two elements: 1) defining one or more directories for the local CAMPARI distribution, and 2) defining compilers and compiler flags to be used:
Example of location-specific parameters in "Makefile.local":
CAMPARI_HOME=/software/campari
ARCH=Intel
BIN_DIR=${CAMPARI_HOME}/bin
LIB_DIR=${CAMPARI_HOME}/lib
SRC_DIR=${CAMPARI_HOME}/source
#
# compiler settings
WARNFLAGS= -warn alignments -warn declarations -warn usage -warn general -warn ignore_loc -warn truncated_source -warn uncalled -warn unused
FF =ifort
LFLAGS =-O3
EXTRA_LIBS=/software/NetCDF/Intel/lib/libnetcdff.a /software/NetCDF/Intel/lib/libnetcdf.a /software/fftw/lib/libfftw3.a /software/fftw/lib/libfftw3_threads.a -llapack -lpython3.8 -L/usr/lib -lcrypt -lpthread -ldl
FFLAGS =-DLINK_NETCDF -DLINK_XDR -DLINK_LAPACK -DDISABLE_ERFTAB -DDISABLE_FLOAT -DLINK_FFTW -DLINK_PSQL -DLINK_HSL -DENABLE_PYTHON -I/software/fftw-3.2.2/include -I/software/NetCDF/netcdf-4.1.1/include ${WARNFLAGS} -axAVX -xAVX
MPIFF =mpif90 # wrapper around ifort
MPILFLAGS =-O3
MPIFFLAGS =${FFLAGS}
MPIEXTRA_LIBS=${EXTRA_LIBS}
THREADFLAGS=-qopenmp # or -openmp for older versions
CC =icc # Intel C compiler for XDR
XDRFFLAGS=-O3 -axAVX -xAVX # not that the default options added to CAMPARI source file compilations are not included
XDRCFLAGS=-O3 -axAVX -xAVX
CXX=icpc
PSQLCXXFLAGS=-O3 -I/software/libpqxx/Intel/include # ## ########################################################### -Wall -std=c++17
PSQLEXTRALIBS=/software/libpqxx/Intel/lib/libpqxx.a -lstdc++ -lcrypto -lboost_system -lboost_filesystem -lpq
HSLFFLAGS=${XDRFFLAGS}
HSLEXTRALIBS=-lblas
The above "Makefile.local" illustrates settings for a standard installation framework using Intel compilers with compiler support for AVX and lower (but not AVX2) instructions. If an installation on only a single or otherwise homogeneous architecture is required, these instruction set support options should for simplicity be set to let the compiler detect the supported architecture itself. Note that the settings for CAMPARI_HOME and SRC_DIR have to correspond to the extracted archive (see above) while those for BIN_DIR and LIB_DIR are arbitrary. The specifications for "CAMPARI_HOME" (root directory) and "ARCH" (subdirectory in "lib" and "bin") should determine entirely the location of compiler-generated files. Because "Makefile.local" is read and processed by "Makefile.manual", it is possible to maintain multiple versions (differing in compiler, debugging options, architecture, etc.) from the same source tree by simply making copies of both files, , e.g., to "Makefile_New" and "Makefile_New.local". Then, in "Makefile_New" the include statement should be changed from "include Makefile.local" to "include Makefile_New.local". By now modifying both files according to the desired outcome, a completely independent installation is obtained when running "make -f Makefile_new <target>" where the various targets are explained below. Whenever you do this, it is essential that the different specifications for "ARCH" are all unique of course.
Further modifications to the global "Makefile.manual" should be restricted to the lines controlling default compiler options. For the Oracle, Cray, PGI, Gnu, and Intel compilers, "Makefile.manual" contains a fair amount of default settings (SUNDEFAULTS, CRAYDEFAULTS, PGIDEFAULTS, GNUDEFAULTS, INTELDEFAULTS) which on Linux systems can often be used "as is". In addition, there are analogous (but less complete) debugging options (SUNDEBUG, CRAYDEBUG, PGIDEBUG, GNUDEBUG, INTELDEBUG). The variable COMPDEFAULTS holds the actual default compiler options and can be set, for example, as "COMPDEFAULTS=${GNUDEFAULTS} ${GNUDEBUG}". These options are further appended by "FFLAGS" or "MPIFFLAGS" as specified in the localization file (see example above). Changes to default options can be required for a number reasons, for example, because of a change in software or hardware architecture. Also, files included from external libraries may occasionally clash with pedantic language standard options. The reason that only a few settings are left to "Makefile.local" is that the others should not be changed from the behavior described above. Otherwise, the resultant code may fail or not produce correct results. Lastly, the variable "THREADFLAGS" should hold all options desired and required to enable OpenMP support for the compiler in use. Compilers have been lagging somewhat in supporting the OpenMP standard fully, and consequently options may continue to evolve (for Intel and Gnu, respectively, "-openmp" and "-fopenmp" are generally sufficient).
Two possible libraries, when requested for linking (HSL and XDR), can be embedded during the CAMPARI build because they cannot be obtained easily through system or other precompiled packages (which NetCDF, FFTW, and LAPACK can be). There are flags in "Makefile.manual" to enable this embedded compiling and linking of this library, which is still conditional upon passing the flags "-DLINK_XDR" and "-DLINK_HSL", respectively. These flags are called "USE_INTERNAL_XDR" and "USE_INTERNAL_HSL" and setting them to zero allows linking against versions of these libraries compiled externally. Obviously, compilation and/or linking will fail in this case if the external version(s) are missing. The internal (embedded) compilation happens in subdirectories "xdr" and "hsl" of the source directory, use separate Makefiles, and should be largely automatic (dependencies are tracked). For XDR, CAMPARI distributes the source code, so there is no obvious benefit of compiling and linking it separately. The XDR library is used to write compressed trajectories in xtc-format (see here). The situation for HSL is different due to the licensing terms of HSL. You will need to obtain your own copy of HSL. Then, you can either copy the required source files into the "hsl" subdirectory (as decribed in the download instructions), or you can treat is an external library as desribed above. The embedded compilations of libraries can be customized with regards to options by additional specifications in "Makefile.local", specifically "CC" and "XDRCFLAGS" (C compiler and flags for XDR), "XDRFFLAGS" and "HSLFFLAGS" (Fortran compiler options for XDR and HSL, respectively), and "HSLEXTRALIBS" (which may be needed to solve LAPACK/BLAS dependencies of HSL). The Python module is slightly different in that it only requires ForPy (present as mod_forpy_mod.f90) to be compiled successfully, which is primarily a linking problem. The general solution to find the necessary libraries (Python is generally present at system level) is to run either:
python3-config --ldflags # pre 3.8
python3-config --embed --ldflags # 3.8 and later
The output of this should allow you to identify the required flags for linking successfully. That does not mean all of them are required, however, and even the simple "-lpython3.8" (for 3.8) might be sufficient.
Two notes on the passed flags DISABLE_FLOAT and DISABLE_ERFTAB. CAMPARI could in the (distant) past be compiled globally as a single-precision variant by omitting this flag. This is presently not possible. Single precision floating point arithmetics are only permissible in certain computations, and the necessary effort to identify just the eligible ones has simply not been undertaken. That is why DISABLE_FLOAT should always be passed to the compiler (even though it presently has no effect). DISABLE_ERFTAB controls whether to use the compiler-provided "erf" and "erfc" (see above) or whether to use a tabulated approximation (which does not avoid the need for "erf" and "erfc"). With modern (2016) compilers, the speed of the intrinsics is usually sufficient to warrant passing this flag.
To provide additional pointers for the users, we also show an example "Makefile.local" for the GNU compiler below (this assumes that NetCDF, FFTW, and LPAPACK/BLAS are system-installed, that USE_INTERNAL_XDR and USE_INTERNAL_HSL are both 1, and that "mpif90" is a wrapper around "gfortran"):
CAMPARI_HOME=/software/campari
ARCH=Gnu
BIN_DIR=${CAMPARI_HOME}/bin
LIB_DIR=${CAMPARI_HOME}/lib
SRC_DIR=${CAMPARI_HOME}/source
FF = gfortran
LFLAGS = -O3
EXTRA_LIBS=-lfftw3 -lfftw3_threads -lblas -llapack -lnetcdf -lnetcdff -lpython3.8 -L/usr/lib -lcrypt -lpthread -ldl
FFLAGS =-DLINK_NETCDF -DLINK_XDR -DLINK_LAPACK -DDISABLE_ERFTAB -DDISABLE_FLOAT -DLINK_FFTW -DLINK_PSQL -DLINK_HSL -DENABLE_PYTHON -march=native -funroll-loops # note that sometimes the include path for NetCDF has to be added here as well
MPIFF =mpif90 # wrapper around gfortran
MPILFLAGS =-O3
MPIFFLAGS =${FFLAGS}
MPIEXTRA_LIBS=${EXTRA_LIBS}
THREADFLAGS=-fopenmp
CC =gcc # Gnu C compiler for XDR
XDRFFLAGS=-O3 -march=native -funroll-loops
XDRCFLAGS=-O3 -march=native -funroll-loops
CXX=g++ # Gnu C++ compiler for PSQL module
PSQLCXXFLAGS=-O3 -I/software/libpqxx/Gnu/include -Wall -std=c++17 -march=native -msse4 -funroll-loops
PSQLEXTRALIBS=/software/libpqxx/Gnu/lib/libpqxx.a -lstdc++ -lcrypto -lboost_system -lboost_filesystem -lpq
HSLFFLAGS=${XDRFFLAGS}
HSLEXTRALIBS=-lblas
Standard (Serial) Installation (Executables "campari" and "camp_ncminer"):
To actually compile the code for serial execution, first create the directories ${LIB_DIR}/${ARCH} and ${BIN_DIR}/${ARCH} if they do not exist already (this follows the example above):
[user source]$ mkdir ../lib
[user source]$ mkdir ../lib/Intel
[user source]$ mkdir ../bin
[user source]$ mkdir ../bin/Intel
Now do:
[user source]$ make -f Makefile.manual campari
Given the warning messages requested, you should see warnings about the use of intrinsic erf() in Fortran 2003 code, potential warnings about unused variables but no major or other minor complaints. The compilation can take a while to complete since the optimization procedures (vectorization, etc ...) require variable dependency checks in complicated loop structures, which often will require a lot of physical memory and CPU time.
The processes executed by "Makefile.manual" will compile all module definition files mod_bar.f90 into module information files ${LIB_DIR}/${ARCH}/bar.mod as well as regular module object files ${LIB_DIR}/${ARCH}/bar.o. Similarly, source-files foo.f90 will be compiled into object files ${LIB_DIR}/${ARCH}/foo.o. Once finished, "Makefile.manual" will create a library out of all the object files called lcampari.a in ${LIB_DIR}/${ARCH}, and finally link the actual executable ("campari" in ${BIN_DIR}/${ARCH}/). In the future, after applying certain changes to the source code, CAMPARI may be compiled incrementally by executing the same command again. Note that if heavily used module files are changed, many source files will have to be re-compiled (see DEPENDENCIES).
By the same technique, and relying on the same library, it is possible to also compile a different executable that is specialized in data mining of arbitrary input files (see the section on NetCDF data mining in the keywords documentation as a starting point). The target name is camp_ncminer instead of campari in the make command. This executable has an obligate requirement for linking to the NetCDF library (see "Linking against external libraries" below).
Standard Thread-Parallel Installation (Executable "campari_threads" and "camp_ncminer_threads"):
The shared memory, thread-based parallelization of CAMPARI is compiled in exactly the same way with two additional requirements: i) specifying "THREADFLAGS" in the "Makefile.local" (see above), ii) linking libraries also in their thread-parallel versions (currently, this only applies to FFTW). Thus:
[user source]$ mkdir ../lib
[user source]$ mkdir ../lib/Intel
[user source]$ mkdir ../lib/Intel/threads
[user source]$ mkdir ../bin
[user source]$ mkdir ../bin/Intel
Now do:
[user source]$ make -f Makefile.manual campari_threads
The processes executed by "Makefile.manual" will compile all module definition files mod_bar.f90 into module information files ${LIB_DIR}/${ARCH}/threads/bar.mod as well as regular module object files ${LIB_DIR}/${ARCH}/threads/bar.o. Similarly, source-files foo.f90 will be compiled into object files ${LIB_DIR}/${ARCH}/threads/foo.o. Once finished, "Makefile.manual" will create a library out of all the object files called lcampari_threads.a in ${LIB_DIR}/${ARCH}/threads, and finally link the actual executable ("campari_threads" in ${BIN_DIR}/${ARCH}/).
As for the serial case, it is possible to also compile a different executable that is specialized in data mining of arbitrary input files (see the section on NetCDF data mining in the keywords documentation as a starting point), and the target name is camp_ncminer_threads instead of campari_threads.
Linking Against External Libraries:
In general, external libraries are necessary to support certain features within CAMPARI as outlined above. The user may instruct the preprocessor to include the respective sections of code by passing flags to it: LINK_FFTW to link against the library for fast discrete Fourier transforms, LINK_LAPACK to link against the LAPACK linear algebra library, LINK_XDR to link against compression routines needed to write trajectory data in .xtc-format, LINK_NETCDF to link against routines to create NetCDF data archives, and LINK_HSL to link against the sparse matrix linear algebra library, which additionally depends on BLAS (see example "Makefile.local" above and section on installation requirements). As mentioned, XDR and HSL are embedded with CAMPARI and can be compiled automatically, which is enabled by control flags in the "Makefile.manual" ("USE_INTERNAL_XDR" and "USE_INTERNAL_HSL"). For successful compilation and linking, one (for embedded libraries) or two (external) extensions are required:- Pass the environmental LINK_FOO to the compiler (flag -DLINK_FOO).
- Point the compiler to the library using the EXTRA_LIBS macro in "Makefile.local" (e.g., "EXTRA_LIBS=${whatever_path}/libfoo.a" or "EXTRA_LIBS=-lfoo"). This is not necessary for embedded compilation of HSL and XDR.
The XDR library requires special comments: CAMPARI supports .xtc-files for writing and reading trajectory data. These highly compressed binary files are employed by the GROMACS simulation software which has by now incorporated its own XDR support due to a lack of an officially supported distribution (original credits to Frans van Hoesel). For reference, CAMPARI provides a minimal XDR distribution. For setup as an external library, go to the root directory and find the file called "xdr_basic.tar.bz2". Unpack this archive, edit the Makefile it contains and compile it (see supplied README.txt). Conversely, the embedded version is found in the source subdirectory "xdr" and should be compiled automatically if the required customization in "Makefile.local" has been done (see above).
Similarly, the required HSL routines are embedded in a source subdirectory called "hsl". Compilation will again be automatic if "DLINK_HSL" is passed to the compiler. You should already have or independently obtain a license for using HSL (see Download page).
External libraries may have been compiled into thread-aware versions, but the only example where CAMPARI currently makes use of this is for the discrete Fourier transforms provided by FFTW. If "campari_threads" is built with FFTW support, it is thus necessary to also supply the threads version of the FFTW library to "EXTRA_LIBS" as was done in the examples above.
When building custom versions of external libraries for linking, the NetCDF library tends to be the trickiest. One reason is that the structure of NetCDF has changed considerably in recent years: The Fortran version of the library was split off, and the reliance on other libraries increased. Thus, if you build a custom NetCDF library and try to link it to CAMPARI, opaque errors might occur. Following the Intel compiler example above, you may require something like this for linking static NetCDF libraries:
EXTRA_LIBS=-L/software/NetCDF/Intel/lib /software/NetCDF/Intel/lib/libnetcdff.a /software/NetCDF/Intel/lib/libnetcdf.a -lhdf5_hl -lhdf5 -lz -lcurl -lm
/software/fftw/lib/libfftw3.a /software/fftw/lib/libfftw3_threads.a -llapack
This gets progrssively more complicated if you also want to use custom installations of "libhdf5" and so on. You can find more information on the UniData pages, e.g. here. Conversely, the linking of system-installed libraries (meaning those in default locations using default compilers), which is currently possible for FFTW, NetCDF, and LAPACK (see the example "Makefile.local" for GNU above), is simpler. Here, it is critical to just install all relevant packages. For FFTW, this usually means installing at least two packages (core and development, both in double precision). For NetCDF it can imply installing several packages (core and development for both standard C and Fortran interfaces). The compilation of CAMPARI can fail before the linking step if module or include files for NetCDF and/or FFTW are missing. In these cases, it is recommended to try to locate the files on your computer and add the include path to the compiler flags. If they are not found, the most common reason is that the corresponding development package was not installed. The linking fails if one or more symbols cannot be resolved. The internet is usually a good resource to at least pin down the origin of linking errors (for common libraries). If the problem persists, you are of course invited to post on the SourceForge page.
Pure MPI Installation (Executable "campari_mpi"):
The Message Passing Interface (MPI) is a standard for creating parallel software, i.e., programs running simultaneously in multiple instances on different processors or processor cores. These instances exchange information by passing messages to one another. This is what MPI libraries accomplish.Changes from serial to parallel versions of a program are often relatively large since meaningful parallelism requires exchange and synchronization of information between the individual instances as well as ordering mechanism for sensitive operations (writing to files, etc ...) which may otherwise entail race conditions. "Makefile.manual" therefore treats the serial and parallel version independently with completely separate directories for all libraries and binaries.
To install CAMPARI's MPI version, install an MPI distribution first (like OpenMPI), if it is not already installed on your system. This MPI distribution should be built such that it makes the Fortran 08 version of its Fortran module available (mpi_f08) or, this might be required on older operating systems, the previous Fortran module (mpi). In order to use the latter, "-DDISABLE_MPIF08" must be passed to the compiler when compiling. Usually, when building MPI from source, there is a chance to supply a Fortran90 compiler, which will then generate an MPI-compiler (like "mpif90") which is automatically linked against that MPI library. It is beneficial to compile the MPI library yourself. If this is not possible or desired but the precompiled libraries such as those found in RPMs do not provide adequate wrapper compilers, it will be necessary to create an appropriate wrapper compiler by passing on additional flags and libraries to MPIEXTRA_LIBS and MPIFFLAGS in "Makefile.local". If a wrapper compiler is available - they might be provided by the corresponding *-dev packages - it might be unclear which compiler was used underneath (normally gfortran on Linux). To answer this question, "mpif90 -showme" or "mpif90 -show" are commands that might work.
Using the above example, next create a new directory (if it does not exist already):
[user source]$ mkdir ../lib
[user source]$ mkdir ../lib/Intel
[user source]$ mkdir ../lib/Intel/mpi
[user source]$ mkdir ../bin
[user source]$ mkdir ../bin/Intel
Now do:
[user source]$ make -f Makefile.manual campari_mpi
This will compile all module definition files mod_bar.f90 into module information files ${LIB_DIR}/${ARCH}/mpi/bar.mod as well as regular module object files ${LIB_DIR}/${ARCH}/mpi/bar.o. It will furthermore compile all source-files foo.f90 into ${LIB_DIR}/${ARCH}/mpi/foo.o, create a library out of all the object files called lcampari_mpi.a in ${LIB_DIR}/${ARCH}/mpi and finally compile and link the actual executable ("campari_mpi" in ${BIN_DIR}/${ARCH} which can be used in conjunction with mpirun or its equivalent).
As you can tell, all object files (and even the Fortran90 module declarations) are kept in separate directories such that both targets can be dealt with independently.
Note that CAMPARI's support for MPI is restricted to an "outer" parallel approach, i.e. managing multiple copies of the same or similar tasks that sporadically need to exchange information. Conversely, it does not support standard domain-wise parallelization as most other molecular simulation packages (see here). The inner layer of parallelism (speeding up a single task by distributing its workload) is handled exclusively by the OpenMP shared memory parallelization. This is also why it is not necessary or correct to link specifically to MPI-enabled versions of external libraries. Both MPI and OpenMP can be used simultaneously, and this is described next.
Hybrid MPI/OpenMP Installation (Executable "campari_mpi_threads"):
CAMPARI allows a hybrid parallelization using both MPI (for communication between multiple copies of the same or similar tasks) and OpenMP (for speeding up individual tasks). To install CAMPARI's hybrid MPI/OpenMP version, the requirements for the production of both the pure MPI and OpenMP executables described above have to be met.Using the above example, create new directories (if they do not exist already):
[user source]$ mkdir ../lib
[user source]$ mkdir ../lib/Intel
[user source]$ mkdir ../lib/Intel/mpi_threads
[user source]$ mkdir ../bin
[user source]$ mkdir ../bin/Intel
Now do:
[user source]$ make -f Makefile.manual campari_mpi_threads
This will compile all module definition files mod_bar.f90 into module information files ${LIB_DIR}/${ARCH}/mpi_threads/bar.mod as well as regular module object files ${LIB_DIR}/${ARCH}/mpi_threads/bar.o. It will furthermore compile all source-files foo.f90 into ${LIB_DIR}/${ARCH}/mpi_threads/foo.o, create a library out of all the object files called lcampari_mpi_threads.a in ${LIB_DIR}/${ARCH}/mpi_threads and finally compile and link the actual executable ("campari_mpi_threads" in ${BIN_DIR}/${ARCH} which can be used in conjunction with mpirun or its equivalent). On modern multicore and multi-socket machines, the mapping of program threads created by such an executable to the actual hardware resources is a complicated issue (e.g., a process of 4 MPI tasks and an allocation of 2 machines with 2 sockets of 16 core CPUs each should probably run such that each MPI task occupies all the cores of an entire socket but can be mapped in a myriad number of suboptimal ways).
Cleaning Up:
Execute:[user source]$ make -f Makefile.manual clean
to delete all objects, compiled module files, and libraries in ${LIB_DIR}/${ARCH}, ${LIB_DIR}/${ARCH}/threads, ${LIB_DIR}/${ARCH}/mpi, and ${LIB_DIR}/${ARCH}/mpi_threads, and also wipe out all binaries in ${BIN_DIR}/${ARCH}. This includes removing compiled files associated with embedded libraries (XDR and HSL). Note that it is possible to support multiple architectures within the same tree, since ${ARCH} will be different, and only the "currently active" distributions will be cleaned up (or compiled for that matter). Note that this will also delete copies of object, module, and library files (or symbolic links to those files) that were placed in the corresponding directory manually, e.g. to simplify linking to external libraries, as long as these files feature the same ending.
Execute:
[user source]$ make -f Makefile.manual objclean
to just delete all object files and libraries.
Execute:
[user source]$ make -f Makefile.manual extclean
to just delete all object files and libraries associated with embedded versions of XDR and HSL.
Debugging:
Particular compilers or older operating systems can of course cause problems. When we have encountered these issues in the past and there was no workaround from the compiler side, we introduced additional compiler flags to be passed. Currently these are "ORACLE_FORTRAN", "PGI_FORTRAN", and "DISABLE_OPENMP4" (not explained here any further). If you run into problems, do not hesitate to post on the SourceForge forums.Partially Automatic Installation
A configure script is a complicated shell script that tries to produce a specific Makefile from a template, "Makefile.in". This classical build framework is primarily developed for code written in C, and the capabilities for Fortran are much more limited. The goal of this approach is primarily in build automation and platform abstraction, in particular if you try to compile and environments where you are not fully aware of the details of how libraries are installed, configured, etc. The particular configure script shipping with CAMPARI is trying to anticipate most of the problems encountered during a build. These include:- Having the Fortran compiler invoke the C preprocessor
- Enforcing CAMPARI's standard compilation settings (see above) for different, recognized compilers
- Adding performance options for different, recognized compilers
- Having an MPI-ready compiler for the MPI and hybrid MPI/OpenMP executables
- Having an OpenMP-ready compiler for the OpenMP and hybrid MPI/OpenMP executables
- Having the NetCDF library installed
- Having the FFTW library installed
- Having an external XDR library installed
- Having an external HSL library installed
- Having the C++-PostgreSQL library (libpqxx) installed
- Having Python 3 installed
[user {subdir}]$ cd campari/source
[user source]$ ./configure --enable-mpi=auto
[user source]$ make all install
This is the standard procedure of a configure script. To clean up compiled objects/binaries, you would execute:
[user source]$ make clean
Or, if you want to remove also the files created by the configure script:
[user source]$ make distclean
During the configuration, you will get a report on which of the external libraries could be found and appear linkable. You will also get information on OpenMP and MPI support. Several options might be disabled automatically, and you should check the output carefully. Toward the end, there is a report of "Flavour options" that have been enabled along with the missing ones. If everything is present in a default location and everything works, this should give you all CAMPARI exceutables usually compiled with the GNU compiler suite and either OpenMPI or MPICH. If libraries and compiler executables are not in default locations, the configuration will be incomplete or fail completely. The most likely scenario on a standard Linux system is that the link to the PostgreSQL database will fail (libpqxx is not usually a default system package), that the links to FFTW, Python, and MPI will work automatically if basic support is available on the system, and that the link to NetCDF will probably work (which is slightly trickier). It will also determine to build XDR automatically from the CAMPARI-included version and recognize that HSL source files are missing and thus disable linking to this library.
The rest of this document briefly describes some of the options to work around these potential issues. An overview of options is available from "./configure --help".
Custom Location
One failure might occur during the installation step because the default prefix directory is not writable by a regular user. The easiest workaround is to use "--exec-prefix='<directory_name>'" as an additional option to configure, where <directory_name> is your target directory (will then contain subdirectories "bin" and "lib" with the relevant files). For more fine-grained control, use "--bindir" and "--libdir". The other standard directory customizations are not supported by the CAMPARI build process.The files compiled and linked initially are placed in subdirectories of the CAMPARI root directory, which can be overridden by "--with-campari-home='<other_CAMPARI_root>' but this is rarely useful. These subdirectories are "lib/${ARCH}" and bin/${ARCH}" where "${ARCH}" is a string describing the basic architecture type of your CPU (something like "x86_64"). You should avoid using these subdirectories with the aforementioned customization options since the install will throw an error. If you do not wish to copy executables and libraries to other destinations, simply leave out the "install" from the make command.
Custom Compilers
The default build will rely on Linux-centric macros to identify suitable target compilers. Unlike in the manual installation mode, there is no complete differentiation between the MPI-ready and the standard compiler. Normally, an MPI-ready compiler is just a wrapper around a standard compiler. The configure script will try to deconstruct this wrapper and invoke the standard compiler directly. For example, if you have mpif90 in your path, which is the name of the wrapped compiler in both MPICH and OpenMPI, the following will perform this deconstruction.[user source]$ export FC=mpif90
[user source]$ ./configure --enable-mpi=yes
Note that "yes" is used instead of "auto" for the MPI option ("auto" implies that you want to look for an MPI-ready compiler without specifying one). If you want to prevent this compiler deconstruction, use "--enable-mpi=force". This will lead to all executables compiled with MPI support at the compiler level, which is usually not a problem. Importantly, it will not lead to CAMPARI's MPI features being compiled into "campari" or "campari_threads". The deconstruction is important because it allows the configure script to recognize which basic compiler is underneath and to set corresponding default options. Generally, it is possible to try to enforce a specific compiler with the "--enable-compiler" option, for example "enable-compiler=intel", but it is rarely needed as the compilers are recognized anyway should they be available. The most common scenario would be in conjunction with "--enable-mpi=force" to automatically put the options for a wrapped base compiler that the script is unable to unwrap. If the MPI option fails during configuration, try disabling the support for the Fortran 08 module by passing "--disable-mpif08". Note that to build the XDR libary also a matching C compiler should be set, for example for Intel (the C compiler does not need to possess MPI capabilities):
[user source]$ export FC=mpif90 # assuming it is based on ifort
[user source]$ export CC=icc
If you have a compiler that the development team has no experience with, it consequently means that you have to take care of setting the appropriate flags. The requirements are listed under "Coding Style" at the top of this document. To do so, the best way is to set them through "--with-trailing-user-fcflags='<options>'". This works well for options that are not order-dependent. If the compiler requires custom libraries for linking code, it gets more problematic. You can try to embed everything a priori in "--with-user-fcflags='<options>'". This will override most automatically determined choices. If this does not work, you need to define values for FCFLAGS, LDFLAGS, and LIBS on the command line (these are standard options parsed by the configure script).