Installation

Installation#

Hermes-3 can be installed using CMake or spack. Using CMake is a more manual process which requires you to provide all of the dependencies yourself, but it has been used extensively and the documentation provides a module list for several HPC systems.

Spack capability has recently been added and allows an easy way to install the dependencies, making it much easier to install Hermes-3 without a module environment, e.g. on a workstation or laptop.

Using CMake#

Compilation process#

Compilation is achieved in two stages - the first is configuration where all the compile-time options are read in. The second is the build which results in a ready-to-use Hermes-3 installation in a directory named build by default. The build directory name can be changed to have multiple builds available at the same time.

If you make changes to the code, you can skip straight to the build stage to save time. Only modified files will be recompiled.

Hermes-3 is built using CMake. During configuration BOUT++ will be automatically downloaded as a submodule, together with some dependencies. The correct version of netCDF is downloaded and compiled automatically for convenience. FFTW is assumed to be installed already.

Hermes-3 uses two solvers: SUNDIALS cvode for time-dependent simulations and the faster PETSc beuler for steady-state transport problems. While SUNDIALS can be downloaded and installed automatically, PETSc requires manual installation.

Installing with SUNDIALS (cvode) only#

If you only want to use the cvode solver, then the recommended way to build Hermes-3 links to the SUNDIALS library:

  1. Configure with cmake, downloading and linking to SUNDIALS and NetCDF4:

    cmake . -B build -DBOUT_DOWNLOAD_SUNDIALS=ON -DBOUT_DOWNLOAD_NETCDF_CXX4=ON

  2. Build, compiling Hermes-3 and all dependencies using 4 parallel cores (adjust as necessary):

    cmake --build build -j 4

  3. Run the unit and integrated tests to check that everything is working:

    cd build
ctest

Note that the integrated tests require MPI, and so may not run on the head nodes of many computing clusters.

Installing with SUNDIALS and PETSc (beuler)#

The steady-state solver beuler requires PETSc and is often preconditioned using the hypre package, which is automatically downloaded and configured during PETSc installation.

Here is an example PETSc configure setup:

./configure --with-mpi=yes --download-hypre --download-make --with-fortran-bindings=0 --with-debugging=0

Here is an example working script to automatically download and compile PETSc on Viking2:

mkdir petsc-build
wget https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz
tar xzf petsc-3.17.4.tar.gz
cd petsc-3.17.4
./configure COPTFLAGS="-O3" CXXOPTFLAGS="-O3" FOPTFLAGS="-O3" --download-hypre --with-debugging=0 --prefix=../petsc-build
make -j 4 PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-opt all
make -j 4 PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-opt install
make -j 4 PETSC_DIR=$PWD/../petsc-build PETSC_ARCH="" check

and on ARCHER2:

mkdir petsc-build
wget https://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.17.4.tar.gz
tar xzf petsc-3.17.4.tar.gz
cd petsc-3.17.4
./configure --CC=cc --CXX=CC --FC=ftn COPTFLAGS="-Ofast" CXXOPTFLAGS="-Ofast" FOPTFLAGS="-Ofast" --with-batch --known-64-bit-blas-indices=0 --known-sdor-returns-double=0 --known-snrm2-returns-double=0 --with-fortran-bindings=0 --download-hypre --with-debugging=0 --prefix=../petsc-build
make -j 4 PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-opt all
make -j 4 PETSC_DIR=$PWD PETSC_ARCH=arch-linux-c-opt install
make -j 4 PETSC_DIR=$PWD/../petsc-build PETSC_ARCH="" check

And here is a working configure example for Perlmutter:

./configure \
  --with-mpi=yes --with-precision=double --with-scalar-type=real --with-shared-libraries=1 \
  --with-debugging=0 {C,CXX,F}OPTFLAGS="-O3 -march=native" \
  --download-hypre --download-fblaslapack=1 \
  --prefix=$HOME/local/petsc-3.22.3

Once PETSc is installed, link it to Hermes-3 using the -DBOUT_USE_PETSC=ON CMake flag:

cmake . -B build -DBOUT_DOWNLOAD_SUNDIALS=ON -DBOUT_DOWNLOAD_NETCDF_CXX4=ON -DBOUT_USE_PETSC=ON

If the PETSC_DIR and PETSC_ARCH environment variables have been set, then CMake should pick them up. If it doesn’t, try doing a clean build by removing any previously generated build directories.

Dependencies#

Since Hermes-3 heavily relies on BOUT++, the BOUT++ documentation on installation and dependencies contains a lot of useful information. Below is a selection of working module lists for several HPC systems. It is recommended you start with a clean module environment by executing module purge first.

YPI Workstations:

module load mpi/OpenMPI/4.1.1-GCC-10.3.0
module load devel/CMake/3.20.1-GCCcore-10.3.0
module load numlib/OpenBLAS/0.3.15-GCC-10.3.0
module load lib/FlexiBLAS/3.0.4-GCC-10.3.0

ARCHER2:

module swap PrgEnv-cray/8.3.3
module swap cce/15.0.0
module swap cray-mpich/8.1.23
module load cray-python/3.9.13.1
module load netcdf4
module load cmake
module load cray-hdf5
module load cray-netcdf/4.9.0.1
module load cray-parallel-netcdf/1.12.3.1
module load cray-fftw/3.3.10.3
module load valgrind4hpc

Marconi:

module load tools/git/2.32.0-GCCcore-10.3.0-nodocs
module load mpi/OpenMPI/4.1.1-GCC-10.3.0
module load devel/CMake/3.20.1-GCCcore-10.3.0
module load numlib/OpenBLAS/0.3.15-GCC-10.3.0
module load data/netCDF/4.8.0-gompi-2021a
module load lang/SciPy-bundle/2021.05-foss-2021a

Viking2:

module load OpenMPI/4.1.1-GCC-10.3.0
module load git/2.32.0-GCCcore-10.3.0-nodocs
module load CMake/3.20.1-GCCcore-10.3.0
module load OpenBLAS/0.3.15-GCC-10.3.0
module load netCDF/4.8.0-gompi-2021a
module load SciPy-bundle/2021.05-foss-2021a

Ancalagon:

module load OpenMPI/4.1.1-GCC-10.3.0
module load CMake/3.20.1-GCCcore-10.3.0
module load OpenBLAS/0.3.15-GCC-10.3.0
module load SciPy-bundle/2021.05-foss-2021a
module load netCDF/4.8.0-gompi-2021a

Perlmutter:

source /opt/cray/pe/cpe/23.03/restore_lmod_system_defaults.sh
module load craype-x86-rome
module load libfabric
module load craype-network-ofi
module load xpmem
module load cray-libsci
module load PrgEnv-gnu
module load cray-mpich
module load python
module load cray-fftw
module load cray-hdf5
module load cray-netcdf

Slope (flux) limiter settings#

Advection operators in Hermes-3 use slope limiters, also called flux limiters to suppress spurious numerical oscillations near sharp features, while converging at 2nd-order in smooth regions. In general there is a trade-off between suppression of numerical oscillations and dissipation: Too little dissipation results in oscillations that can cause problems (e.g. negative densities), while too much dissipation smooths out real features and requires higher resolution to converge to the same accuracy. The optimal choice of method is problem-dependent.

The CMake flag -DHERMES_SLOPE_LIMITER sets the choice of slope limiter. The default method is MC, which has been found to provide a good balance for problems of interest. If more dissipation is required then this can be changed to MinMod; if less dissipation is required then this can be changed to Superbee.

The appropriate limiter is problem-dependent. MinMod can work well for 1D tokamak simulations with steep gradients, e.g. simulations of detachment transients in high power machines which are already under-dissipative due to the lack of cross-field transport. The use of MinMod in 2D or 3D can lead to over-dissipation, but greater robustness.

Compiling in debug mode#

Please see the relevant page in the BOUT++ documentation.

Custom versions of BOUT++#

If you have already installed BOUT++ and want to use that rather than configure and build BOUT++ again, set -DHERMES_BUILD_BOUT=OFF and pass CMake the path to the BOUT++ build directory e.g.

cmake . -B build -DHERMES_BUILD_BOUT=OFF -DCMAKE_PREFIX_PATH=$HOME/BOUT-dev/build

The version of BOUT++ required by Hermes-3 is periodically updated, and is usually derived from a commit on the next branch of BOUT++. The up to date commit can be found in the “external” directory of the Hermes-3 repo.

Custom configuration of CMake#

The CMake configuration can be customised: See the BOUT++ documentation for examples of using cmake arguments, or edit the compile options interactively before building:

ccmake . -B build

Troubleshooting issues#

The first step to troubleshooting compilation issues should always to delete build folder for a fresh compilation. This can resolve several types of issues.

There have also been several reported issues due to Conda (e.g. making BOUT++ pick up the Conda MPI installation instead of the module one). A workaround is to compile with the CMake flag -DBOUT_IGNORE_CONDA_ENV=ON.

Using Spack#

In this section we describe how to build Hermes-3 using spack to manage the installation of standard packages to your local environment. By default, dependencies like NetCDF4, PETSc and SUNDIALS will be installed automatically, as required, but note that it’s also possible to use your own versions of packages (either system-installed or locally-built). While spack can be used to build Hermes-3 at a particular version, the instructions below also allow you to compile BOUT++ and Hermes-3 as part of a development workflow, i.e. using any changes you have made to the code in the working tree of your local repository.

These instructions were last tested using Ubuntu 22.04.1 and spack version 1.1.0. Spack versions prior to 1.0.0 are not compatible with our package files and are no longer supported. The default environment configuration assumes you have gcc installed.

Install Spack#

Instructions for installing spack on a variety of operating systems can be found in the spack docs. The commands below should work for most Debian-based Linux distributions. First, install some prerequisites, e.g for Ubuntu:

sudo apt update
sudo apt install -y build-essential ca-certificates coreutils curl environment-modules gfortran git gpg lsb-release python3 python3-distutils python3-venv unzip zip

Now, clone spack. You must use version v1.0.0 or newer. At the time of writing this section, version v1.1.0 has been tested:

git clone -c feature.manyFiles=true --depth=2 https://github.com/spack/spack.git -b v1.1.0

Important

If you have an existing spack installation of a version before v1.0.0, you will need to uninstall it first. Do this by uninstalling all packages and deleting the spack directory:

spack uninstall --all
rm -rf ~/.spack

Initialise spack. Add this command to your .bashrc or similar to make spack available in all new shells:

. spack/share/spack/setup-env.sh

The spack command should now be available; e.g.

spack --version
 > 1.1.0 (0c2be44e4ece21eb091ad5de4c97716b7c6d4c87)

Libraries and executables associated with spack packages are installed to $SPACK_ROOT/opt/spack by default ($SPACK_ROOT is wherever you cloned spack to). Build files, however, are placed in $tmpdir, which usually resolves to /tmp/$USER on Linux systems. Since /tmp is typically cleaned when you reboot your machine, this can be a problem if you need access to a build at some later time. To change this behaviour, create a config.yml file:

touch $HOME/.spack/config.yaml

and edit it to include

config:
   # Put builds in $SPACK_ROOT rather than $tmpdir
   build_stage:
      - $spack/var/spack/stage
      - $user_cache_path/stage

To add gcc to your user-level package list, and register it as a compiler that spack can use, run

spack compiler find

If you don’t do this, the below instructions should still work, but spack will modify the (version-controlled) environment file, spack.yaml, to include the gcc installation details.

Important

Version 1.1.0 of spack sets the built-in package repository to version releases/v2025.11. The PETSc package in that version has a bug, related to the Hypre dependency, which causes the installation of BOUT++ to fail. To work around this you can use the develop version of the spack-packages repo instead. Edit $SPACK_ROOT/etc/spack/defaults/base/repos.yaml as follows:

repos:
   builtin:
      git: https://github.com/spack/spack-packages.git
      branch: develop # branch: releases/v2025.11

and then run spack repo update.

This workaround should not be required when using spack versions newer than 1.1.0.

Install Hermes-3 and dependencies#

The following instructions assume you have already git-cloned Hermes-3 and are in the root directory of the repository.

First, update the git submodules to ensure that you have the correct version of the BOUT-spack repository checked out:

git submodule update --init --recursive

Then activate the spack environment described in spack.yaml by using the wrapper script:

. activate_h3env

This file provides some useful bash functions and aliases, but it’s also possible to use standard spack commands to activate the environment. You should see your prompt change to [hermes-3], indicating that the spack environment is active.

Note

The wrapper script runs spacktivate . -p -v gcc to load a ‘view’ when activating the environment. If you choose not to use the wrapper, you’ll need to run a similar command in order for the instructions below to work. For more info on views, see the spack documentation.

Note

If you’ve run spack install in this environment before, it’s advisable to run spack concretize -f at this point to ensure the concretized ‘spec’ is up to date. See the spack docs for more details.

To install Hermes-3 and all of its dependencies:

spack install -j 8

where the -j argument controls the number of parallel processes used to build packages.

This initial install takes some time to complete, because spack builds a large number of low-level packages. It’s possible to speed things up by adding system-installed packages to your packages.yaml file, but unless storage space is a big concern, letting spack build its own versions is usually the most trouble-free approach. This step rarely need to be repeated in its entirety unless moving to another version of the same compiler, or switching to a different version of spack itself.

The location in which spack builds packages is set via your user configuration options (see Install Spack), but for convenience, the environment wrapper script automatically generates links to the BOUT++ and hermes-3 builds in ./builds/spack/boutpp/[hash] and ./builds/spack/hermes-3/[hash] respectively, (where [hash] is a label that spack assigns). CMake and compiler output can be found in the spack log files, and the the build itself in a subdirectory spack-build-[hash]. Build directory links are updated every time a spack install or spack uninstall command is run.

Changing Hermes-3 and BOUT++ versions#

Sometimes, you may want to change the version of either Hermes-3 or BOUT++. They are included in the spack environment as develop modules, which means they are compiled from their local directories. To change the version, simply check out a different commit in either your Hermes-3 repo or the BOUT++ submodule in external/BOUT-dev and run spack install again.

Tip

It is possible to set a build directory with, e.g. spack develop -p . --build-directory ./builds/mybuild hermes-3, but this modifies spack.yaml and is a bit awkward if you need to switch between builds frequently. A more straightforward approach is to use CMake directly, as described below.

Developing Hermes-3 and BOUT++ in the Spack environment#

A convenient way to build Hermes-3 and BOUT++ in development is to use the dependencies installed via the instructions above, or only install the dependencies in the first place:

spack install --only dependencies -j 8

Having the environment activated provides all of the dependencies for compilation using CMake as described in Using CMake. The wrapper script provides a bash function in_h3env that runs commands in the hermes-3 build environment, setting all of the necessary paths to find headers, link libraries etc. To build with CMake, run (e.g.):

export h3_build="./builds/my_build"
in_h3env cmake -DBOUT_USE_SUNDIALS=ON -DBOUT_USE_PETSC=ON -B "$h3_build"
in_h3env cmake --build "$h3_build" -j8

In the above example, PETSc and SUNDIALS support in BOUT++ are turned on at the configuration stage, which achieves the same result as spack-installing the hermes-3 package directly, assuming that the previous step was run without changing the default spec in spack.yaml.

The tests can then be run in the usual way with:

cd ./builds/my_build
ctest -j 3

If Hermes-3 was installed with the +xhermes variant (as it is by default), the tests should pass without having to modify any paths.

Tip

If you don’t need to modify the BOUT++ source code, or only need to make infrequent changes, Hermes-3 builds can be sped up by integrating any changes into the BOUT++ spack package (rerun spack install --only dependencies -j 8), then configuring Hermes-3 with -DHERMES_BUILD_BOUT=OFF. This should automatically pick up the spack-installed BOUT++ library. Without that configuration setting, the Hermes-3 build will compile its own version of BOUT++ and will always prefer that to any installed in the spack environment.

Configuration options#

To see which variants of Hermes-3 and BOUT++ are available, run

spack info --no-dependencies --no-versions hermes-3
spack info --no-dependencies --no-versions boutpp

Tip

If you know that CMake functionality exists in BOUT++ or hermes-3 which is not configurable via their respective spack packages, you can create an issue in the package repository to request that the option(s) be added.

By default, the top-level ‘spec’ in the environment is hermes-3%gcc ^boutpp+petsc+sundials ^petsc+hypre+mumps, which tells spack to configure BOUT++ with SUNDIALS and PETSc support (including HYPRE and MUMPS) and to build Hermes-3 with gcc. To change this, first modify spack.yaml. to (e.g.) include superlu-dist in the PETSc build instead of MUMPS:

spack:
   specs:
      - hermes-3%gcc ^boutpp+petsc+sundials ^petsc+hypre+superlu-dist
...

then (re-)install dependencies as necessary:

spack install --only dependencies -j 8

Important

If you’re installing Hermes-3 and/or BOUT++ with CMake (i.e. after running spack install --only dependencies), the above command will update the installed dependencies according to the spec in spack.yaml, but you’ll still need to supply the appropriate configuration options to CMake. If you’re installing Hermes-3 with spack directly, the correct configuration options will be set automatically.

Using Spack in HPC environments#

You should be able to use spack on any HPC system by cloning the spack repository and following the instructions as normal. However, many HPC systems have module environments compiled in a way which is optimised for that system. In order to take advantage of this, you can configure spack to use the system modules as ‘external’ packages. You can also use spack external find to automatically detect any enabled modules and make them available in the spack environment. See the spack HPC tutorial for more information on using spack in HPC environments.

Useful spack commands/tips#

Updating your environment#

Most of the time, spack will detect when the ‘spec’ in spack.yaml has changed and run spack concretize automatically to regenerate all of the package details in spack.lock. If you’ve made changes that aren’t captured by the ‘spec’ string (for example, if the packages in the BOUT-spack submodule have been updated) , you’ll need to force the update with

spack concretize -f

If in any doubt, run the above to ensure your environment is updated correctly.

Examining the current environment spec#

To see a list of the packages that are installed / will be installed, run

spack spec

Checking package paths#

To see lib, bin, and include paths for installed packages, run

spack find --paths [package_name]

Changing the build type#

To change the build type of Hermes-3 and/or BOUT++ when spack-installing the packages directly, set their ‘build_type’ variants in spack.yaml, e.g.:

spack:
   specs:
      - hermes-3%gcc build_type=Debug ^boutpp+petsc+sundials ^petsc+hypre+mumps

Next Steps#

You are now ready to try running the examples in the build/examples/ folder. See https://hermes3.readthedocs.io/en/latest/examples.html.