GeoChemFoam on ARCHER2 - GeoChemFoam/GeoChemFoam GitHub Wiki

This document describes how to employ GeoChemFOAM on ARCHER2 using modules and how to install it yourself from scratch.

NB In all the examples in this document and in the example batch scripts provided here and in the git repositories, take care to replace my username, gavingcf, with your own username, and replace my project code, y23, with your own project code.

ssh [email protected]

The GeoChemFoam tutorials require a number of Python libraries to be available. On Archer2, users are required to maintain their own Python library via a so-called Python Virtual Environment, or PVE.. Your PVE is created as follows (remembering to replacing y23 with your project code and gavingcf with your own username).

module load cray-python
python -m venv --system-site-packages /work/y23/y23/gavingcf/myvenv
source /work/y23/y23/gavingcf/myvenv/bin/activate
python -m pip install -U pip setuptools
python -m pip install -U matplotlib scikit-image[optional] numpy-stl h5py

The first pip install line ensures both pip itself and setuptools are both up-to-date.

Then, to deactivate this environment, simply type:

deactivate

Once installed, you must activate your PVE before use either from the command line or in a batch script, using the two commands:

module load cray-python
source /work/y23/y23/gavingcf/myvenv/bin/activate

It is a good idea to ensure your PVE and its contents are up-to-date. This can be done by simply running the two install commands above, namely:

python -m pip install -U pip setuptools
python -m pip install -U matplotlib scikit-image[optional] numpy-stl h5py

And it can then be deactivated as above, using deactivate.

Further details regarding installing python packages on Archer2 can be found via https://docs.archer2.ac.uk/user-guide/python/.

You can either employ the centrally-installed version via the modules command, or you may install all the components yourself.

Here we describe how to use the modules of ARCHER2 to employ the centrally-installed version of GeoChemFoam-v5.0.

NB in these examples please remember: replace the username, gavingcf, and project code, y23, with your own username and project code.

When using the module for the first time, please create a runs directory in your work folder, as follows:

mkdir -p /work/y23/y23/gavingcf/works/GeoChemFoam-5.0/runs

To employ the centrally-installed version, and whenever you start a new terminal window or within a batch script, you must employ the following four commands:

module load other-software
module load gcfoam/5.0
source $GCFOAM_DIR/etc/bashrc_archer2

For ease of use, we recommend setting the following environment variable, replacing the username, gavingcf, and project code, y23, with your own username and project code.

export GCFOAM_RUNS=/work/y23/y23/gavingcf/works/GeoChemFoam-5.0/runs

You may also wish to employ the following aliases

alias run='cd $GCFOAM_RUNS'
alias dir='cd $GCFOAM_DIR'
alias src='cd $GCFOAM_DIR/src'
alias app='cd $GCFOAM_DIR/applications'
alias tut='cd $GCFOAM_DIR/tutorials'

This Section employs the centrally-installed version of GeoChemFoam-v5.0 via the module commands, and your own PVE. For Tutorial 1, the following Python libraries must be available via your PVE: “matplotlib”, “scikit-image[optional]”, “numpy-stl”, and “h5py”.

Before running GeoChemFoam, we must first enable our environment:

# enable GeoChemFoam-v5.0
module load other-software
module load gcfoam/5.0
source $GCFOAM_DIR/etc/bashrc_archer2

Replace username, gavingcf, and project code, y23, with your own username and project code.

export GCFOAM_RUNS=/work/y23/y23/gavingcf/works/GeoChemFoam-5.0/runs

# enable your PVE
module load cray-python
source /work/y23/y23/gavingcf/myvenv/bin/activate

# set these alias for ease of use
alias run='cd $GCFOAM_RUNS'
alias dir='cd $GCFOAM_DIR'
alias src='cd $GCFOAM_DIR/src'
alias app='cd $GCFOAM_DIR/applications'
alias tut='cd $GCFOAM_DIR/tutorials'

Finally, we access the GeoChemFoam wiki at https://github.com/GeoChemFoam/GeoChemFoam/wiki and then follow the instructions for Tutorial 1: https://github.com/GeoChemFoam/GeoChemFoam/wiki/Test-Case-01----Species-transport

cp -r $GCFOAM_TUTORIALS/transport/multiSpeciesTransportFoam/Ketton $GCFOAM_RUNS/.

Then change your directory to your copy of the tutorial 1 directory:

cd $GCFOAM_RUNS/Ketton

Before we progress Tutorial 1, we must first understand that running jobs from the command line, e.g., simply entering the command “./createMesh.sh”, will run the createMesh script on Archer2 “Login Nodes”; however, these nodes are designed for compilation, debugging and visualisation, and should not be used for running jobs.

All jobs should be run on the Compute Nodes, which are only accessed using the Archer2 batch system, Slurm

For Tutorial 1, there are a number of jobs which are executed in serial, e.g., createMesh.sh, and there are jobs which are run in parallel, e.g., runCaseFlow.sh. The following Archer2 examples of both a serial batch script and a parallel batch script are provided with the Tutorial in the files serial.bat and parallel.bat.

#!/bin/bash
#SBATCH --job-name=serial_jobs
#SBATCH --time=0:20:0
#SBATCH --ntasks=1
#SBATCH --account=y23
#SBATCH --partition=serial
#SBATCH --qos=serial

# Define memory required for this job. By default, you would
# get just under 2 GB, but you can ask for up to 125 GB.
#SBATCH --mem=4G

# Set the number of threads to 1 to avoid auto-threading
export OMP_NUM_THREADS=1

# Configure GCF
module load other-software
module load gcfoam/5.0
source $GCFOAM_DIR/etc/bashrc_archer2

# Enable your PVE
module load cray-python
source /work/y23/y23/gavingcf/myvenv/bin/activate

# Choose just one of the following serial scripts
./createMesh.sh
#./initCaseFlow.sh
#./initCaseTransport.sh

#!/bin/bash
#SBATCH --job-name=parallel
#SBATCH --time=0:20:0
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=1

# Replace [budget code] below with your budget code (e.g. t01)
#SBATCH --account=y23
#SBATCH --partition=standard
#SBATCH --qos=standard

# Set the number of threads to 1 to avoid auto-threading
export OMP_NUM_THREADS=1

# Propagate the cpus-per-task setting from script to srun commands
export SRUN_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK

# Configure GCF
module load other-software
module load gcfoam/5.0
source $GCFOAM_DIR/etc/bashrc_archer2

# Enable your PVE
module load cray-python
source /work/y23/y23/gavingcf/myvenv/bin/activate

# Exit if the total number of tasks do not equal the total number of processor directories.
export NP="$(find processor* -maxdepth 0 -type d -print| wc -l)"
if [ "$NP" -ne "$((SLURM_TASKS_PER_NODE*SLURM_JOB_NUM_NODES))" ]; then
     echo "Error: number of tasks does not equal the number of processor directories"
     echo "SBATCH ntasks-per-node times nodes must equal $NP"
     echo "However they are $SLURM_TASKS_PER_NODE and $SLURM_JOB_NUM_NODES, respectively"
     exit 1
fi

export PLATFORM=ARCHER2
echo -e "Run job_name in parallel on $NP $PLATFORM processors"

# Choose just one of the following parallel scripts
./runSnappyHexMesh.sh
#./runCaseFlow.sh
#./processFlow.sh
#./runCaseTransport.sh
#./processTransport.sh

You can now submit your jobs using

sbatch serial.bat

or

sbatch parallel.bat

You can monitor your job using your own username instead of gavingcf

squeue -u gavingcf

You can cancel your job using

scancel <jobid>

Here we present how to run Tutorial 01: Flow, Transport, and Permeability in a microCT image, as presented in the wiki: https://github.com/GeoChemFoam/GeoChemFoam/wiki/Test-Case-01----Species-transport

Follow the instructions in the wiki for the ‘Native version’. (The Docker instructions can be safely ignored). NB, all executables must be run using the batch system rather than the command line.

• Copy the tutorial into the runs directory.

  cp -r $GCFOAM_TUTORIALS/transport/multiSpeciesTransportFoam/Ketton $GCFOAM_RUNS/.
  

• As we are using the Native version of GeoChemFoam, run the test case from your runs directory.

  cd $GCFOAM_RUNS/Ketton
  

• We first need to define the domain, unpack the image and create a mesh, and this is done using createMesh.sh.

To run this on Archer2, edit the example serial batch script, namely serial.bat, and ensure the createMesh.sh line is the only script file to run, i.e., the last four lines should read as follows.

# Choose just one of the following serial scripts
./createMesh.sh
#./initCaseFlow.sh
#./initCaseTransport.sh

Save then submit this script by using

sbatch serial.bat

The file createMesh.sh is a shell script that runs various OpenFOAM and Python modules to create/prepare the input image and create a cartesian block mesh of the domain in order to run snappyHexMesh in parallel. For more scientific details, please visit the wiki.

• A mesh of the pore-space is then extracted using the parallel ./runSnappyHexMesh.sh script.

To run this on Archer2, edit the example parallel batch script, namely parallel.bat, and ensure the runSnappyHexMesh.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
./runSnappyHexMesh.sh
#./runCaseFlow.sh
#./processFlow.sh
#./runCaseTransport.sh
#./processTransport.sh

Save then submit this script by using

sbatch parallel.bat

Note, the wiki recommends employing paraview to check your mesh; however, paraview is not supported at this time.

• The next step is to prepare the flow field calculation using the ./initCaseFlow.sh script. As the mesh was created in parallel, the mesh is reconstructed and the velocity and pressure fields are decomposed on the processors.

To run this on Archer2, edit the example serial batch script, namely serial.bat, and ensure the initCaseFlow.sh line is the only script file to run, i.e., the last four lines should read as follows.

# Choose just one of the following serial scripts
#./createMesh.sh
./initCaseFlow.sh
#./initCaseTransport.sh

Then submit this script by using

sbatch serial.bat

• Calculate the flow field in the X direction from left to right using the runCaseFlow.sh script: ./runCaseFlow.sh. Simple foam is run to compute the flow in the pore space from the pressure drop defined in the initCaseFlow.sh script.

To run this on Archer2, edit the example parallel batch script, namely parallel.bat, and ensure the runCaseFlow.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
#./runSnappyHexMesh.sh
./runCaseFlow.sh
#./processFlow.sh
#./runCaseTransport.sh
#./processTransport.sh

Save then submit this script by using

sbatch parallel.bat

• The script ./processFlow.sh moves the results of simpleFoam to the '0' folder and calculates the permeability using 'processPoroPerm'.

To run this on Archer2, edit the example parallel batch script, namely parallel.bat, and ensure the processFlow.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
#./runSnappyHexMesh.sh
#./runCaseFlow.sh
./processFlow.sh
#./runCaseTransport.sh
#./processTransport.sh

Save then submit this script by using

sbatch parallel.bat

• The ./initCaseTransport.sh script is then run to prepare the case to run species transport. The initial concentration field for each species is created and decomposed in the processors as the mesh was created in parallel.

To run this on Archer2, edit the example serial batch script, namely serial.bat, and ensure the initCaseTransport.sh line is the only script file to run, i.e., the last four lines should read as follows.

# Choose just one of the following serial scripts
#./createMesh.sh
#./initCaseFlow.sh
./initCaseTransport.sh

Save then submit this script by using

sbatch serial.bat

• Calculate the transport using the runCaseTransport.sh script: ./runCaseTransport.sh. The script uses the 'multiSpeciesTransportFoam' solver.

To run this on Archer2, edit the example parallel batch script, namely parallel.bat, and ensure the runCaseTransport.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
#./runSnappyHexMesh.sh
#./runCaseFlow.sh
#./processFlow.sh
./runCaseTransport.sh
#./processTransport.sh

Save then submit this script by using

sbatch parallel.bat

• Finally, the concentrations are processed using the ./processTransport.sh script

To run this on Archer2, edit the example parallel batch script, namely parallel.bat, and ensure the processTransport.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
#./runSnappyHexMesh.sh
#./runCaseFlow.sh
#./processFlow.sh
#./runCaseTransport.sh
./processTransport.sh

Save then submit this script by using

sbatch parallel.bat

• Check for correctness

>cat A_conc.csv B_conc.csv
time C
0 0
0.01 0.274137
0.02 0.432351
0.03 0.554637
0.04 0.652788
0.05 0.730438
0.06 0.791012
0.07 0.838068
0.08 0.87449
0.09 0.90264
0.1 0.924441
time C
0 0
0.01 0.126229
0.02 0.239035
0.03 0.345767
0.04 0.443858
0.05 0.533867
0.06 0.613993
0.07 0.681486
0.08 0.737184
0.09 0.782178
0.1 0.818111

In the Tutorial 1 directly, namely Ketton, there is a folder named Archer2_workflow which contains 8 batch scripts: workflow_step_1.bat, workflow_step_2.bat, …, workflow_step_8.bat.

These 8 batch scripts are the steps of the workflow which automatically runs Tutorial 1 in full, where each job submits the next, in order.

To launch this workflow,

• copy all 8 files from the Archer2_workflow folder into current (Ketton) directory

cp Archer2_workflow/*.bat .

• Edit all 8 *.bat files and change the username, gavingcf, and project code, y23, to your username and project code,
• launch the workflow but submitting the first batch script:

sbatch workflow_step_1.bat

Only Tutorial 1 has been adapted to run on Archer2. All other tutorials require to be adapted by the user.

Firstly, all tutorials contain two types of scripts: scripts that run in serial and scripts that run in parallel. Scripts that run in parallel use a parallel job launcher called mpirun or mpiexec. Neither of these commands exist on Archer2, where srun is the parallel job launcher.

Therefore, for each tutorial, users must locate each script file that contains either mpirun or mpiexec and adapt it as follows.

For this example, let’s consider Tutorial 4, where four scripts employ mpiexec. If we read the script processCase.sh, we can see the line

mpiexec -np $NP processSolidArea -parallel > processSolidAreaRT.out

This single line must be replaced with the following six lines

# if PLATFORM is ARCHER2 then use srun, otherwise use mpiexec
    if [[ "${PLATFORM}" == "ARCHER2" ]]; then
       srun --distribution=block:block --hint=nomultithread -n $NP processSolidArea -parallel > processSolidAreaRT.out
    else
       mpiexec -np $NP processSolidArea -parallel > processSolidAreaRT.out
    fi

This new code checks if the platform is Archer2 and, if so, launches the parallel job using the srun command, otherwise the mpiexec command is run as before. This is a portable change, i.e., the script will continue to run as before but now it is also Archer2-enabled.

NB, all scripts, serial or parallel, must be submitted via Archer2’s batch system, where a script akin to parallel_archer2.bat above is used to employ the parallel scripts, and serial_archer2.bat above is used to employ the serial scripts.

The Sections above describe how to use the modules of ARCHER2 to employ the centrally-installed version of GeoChemFoam-v5.0. In this Section, we describe how to install GeoChemFoam-v5.0 in your /work directory and how to get it to employ the centrally-installed version of OpenFOAM. This work is close to the bare metal and is intended for code developers.

NB The following employs the project code y23 and the username gavingcf. Change all occurrences to your own project code and your own username.

Usually, users will employ the install the GeoChemFoam native version install instructions found in the GeoChemFoam wiki here: https://github.com/GeoChemFoam/GeoChemFoam/wiki

As stated in the wiki, GeoChemFoam-5.0 uses the ESI OpenCFD Release OpenFOAM-v2212. This is now loaded via modules on Archer2:

module load openfoam/com/v2212

The wiki then instructs users to create a folder named ‘works’ in their home folder. However, on Archer2, the home folder is not visible to the “compute” nodes. As such, we must create this ‘works’ folder in our work folder instead, specifically

cd /work/y23/y23/gavingcf
mkdir works
cd works

As per the wiki, we now clone GeoChemFoam

git clone https://github.com/GeoChemFoam/GeoChemFoam-5.0

The next stage is to set-up the GeoChemFoam’s own bashrc script, which is located in its ‘etc’ directory. The wiki informs us to change lines #13 and #14. For Archer2, we change these two lines from

export FOAM_INST_DIR=/usr/lib/openfoam
source /usr/lib/openfoam/openfoam2212/etc/bashrc

to read

export FOAM_INST_DIR=$FOAM_INSTALL_DIR
source $FOAM_INSTALL_DIR/etc/bashrc

For Archer2, we also need to change one further line, namely line #4, from

GCFOAM_DIR=$HOME/works/GeoChemFoam-5.0

to read

GCFOAM_DIR=/work/y23/y23/gavingcf/works/GeoChemFoam-5.0

The wiki suggests to create an alias for gf50 using

echo "alias gf50='source $HOME/works/GeoChemFoam-5.0/etc/bashrc'" >> $HOME/.bashrc

However, for our Archer2 installation, this reads

echo "alias gf50='source /work/y23/y23/gavingcf/works/GeoChemFoam-5.0/etc/bashrc'" >> $HOME/.bashrc

NB whenever you start a new terminal window, you must run either

source /work/y23/y23/gavingcf/works/GeoChemFoam-5.0/etc/bashrc

or simply use the alias:

gf50

Compilation follows as per the wiki:

The following command compile the code:

./Allwmake

To check that the code has correctly compiled, one can run the checkInstall command:

./checkInstall.sh

If the code has compiled correctly, the final line should be:

Installation successful

Once your own installation is installed, the example batch scripts can be altered simply to use your own installation rather than the central module version.

For all the example batch scripts, simply locate the lines

# Configure GCF using Archer2 module
module load other-software
module load gcfoam/5.0
source $GCFOAM_DIR/etc/bashrc_archer2

and replace these lines with the following:

# Configure GCF using own installation
module load openfoam/com/v2212
source /work/y23/y23/gavingcf/works/GeoChemFoam-5.0/etc/bashrc

NB replace y23 with your project code, and gavingcf with your username.

Once complete, simply follow the instructions above for “Running Tutorial 1 using gcfoam module”, “Running Tutorial 1 as a Single Workflow” and “Running other Tutorials”