GeoChemFoam on ARCHER2 - GeoChemFoam/GeoChemFoam GitHub Wiki

Introduction

This document describes how to employ GeoChemFOAM on ARCHER2 using modules and how to install it yourself from scratch.

NB In all the examples in this document and in the example batch scripts provided here and in the git repositories, take care to replace my username, <code>gavingcf</code>, with your own username, and replace my project code, <code>y23</code>, with your own project code.

Logging in to Archer2

Python Virtual Environment

The GeoChemFoam tutorials require a number of Python libraries to be available. On Archer2, users are required to maintain their own Python library via a so-called Python Virtual Environment, or PVE.. Your PVE is created as follows (remembering to replacing y23 with your project code and gavingcf with your own username).

module load cray-python
python -m venv --system-site-packages /work/y23/y23/gavingcf/myvenv
source /work/y23/y23/gavingcf/myvenv/bin/activate
python -m pip install -U pip setuptools
python -m pip install -U matplotlib scikit-image[optional] numpy-stl h5py
The first pip install line ensures both pip itself and setuptools are both up-to-date.

Then, to deactivate this environment, simply type:

deactivate

Once installed, you must activate your PVE before use either from the command line or in a batch script, using the two commands:

module load cray-python
source /work/y23/y23/gavingcf/myvenv/bin/activate

It is a good idea to ensure your PVE and its contents are up-to-date. This can be done by simply running the two install commands above, namely:

python -m pip install -U pip setuptools
python -m pip install -U matplotlib scikit-image[optional] numpy-stl h5py

And it can then be deactivated as above, using deactivate.

Further details regarding installing python packages on Archer2 can be found via [https://docs.archer2.ac.uk/user-guide/python/](https://docs.archer2.ac.uk/user-guide/python/).

Accessing GeoGhemFoam using centrally-installed modules

You can either employ the centrally-installed version via the modules command, or you may install all the components yourself.

Here we describe how to use the modules of ARCHER2 to employ the centrally-installed version of GeoChemFoam-v5.1.

NB in these examples please remember: replace the username, gavingcf, and project code, y23, with your own username and project code.

When using the module for the first time, please create a runs directory in your work folder, as follows:

mkdir -p /work/y23/y23/gavingcf/works/GeoChemFoam-5.1/runs

Loading the module for every new terminal window

To employ the centrally-installed version, and whenever you start a new terminal window or within a batch script, you must employ the following four commands:

module load other-software
module load gcfoam/5.1
source $GCFOAM_DIR/etc/bashrc_archer2

For ease of use, we recommend setting the following environment variable, replacing the username, gavingcf, and project code, y23, with your own username and project code.

export GCFOAM_RUNS=/work/y23/y23/gavingcf/works/GeoChemFoam-5.1/runs

You may also wish to employ the following aliases

alias run='cd $GCFOAM_RUNS'
alias dir='cd $GCFOAM_DIR'
alias src='cd $GCFOAM_DIR/src'
alias app='cd $GCFOAM_DIR/applications'
alias tut='cd $GCFOAM_DIR/tutorials'

Tutorial 1 on Archer2 using module gcfoam

This Section employs the centrally-installed version of GeoChemFoam-v5.1 via the module commands, and your own PVE. For Tutorial 1, the following Python libraries must be available via your PVE: “matplotlib”, “scikit-image[optional]”, “numpy-stl”, and “h5py”.

Configuration

Before running GeoChemFoam, we must first enable our environment:

# enable GeoChemFoam-v5.1
module load other-software
module load gcfoam/5.1
source $GCFOAM_DIR/etc/bashrc_archer2

Replace username, gavingcf, and project code, y23, with your own username and project code.

export GCFOAM_RUNS=/work/y23/y23/gavingcf/works/GeoChemFoam-5.1/runs

# enable your PVE
module load cray-python
source /work/y23/y23/gavingcf/myvenv/bin/activate

# set these alias for ease of use
alias run='cd $GCFOAM_RUNS'
alias dir='cd $GCFOAM_DIR'
alias src='cd $GCFOAM_DIR/src'
alias app='cd $GCFOAM_DIR/applications'
alias tut='cd $GCFOAM_DIR/tutorials'
cp -r $GCFOAM_TUTORIALS/transport/multiSpeciesTransportFoam/Ketton $GCFOAM_RUNS/.
Then change your directory to your copy of the tutorial 1 directory: 
cd $GCFOAM_RUNS/Ketton

How to run GeoChemFoam jobs on Archer2

Before we progress Tutorial 1, we must first understand that running jobs from the command line, e.g., simply entering the command “./createMesh.sh”, will run the createMesh script on Archer2 “Login Nodes”; however, these nodes are designed for compilation, debugging and visualisation, and should not be used for running jobs.

All jobs should be run on the Compute Nodes, which are only accessed using the Archer2 batch system, Slurm

For Tutorial 1, there are a number of jobs which are executed in serial, e.g., createMesh.sh, and there are jobs which are run in parallel, e.g., runCaseFlow.sh. The following Archer2 examples of both a serial batch script and a parallel batch script are provided with the Tutorial in the files serial_archer2.bat and parallel_archer2.bat.

Serial batch script

#!/bin/bash
#SBATCH --job-name=serial_jobs
#SBATCH --time=0:20:0
#SBATCH --ntasks=1
#SBATCH --account=y23
#SBATCH --partition=serial
#SBATCH --qos=serial

# Define memory required for this job. By default, you would
# get just under 2 GB, but you can ask for up to 125 GB.
#SBATCH --mem=4G

# Set the number of threads to 1 to avoid auto-threading
export OMP_NUM_THREADS=1

# Configure GCF
module load other-software
module load gcfoam/5.1
source $GCFOAM_DIR/etc/bashrc_archer2

# Enable your PVE
module load cray-python
source /work/y23/y23/gavingcf/myvenv/bin/activate

# Choose just one of the following serial scripts
./createMesh.sh
#./initCaseFlow.sh
#./initCaseTransport.sh

Parallel batch script

#!/bin/bash
#SBATCH --job-name=parallel
#SBATCH --time=0:20:0
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=8
#SBATCH --cpus-per-task=1

# Replace [budget code] below with your budget code (e.g. t01)
#SBATCH --account=y23
#SBATCH --partition=standard
#SBATCH --qos=standard

# Set the number of threads to 1 to avoid auto-threading
export OMP_NUM_THREADS=1

# Propagate the cpus-per-task setting from script to srun commands
export SRUN_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK

# Configure GCF
module load other-software
module load gcfoam/5.1
source $GCFOAM_DIR/etc/bashrc_archer2

# Enable your PVE
module load cray-python
source /work/y23/y23/gavingcf/myvenv/bin/activate

# Exit if the total number of tasks do not equal the total number of processor directories.
export NP="$(find processor* -maxdepth 0 -type d -print| wc -l)"
if [ "$NP" -ne "$((SLURM_TASKS_PER_NODE*SLURM_JOB_NUM_NODES))" ]; then
     echo "Error: number of tasks does not equal the number of processor directories"
     echo "SBATCH ntasks-per-node times nodes must equal $NP"
     echo "However they are $SLURM_TASKS_PER_NODE and $SLURM_JOB_NUM_NODES, respectively"
     exit 1
fi

export PLATFORM=ARCHER2
echo -e "Run job_name in parallel on $NP $PLATFORM processors"

# Choose just one of the following parallel scripts
./runSnappyHexMesh.sh
#./runCaseFlow.sh
#./processFlow.sh
#./runCaseTransport.sh
#./processTransport.sh

Managing Slurm batch jobs

You can now submit your jobs using

sbatch serial_archer2.bat

or

sbatch parallel_archer2.bat

You can monitor your job using your own username instead of gavingcf

squeue -u gavingcf

You can cancel your job using

scancel <jobid>

Running Tutorial 1 using gcfoam module

Here we present how to run Tutorial 01: Flow, Transport, and Permeability in a microCT image, as presented in the wiki: https://github.com/GeoChemFoam/GeoChemFoam/wiki/Test-Case-01----Species-transport

Follow the instructions in the wiki for the ‘Native version’. (The Docker instructions can be safely ignored). NB, all executables must be run using the batch system rather than the command line.

  • Copy the tutorial into the runs directory.

    cp -r $GCFOAM_TUTORIALS/transport/multiSpeciesTransportFoam/Ketton $GCFOAM_RUNS/.

  • As we are using the Native version of GeoChemFoam, run the test case from your runs directory.

    cd $GCFOAM_RUNS/Ketton

  • We first need to define the domain, unpack the image and create a mesh, and this is done using createMesh.sh.

To run this on Archer2, edit the example serial batch script, namely serial_archer2.bat, and ensure the createMesh.sh line is the only script file to run, i.e., the last four lines should read as follows.

# Choose just one of the following serial scripts
./createMesh.sh
#./initCaseFlow.sh
#./initCaseTransport.sh
Save then submit this script by using 
sbatch serial_archer2.bat
The file createMesh.sh is a shell script that runs various OpenFOAM and Python modules to create/prepare the input image and create a cartesian block mesh of the domain in order to run snappyHexMesh in parallel. For more scientific details, please visit the wiki.

  • A mesh of the pore-space is then extracted using the parallel ./runSnappyHexMesh.sh script.

To run this on Archer2, edit the example parallel batch script, namely parallel_archer2.bat, and ensure the runSnappyHexMesh.sh line is the only script file to run, i.e., the last fix lines should read as follows. 

# Choose just one of the following parallel scripts.
./runSnappyHexMesh.sh
#./runCaseFlow.sh
#./processFlow.sh
#./runCaseTransport.sh
#./processTransport.sh
Save then submit this script by using 
sbatch parallel_archer2.bat
Note, the wiki recommends employing paraview to check your mesh. Instructions on how to run paraview on Archer2 can be found at the bottom of this wiki page.

  • The next step is to prepare the flow field calculation using the ./initCaseFlow.sh script. As the mesh was created in parallel, the mesh is reconstructed and the velocity and pressure fields are decomposed on the processors.

To run this on Archer2, edit the example serial batch script, namely serial_archer2.bat, and ensure the initCaseFlow.sh line is the only script file to run, i.e., the last four lines should read as follows. 

# Choose just one of the following serial scripts
#./createMesh.sh
./initCaseFlow.sh
#./initCaseTransport.sh
Then submit this script by using 
sbatch serial_archer2.bat

  • Calculate the flow field in the X direction from left to right using the runCaseFlow.sh script: ./runCaseFlow.sh. Simple foam is run to compute the flow in the pore space from the pressure drop defined in the initCaseFlow.sh script.

To run this on Archer2, edit the example parallel batch script, namely parallel_archer2.bat, and ensure the runCaseFlow.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
#./runSnappyHexMesh.sh
./runCaseFlow.sh
#./processFlow.sh
#./runCaseTransport.sh
#./processTransport.sh
Save then submit this script by using 
sbatch parallel_archer2.bat
* The script ./processFlow.sh moves the results of simpleFoam to the '0' folder and calculates the permeability using 'processPoroPerm'.

To run this on Archer2, edit the example parallel batch script, namely parallel_archer2.bat, and ensure the processFlow.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
#./runSnappyHexMesh.sh
#./runCaseFlow.sh
./processFlow.sh
#./runCaseTransport.sh
#./processTransport.sh
Save then submit this script by using 
sbatch parallel_archer2.bat

  • The ./initCaseTransport.sh script is then run to prepare the case to run species transport. The initial concentration field for each species is created and decomposed in the processors as the mesh was created in parallel.

To run this on Archer2, edit the example serial batch script, namely serial_archer2.bat, and ensure the initCaseTransport.sh line is the only script file to run, i.e., the last four lines should read as follows.

# Choose just one of the following serial scripts
#./createMesh.sh
#./initCaseFlow.sh
./initCaseTransport.sh
Save then submit this script by using 
sbatch serial_archer2.bat

  • Calculate the transport using the runCaseTransport.sh script: ./runCaseTransport.sh. The script uses the 'multiSpeciesTransportFoam' solver.

To run this on Archer2, edit the example parallel batch script, namely parallel_archer2.bat, and ensure the runCaseTransport.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
#./runSnappyHexMesh.sh
#./runCaseFlow.sh
#./processFlow.sh
./runCaseTransport.sh
#./processTransport.sh

Save then submit this script by using

sbatch parallel_archer2.bat
* Finally, the concentrations are processed using the ./processTransport.sh script

To run this on Archer2, edit the example parallel batch script, namely parallel_archer2.bat, and ensure the processTransport.sh line is the only script file to run, i.e., the last fix lines should read as follows.

# Choose just one of the following parallel scripts.
#./runSnappyHexMesh.sh
#./runCaseFlow.sh
#./processFlow.sh
#./runCaseTransport.sh
./processTransport.sh
Save then submit this script by using 
sbatch parallel_archer2.bat

  • Check for correctness 

    >cat A_conc.csv B_conc.csv
time C
0 0
0.01 0.274137
0.02 0.432351
0.03 0.554637
0.04 0.652788
0.05 0.730438
0.06 0.791012
0.07 0.838068
0.08 0.87449
0.09 0.90264
0.1 0.924441
time C
0 0
0.01 0.126229
0.02 0.239035
0.03 0.345767
0.04 0.443858
0.05 0.533867
0.06 0.613993
0.07 0.681486
0.08 0.737184
0.09 0.782178
0.1 0.818111

Running Tutorial 1 as a Single Workflow

In the Tutorial 1 directly, namely Ketton, there is a folder named Archer2_workflow which contains 8 batch scripts: workflow_step_1.bat, workflow_step_2.bat, …, workflow_step_8.bat.

These 8 batch scripts are the steps of the workflow which automatically runs Tutorial 1 in full, where each job submits the next, in order.

To launch this workflow,

  • copy all 8 files from the Archer2_workflow folder into current (Ketton) directory 

    cp Archer2_workflow/*.bat .

  • Edit all 8 *.bat files and change the username, gavingcf, and project code, y23, to your username and project code,

  • launch the workflow but submitting the first batch script: 

    sbatch workflow_step_1.bat

Running other Tutorials

Only Tutorial 1 has been adapted to run on Archer2. All other tutorials require to be adapted by the user.

Firstly, all tutorials contain two types of scripts: scripts that run in serial and scripts that run in parallel. Scripts that run in parallel use a parallel job launcher called mpirun or mpiexec. Neither of these commands exist on Archer2, where srun is the parallel job launcher.

Therefore, for each tutorial, users must locate each script file that contains either mpirun or mpiexec and adapt it as follows.

For this example, let’s consider Tutorial 4, where four scripts employ mpiexec. If we read the script processCase.sh, we can see the line

mpiexec -np $NP processSolidArea -parallel > processSolidAreaRT.out
This single line must be replaced with the following six lines 
# if PLATFORM is ARCHER2 then use srun, otherwise use mpiexec
    if [[ "${PLATFORM}" == "ARCHER2" ]]; then
       srun --distribution=block:block --hint=nomultithread -n $NP processSolidArea -parallel > processSolidAreaRT.out
    else
       mpiexec -np $NP processSolidArea -parallel > processSolidAreaRT.out
    fi

This new code checks if the platform is Archer2 and, if so, launches the parallel job using the srun command, otherwise the mpiexec command is run as before. This is a portable change, i.e., the script will continue to run as before but now it is also Archer2-enabled.

NB, all scripts, serial or parallel, must be submitted via Archer2’s batch system, where a script akin to parallel_archer2.bat above is used to employ the parallel scripts, and serial_archer2.bat above is used to employ the serial scripts.

Installing GeoChemFoam on Archer2 from source code

The Sections above describe how to use the modules of ARCHER2 to employ the centrally-installed version of GeoChemFoam-v5.1. In this Section, we describe how to install GeoChemFoam-v5.1 in your /work directory and how to get it to employ the centrally-installed version of OpenFOAM. This work is close to the bare metal and is intended for code developers.

NB The following employs the project code <code>y23</code> and the username <code>gavingcf</code>. Change all occurrences to your own project code and your own username.

Usually, users will employ the install the GeoChemFoam native version install instructions found in the GeoChemFoam wiki here: https://github.com/GeoChemFoam/GeoChemFoam/wiki

Setup default environment

As stated in the wiki, GeoChemFoam-5.1 uses the ESI OpenCFD Release OpenFOAM-v2212. This is now loaded via modules on Archer2:

module load openfoam/com/v2212

To employ a 64-bit version OpenFOAM v2212, the single module load command above with the following two module lines:

module load other-software
module load openfoam/v2212_64

NB the suggested method to simply add '-WM_LABEL_SIZE=64' to the 'source ${FOAM_INSTALL_DIR}/etc/bashrc' has not been enabled.

The wiki then instructs users to create a folder named ‘works’ in their home folder. However, on Archer2, the home folder is not visible to the “compute” nodes. As such, we must create this ‘works’ folder in our work folder instead, specifically

cd /work/y23/y23/gavingcf
mkdir works
cd works
As per the wiki, we now clone GeoChemFoam 
git clone https://github.com/GeoChemFoam/GeoChemFoam-5.1

The next stage is to set-up the GeoChemFoam’s own bashrc script, which is located in its ‘etc’ directory. The wiki informs us to change lines #13 and #14. For Archer2, we change these two lines from

export FOAM_INST_DIR=/usr/lib/openfoam
source /usr/lib/openfoam/openfoam2212/etc/bashrc

to read

export FOAM_INST_DIR=$FOAM_INSTALL_DIR
source $FOAM_INSTALL_DIR/etc/bashrc

For Archer2, we also need to change one further line, namely line #4, from

GCFOAM_DIR=$HOME/works/GeoChemFoam-5.1
to read 
GCFOAM_DIR=/work/y23/y23/gavingcf/works/GeoChemFoam-5.1

The wiki suggests to create an alias for gf50 using

echo "alias gf50='source $HOME/works/GeoChemFoam-5.1/etc/bashrc'" >> $HOME/.bashrc

However, for our Archer2 installation, this reads

echo "alias gf50='source /work/y23/y23/gavingcf/works/GeoChemFoam-5.1/etc/bashrc'" >> $HOME/.bashrc

NB whenever you start a new terminal window, you must run either

source /work/y23/y23/gavingcf/works/GeoChemFoam-5.1/etc/bashrc

or simply use the alias:

gf50

Compilation

Compilation follows as per the wiki:

The following command compile the code:

./Allwmake

To check that the code has correctly compiled, one can run the checkInstall command:

./checkInstall.sh

If the code has compiled correctly, the final line should be:

Installation successful

Tutorial 1 on Archer2 using your own installation

Once your own installation is installed, the example batch scripts can be altered simply to use your own installation rather than the central module version.

For all the example batch scripts, simply locate the lines

# Configure GCF using Archer2 module
module load other-software
module load gcfoam/5.1
source $GCFOAM_DIR/etc/bashrc_archer2

and replace these lines with the following:

# Configure GCF using own installation
module load openfoam/com/v2212
source /work/y23/y23/gavingcf/works/GeoChemFoam-5.1/etc/bashrc

NB replace y23 with your project code, and gavingcf with your username.

Alternatively, if you have installed a 64-bit version, use the following as the replacement lines. 

# Configure GCF using own installation
module load other-software
module load openfoam/v2212_64
source /work/y23/y23/gavingcf/works/GeoChemFoam-5.1/etc/bashrc

NB replace y23 with your project code, and gavingcf with your username.

Once complete, simply follow the instructions above for “Running Tutorial 1 using gcfoam module”, “Running Tutorial 1 as a Single Workflow” and “Running other Tutorials”

Running Paraview on Archer2

ParaView is a data visualisation and analysis package, and is installed on Archer2 so the visualisation libraries and applications can be used to post-process simulation data. The ParaView server (pvserver), batch application (pvbatch), and the Python interface (pvpython) are all available. Users are able to run the server on the compute nodes and connect to a local ParaView client running on their own computer.

You should connect to pvserver from a local ParaView client running on your own computer, ensuring the version of ParaView you have installed locally is the same as the one on Archer2, version 5.10.1.

Further, ensure your private ssh key to your ~/.ssh directory on Archer2.

First, launch the ParaView client on your workstation.

Then, launch a local terminal and log into your Archer2 account. On this Archer2 terminal, type 

srun --nodes=1 --exclusive --time=00:20:00 --partition=standard --qos=short --account=ecseaj02 --pty /bin/bash
remembering to change the account name. Wait for a new prompt and therein type the two lines: 
module load paraview
srun --overlap --oversubscribe -n 4 pvserver --mpi --force-offscreen-rendering
which will return, after a short time, something similar to the following 
Waiting for client…
Connection URL: cs://nid001023:11111
Accepting connection(s): nid001023:11111

Note the digits between nid and the colon (here it’s 001023).

Now launch a second local terminal and use it to run the following command to login into Archer2, correcting those noted nid digits and your username (keep 11111): 

ssh -L 11111:nid001023:11111 [email protected]

Now, on your local client ParaView, click "File" then "Connect…​" to connect to the pvserver on Archer2 using the following connection settings:

Name: archer2 Server Type: Client/Server Host: localhost Port: 11111 Note

If it has connected correctly, you should see the following:

Waiting for client…​ Connection URL: cs://nid001023:11111 Accepting connection(s): nid001023:11111 Client connected.

Then first window reports something similar to the following 

Waiting for client…
Connection URL: cs://nid001023:11111
Accepting connection(s): nid001023:11111

Example Usage: view wireframe of decomposition using Paraview

"File" → "Open" → "Paraview_Loader.foam" (ignore "empty file" warning)→ Under Properties, change “Case Type” from “Reconstructed Case” to “Decomposed Case”. Once rendered, locate the "Representation Toolbar" and change “Surface” to “Outline”.

⚠️ **GitHub.com Fallback** ⚠️