HPC

Using gprMax in an HPC environment depends on the configuration of your cluster: compiler and library modules, programming environments, and job submission processes vary between systems.

Note

General details about the types of acceleration available in gprMax are shown in the OpenMP/CUDA/OpenCL/Apple Metal section.

Installation

Follow the installation guide to choose the package or source route and preserve any existing v3 environment. A cluster may supply Python, compilers and native libraries through modules instead of Conda.

For example, the following is an illustrative source-build recipe using the ARCHER2 module environment. Obtain a separate v4 checkout following Installing from source, select the desired revision, and run these commands from its root. Check your site’s module versions and the supported Python versions in the installation guide; do not assume that an unqualified clone of the default branch is the intended v4 revision:

$ module load PrgEnv-gnu
$ module load cray-python
$ module load cray-fftw
$ module load cray-hdf5-parallel
$ export CC=cc
$ export CXX=CC
$ export FC=ftn
$ python -m venv --system-site-packages --prompt gprMax .venv
$ source .venv/bin/activate
(gprMax)$ python -m pip install --upgrade pip
(gprMax)$ python -m pip install -r requirements.txt
(gprMax)$ HDF5_MPI='ON' python -m pip install --force-reinstall --no-deps --no-cache-dir --no-binary=h5py h5py
(gprMax)$ python -m pip install -e ".[mpi]"
(gprMax)$ python -c "import h5py; print(h5py.get_config().mpi)"

Here --system-site-packages intentionally exposes packages supplied by the loaded HPC Python module; omit it when that access is not wanted. This is a site-specific choice, not a requirement for ordinary PyPI installation. The forced h5py rebuild avoids silently retaining a serial installation; the final check must print True for parallel HDF5 output.

Tip

Consult your system’s documentation for site specific information.

Job Submission examples

High-performance computing (HPC) environments usually require jobs to be submitted to a queue using a job script. The following are examples of job scripts for an HPC environment that uses Open Grid Scheduler/Grid Engine, and are intended as general guidance to help you get started. The names of parallel environments (-pe) and compiler modules will depend on how they were defined by your system administrator.

The Grid Engine examples below use Bash and activate gprMax-v4 through a site-provided Anaconda module. Adapt that setup to the interpreter/environment you installed; Conda is not required. Submit from the working directory that contains mymodel.in. The -cwd directive keeps that directory, so the scripts do not change into a possibly unrelated old gprMax checkout.

OpenMP

gprmax_omp.sh

Here is an example of a job script for running models, e.g. A-scans to make a B-scan, one after another on a single cluster node. This is not as beneficial as the OpenMP/MPI example, but it can be a helpful starting point when getting the software running in your HPC environment. The behaviour of most of the variables is explained in the comments in the script.

#!/bin/bash
#####################################################################################
### Change to current working directory:
#$ -cwd

### Specify runtime (hh:mm:ss):
#$ -l h_rt=01:00:00

### Email options:
#$ -m ea -M joe.bloggs@email.com

### Parallel environment ($NSLOTS):
#$ -pe sharedmem 16

### Job script name:
#$ -N gprmax_omp.sh
#####################################################################################

### Initialise environment module
set -e
. /etc/profile.d/modules.sh

### Load and activate Anaconda environment for gprMax, i.e. Python 3 and required packages
module load anaconda
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate gprMax-v4

### Set number of OpenMP threads for each gprMax model
export OMP_NUM_THREADS=16

### Submit from the working directory containing mymodel.in (#$ -cwd above).
python -m gprMax mymodel.in -n 10

In this example 10 models will be run one after another on a single node of the cluster (on this particular cluster a single node has 16 cores/threads available). Each model will be parallelised using 16 OpenMP threads.

MPI domain decomposition

Here is an example of a job script for running a model across multiple tasks in an HPC environment using MPI. The behaviour of most of the variables is explained in the comments in the script.

Note

This example is based on the ARCHER2 system and uses the SLURM scheduler.

#!/bin/bash

### Job script name:
#SBATCH --job-name="gprMax MPI demo"

### Number of MPI tasks:
#SBATCH --ntasks=8

### Number of CPUs (OpenMP threads) per task:
#SBATCH --cpus-per-task=16

### Runtime limit:
#SBATCH --time=0:10:0

### Partition and quality of service to use (these control the type and
### amount of resources allowed to request):
#SBATCH --partition=standard
#SBATCH --qos=standard

### Hints to control MPI task layout:
#SBATCH --hint=nomultithread
#SBATCH --distribution=block:block


# Set number of OpenMP threads from SLURM environment variables
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# Ensure the cpus-per-task option is propagated to srun commands
export SRUN_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK

# Load system modules
module load PrgEnv-gnu
module load cray-python

# Load Python virtual environment
source .venv/bin/activate

# Run gprMax with input file
srun python -m gprMax my_model.in --mpi 2 2 2

In this example, the model will be divided across 8 MPI ranks in a 2 x 2 x 2 pattern:

Fig. 74 The full model (left) is evenly divided across MPI ranks (right).

The --mpi argument is passed to gprMax which takes three integers to define the number of MPI processes in the x, y, and z dimensions to form a cartesian grid.

Point sources and receivers have a single MPI owner. An internal partition plane belongs to the rank on its positive side; a global upper plane belongs to the last rank along that axis, including on PMC symmetry faces. This ownership rule does not relax source-specific placement restrictions or make every field component valid on a boundary.

Unlike the grid engine examples, here we specify the number of CPUs per task (16) and the number of tasks (8), rather than the total number of CPUs/slots.

Note

Some restrictions apply to the domain decomposition when using fractal geometry as explained here.

MPI task farm

gprmax_omp_taskfarm.sh

Here is an example of a job script for running models, e.g. A-scans to make a B-scan, distributed as independent tasks in an HPC environment using MPI. The behaviour of most of the variables is explained in the comments in the script.

#!/bin/bash
#####################################################################################
### Change to current working directory:
#$ -cwd

### Specify runtime (hh:mm:ss):
#$ -l h_rt=01:00:00

### Email options:
#$ -m ea -M joe.bloggs@email.com

### Resource reservation:
#$ -R y

### Parallel environment ($NSLOTS):
#$ -pe mpi 176

### Job script name:
#$ -N gprmax_omp_taskfarm.sh
#####################################################################################

### Initialise environment module
set -e
. /etc/profile.d/modules.sh

### Load and activate Anaconda environment for gprMax, i.e. Python 3 and required packages
module load anaconda
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate gprMax-v4

### Load OpenMPI
module load openmpi

### Set number of OpenMP threads per MPI task (each gprMax model)
export OMP_NUM_THREADS=16

### Submit from the working directory containing mymodel.in (#$ -cwd above).
mpirun -n 11 python -m gprMax mymodel.in -n 10 --taskfarm

In this example, 10 models will be distributed as independent tasks in an HPC environment using MPI.

--taskfarm is a boolean switch; it does not take a task count. The MPI launcher sets the number of ranks, while -n sets the number of models. One rank coordinates the farm and the remaining ranks execute complete models. Workers can each process several models, so a worker per model is not required. For example, mpiexec -n 3 python -m gprMax model.in -n 10 --taskfarm runs ten models using two workers and one coordinator.

If a worker’s model raises an exception, other submitted models are still processed. After all jobs finish, the coordinator raises TaskfarmError and the command exits unsuccessfully; successful output files are retained. The exception’s results and failures attributes expose the outcomes to Python callers. Distributed-domain failures likewise use a nonzero MPI abort status rather than reporting a successful run.

The NSLOTS variable which is required to set the total number of slots/cores for the parallel environment -pe mpi is usually the number of MPI tasks multiplied by the number of OpenMP threads per task. In this example the number of MPI tasks is 11 and the number of OpenMP threads per task is 16, so 176 slots are required.

Job array

gprmax_omp_jobarray.sh

Here is an example of a job script for running models, e.g. A-scans to make a B-scan, using the job array functionality of Open Grid Scheduler/Grid Engine. A job array is a single submit script that is run multiple times. It has similar functionality, for gprMax, to using the aforementioned MPI task farm. The behaviour of most of the variables is explained in the comments in the script.

#!/bin/bash
#####################################################################################
### Change to current working directory:
#$ -cwd

### Specify runtime (hh:mm:ss):
#$ -l h_rt=01:00:00

### Parallel environment ($NSLOTS):
#$ -pe sharedmem 16

### Job array and task IDs
#$ -t 1-10

### Job script name:
#$ -N gprmax_omp_jobarray.sh
#####################################################################################

### Initialise environment module
set -e
. /etc/profile.d/modules.sh

### Load and activate Anaconda environment for gprMax, i.e. Python 3 and required packages
module load anaconda
source "$(conda info --base)/etc/profile.d/conda.sh"
conda activate gprMax-v4

### Set number of OpenMP threads for each gprMax model
export OMP_NUM_THREADS=16

### Submit from the working directory containing mymodel.in (#$ -cwd above).
### One model per scheduler task; -n is a count, not the array's total size.
### A one-model run needs an explicit unique output prefix for each task.
python -m gprMax mymodel.in -n 1 -i "${SGE_TASK_ID:?Grid Engine must supply SGE_TASK_ID}" -o "mymodel_${SGE_TASK_ID}"

The scheduler’s -t 1-10 launches ten tasks. Each task passes its $SGE_TASK_ID as the one-based starting model number, -i, and uses -n 1 to execute exactly one model. For ordinary runs, -n is the number of models to execute from -i, not the total size of the scheduler array. Using -n 10 in every task would launch overlapping batches.

The explicit -o mymodel_TASK_ID prefix keeps output filenames distinct. A one-model run does not automatically append its -i value. For Python input blocks that depend on the total survey size, supply that survey size separately rather than increasing -n. This example concerns ordinary stepped runs, not study-managed restart rules.

A job array means that exactly the same submit script is going to be run multiple times, the only difference between each run is the environment variable $SGE_TASK_ID.