Apple Metal

The most computationally intensive parts of gprMax, which are the FDTD solver loops, have been parallelized using different CPU and GPU accelerators to offer performance and flexibility.

OpenMP which supports multi-platform shared memory multiprocessing.
NVIDIA CUDA for NVIDIA GPUs.
OpenCL for a wider range of CPU and GPU hardware.
Apple Metal for Apple Silicon (M-series) based Mac GPUs.

Each of these approaches to acceleration have different characteristics and hardware/software support. While all these approaches can offer increased performance, OpenMP + MPI can also increase the modelling capabilities of gprMax when running on a multi-node system (e.g. HPC environments). It does this by distributing models accoss multiple nodes, increasing the total amount of memory available and allowing larger models to be simulated.

Additionally, the Message Passing Interface (MPI) can be utilised to implement a simple task farm that can be used to distribute a series of models as independent tasks. This can be useful in many GPR simulations where a B-scan (composed of multiple A-scans) is required. Each A-scan can be task-farmed as an independent model, and within each model, OpenMP, CUDA, OpenCL, or Metal can still be used for parallelism. This creates mixed mode OpenMP/MPI, CUDA/MPI, OpenCL/MPI, or Metal/MPI environments.

Some of these accelerators and frameworks require additional software to be installed. The guidance below explains how to do that and gives examples of usage.

Use the Python selected during Installation. Conda is optional: a venv or a user-managed Python can run the same commands. If you use an environment, activate it (or select it explicitly) before installing bindings and running models. The examples below assume your working directory contains the examples/ tree; wheel users can obtain it with python -m gprMax.examples copy my-gprmax-work and then work in my-gprmax-work. A checkout is needed only for the source-build commands and repository-specific helpers explicitly identified below.

Installing optional accelerator bindings

The CPU/OpenMP solver is part of the core installation. The Python bindings for the other accelerators are optional package extras. For a released gprMax package installed from the Python Package Index (PyPI), install the binding for the required backend with one of the following commands:

$ python -m pip install "gprMax[cuda]"       # Linux/Windows
$ python -m pip install "gprMax[opencl]"
$ python -m pip install "gprMax[metal]"      # macOS

For an editable source installation, run the corresponding command from the top-level directory of the gprMax checkout:

$ python -m pip install -e ".[cuda]"       # Linux/Windows
$ python -m pip install -e ".[opencl]"
$ python -m pip install -e ".[metal]"      # macOS

The different targets are intentional: gprMax[...] names the released distribution, whereas .[...] refers to the checkout in the current directory and -e makes that source installation editable. Extras can be combined in either form, for example gprMax[cuda,opencl] or .[cuda,opencl]. The accelerators convenience extra, selected as either gprMax[accelerators] or .[accelerators], requests all accelerator bindings that apply to the current operating system. A core installation remains fully usable with the CPU solver when none of these extras is installed.

Warning

Python package installers can select dependencies by operating system but cannot determine whether a compatible accelerator, driver, CUDA toolkit, or OpenCL runtime is installed. The accelerators convenience extra can therefore fail on a machine without the necessary system software. The individual backend extra is the recommended installation method.

Note

You can use the get_host_spec.py module (in gprMax/toolboxes/Utilities) to help you understand what hardware (CPU/GPU) you have and how gprMax can use it with the aforementioned accelerators.

Solver precision

The CPU, CUDA, and OpenCL solvers support single- and double-precision field storage. Single precision is the default for both CPU and accelerator solves because it reduces memory use and generally gives the best accelerator performance.

For a text input file, select CPU precision with -cpu_precision and CUDA or OpenCL precision with -gpu_precision:

$ python -m gprMax model.in -cpu_precision double
$ python -m gprMax model.in -gpu -gpu_precision double

The corresponding Python API arguments are cpu_precision="double" and gpu_precision="double" on gprMax.run(). Each accepts single or double. The CPU option is ignored for an accelerator solve, and the GPU option is ignored for a CPU solve.

Note

Apple Metal supports single precision only. Requesting double precision with Metal is rejected before kernel compilation because the Metal Shading Language has no native double type.
Refining subgrids require double precision and override a requested single precision. A ratio=1 equal-resolution embedded region inherits the main grid’s selected CPU or CUDA precision. HSG subgrids support CPU and CUDA, but not OpenCL, Metal, or distributed MPI execution. See Advanced Model Building (Python API) for interpolation options and placement restrictions.
Output datasets and KSIR complex phasors use the type corresponding to the configured solver precision.

OpenMP

No additional software is required to use OpenMP as it is part of the standard installation of gprMax.

By default, gprMax will try to determine and use the maximum number of OpenMP threads (usually the number of physical CPU cores) available on your machine. You can override this behaviour in two ways: firstly, gprMax will check to see if the #omp_threads command is present in your input file; if not, gprMax will check to see if the environment variable OMP_NUM_THREADS is set. This can be useful if you are running gprMax in a High-Performance Computing (HPC) environment where you might not want to use all of the available CPU cores.

MPI

MPI support is optional and is not installed with the core gprMax package. It requires a system MPI implementation and the gprMax mpi extra. For a released package:

$ python -m pip install "gprMax[mpi]"

For an editable source install, use python -m pip install -e ".[mpi]" from the checkout instead.

The extra installs mpi4py but does not install or configure the system MPI runtime. You will also need to build h5py with MPI support if you plan to use parallel HDF5 output with MPI domain decomposition. For an existing source installation, MPI can instead be enabled without recompiling gprMax by installing the runtime followed by python -m pip install mpi4py.

There are two ways to use MPI with gprMax:

Domain decomposition - divides a single model across multiple MPI ranks.
Task farm - distribute multiple models as independent tasks to each MPI rank.

Domain decomposition

Open a terminal with the intended Python/MPI environment selected and work in the directory containing your copied examples or the matching checkout.

Run one of the 2D test models:

$ mpirun -n 4 python -m gprMax examples/gpr/basic/cylinder_Ascan_2D.in --mpi 2 2 1

The --mpi argument passed to gprMax takes three integers to define the number of MPI processes in the x, y, and z dimensions to form a cartesian grid. The product of these three numbers shoud equal the number of MPI ranks. In this case 2 x 2 x 1 = 4.

Discrete plane waves can span MPI subdomains. Their small one-dimensional DPW state is replicated on every rank, and the TFSF surface corrections are partitioned by Yee-component ownership. Axial layered profiles are assembled once during model construction, so plane waves add no source-specific communication to the timestep loop.

Eigenmode ports can also span MPI subdomains. Their component-resolved modal cross-sections are assembled once during construction, TF/SF source terms are partitioned by field ownership, and the modal spectra are reduced only at finalisation. A virtual waveguide replicates its compact auxiliary Yee grid on each rank and performs one aperture-sized magnetic-field collective per time step to preserve bidirectional coupling.

PEC and PMC symmetry boundaries may be used on MPI domain faces. Boundary construction and the PMC ghost-image update are dispatched only on ranks that touch the selected global face. Physical domain-edge corrections are similarly restricted to ranks that touch both adjoining global faces, so internal halo seams retain the ordinary distributed Yee update.

Internal one-axis PML slabs may cross MPI partitions in any direction. Their global CFS grading is sliced between participating ranks without restarting at a partition, and only rank-local PML history arrays are allocated. The normal field-halo exchanges join the corrected fields, so these slabs introduce no additional per-timestep MPI collective.

MPI domain decomposition diagram — Fig. 72 Example decomposition using 8 MPI ranks in a 2 x 2 x 2 pattern (specified with `--mpi 2 2 2`). The full model (left) is evenly divided across MPI ranks (right).

Decomposition of Fractal Geometry

There are some restrictions when using MPI domain decomposition with fractal user objects.

Warning

gprMax will throw an error during the model build phase if the MPI decomposition is incompatible with the model geometry.

#fractal_box

When a fractal box has a mixing model attached, it will perform a parallel fast Fourier transforms (FFTs) as part of its construction. When performing a parallel FFT in 3D space, the decomposition must be either 1D or 2D - it cannot be decomposed in all 3 dimensions. To support this, the MPI domain decomposition of the fractal box must have size one in at least one dimension:

_images/fractal_domain_decomposition.png — Fig. 73 Example slab and pencil decompositions. These decompositions could be specified with `--mpi 8 1 1` and `--mpi 3 3 1` respectively.

Note

This does not necessarily mean the whole model domain needs to be divided this way. So long as the volume covered by the fractal box is divided into either slabs or pencils, the model can be built. This includes the volume covered by attached surfaces added by the #add_surface_water, #add_surface_roughness, or #add_grass commands.

#add_surface_roughness

When adding surface roughness, a parallel fast Fourier transform is applied across the 2D surface of a fractal box. Therefore, the MPI domain decomposition across the surface must be size one in at least one dimension.

For example, in figure Fig. 73, surface roughness can be attached to any surface when using the slab decomposition. However, if using the pencil decomposition, it could not be attached to the XY surfaces.

#add_grass

Warning

Domain decomposition of grass is not currently supported. Grass can still be built in a model so long as it is fully contained within a single MPI rank.

Task farm

A failed task does not discard the other jobs: the batch finishes, successful outputs remain available, and workers are joined before a TaskfarmError is raised on every MPI rank. An uncaught failed batch therefore reports a nonzero launcher exit status. Python API callers that handle the error must catch it on all ranks; the communicator remains usable. The exception’s failures mapping identifies failed zero-based job indices. Complete results are retained on the master; workers receive failure records with None placeholders for successful results.

By default, the MPI task farm functionality is turned off. It can be used with the --taskfarm command line option, which specifies the total number of MPI tasks, i.e. master + workers, for the MPI task farm. This option is most usefully combined with -n to allow individual models to be farmed out using an MPI task farm, e.g. to create a B-scan with 60 traces and use MPI to farm out each trace:

$ python -m gprMax examples/gpr/basic/cylinder_Bscan_2D.in -n 60 --taskfarm

CUDA

Software required

The following steps provide guidance on how to install the extra components to allow gprMax to run on your NVIDIA GPU:

Install the NVIDIA CUDA Toolkit. You can follow the Installation Guides in the NVIDIA CUDA Toolkit Documentation You must ensure the version of CUDA you install is compatible with the compiler you are using. This information can usually be found in a table in the CUDA Installation Guide under System Requirements.
You may need to add the location of the CUDA compiler (nvcc) to your user path environment variable, for example C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\vX.X\bin on Windows or the toolkit’s bin directory on Linux.
Install the CUDA extra using the command for the relevant installation type under Installing optional accelerator bindings. This installs pycuda. Modern macOS releases do not support NVIDIA CUDA, so the dependency is guarded by a platform marker.

On Windows, PyCUDA compiles kernels when a model starts. Finding nvcc is not sufficient: its MSVC host compiler also needs the C++ headers and libraries. Install Visual Studio Build Tools with Desktop development with C++, using a version supported by your CUDA Toolkit. From PowerShell in the repository root, activate your Python environment first, then initialize the compiler:

# First select your Python environment, for example:
conda activate gprMax-v4
. ./packaging/activate_cuda.ps1
python -m gprMax examples/gpr/basic/cylinder_Ascan_2D.in -gpu

The leading dot loads the environment into the current PowerShell session. Repeat it in a new session. The script locates Visual Studio using vswhere and imports its x64 compiler environment; it does not install software or change the system PATH. If using an installed package without this repository, start an x64 Native Tools Command Prompt for Visual Studio, activate the Python environment there, and run the model from that prompt.

Example

Open a terminal on Linux or a compiler-enabled prompt on Windows with the intended Python environment selected. Work in the directory containing your copied examples or the matching checkout; Conda is not required.

Run one of the test models:

$ python -m gprMax examples/gpr/basic/cylinder_Ascan_2D.in -gpu

Note

If you want to select a specific GPU card on your system, you can specify an integer after the -gpu flag. The integer should be the NVIDIA CUDA device ID for a specific GPU card. If it is not specified it defaults to device ID 0.
You can use the get_host_spec.py module (in gprMax/toolboxes/Utilities) to help you understand what hardware (CPU/GPU) you have and how gprMax can use it.

OpenCL

Software required

The following steps provide guidance on how to install the extra components to allow gprMax to use OpenCL:

Install a vendor OpenCL implementation/ICD for the intended CPU or GPU.
Install the OpenCL extra using the command for the relevant installation type under Installing optional accelerator bindings. This installs pyopencl; it does not install the vendor OpenCL runtime.

Example

Open a terminal with the intended Python environment selected and work in the directory containing your copied examples or the matching checkout.

Run one of the test models:

$ python -m gprMax examples/gpr/basic/cylinder_Ascan_2D.in -opencl

Note

If you want to select a specific computer device on your system, you can specify an integer after the -opencl flag. The integer should be the device ID for a specific compute device. If it is not specified it defaults to device ID 0.
You can use the get_host_spec.py module (in gprMax/toolboxes/Utilities) to help you understand what hardware (CPU/GPU) you have and how gprMax can use it.

Apple Metal

Apple Metal provides high-performance GPU acceleration for Apple Silicon (M-series) based Mac systems. The Metal backend in gprMax leverages the unified memory architecture and optimized compute shaders to deliver significant performance improvements over CPU-only execution.

System requirements

The Apple Metal backend requires:

macOS 11 or later - required by Apple Silicon Macs
Apple Silicon (M-series) based GPU
pyobjc-framework-metal - Python bindings for Apple Metal framework

Software required

The following Python package is required to use Apple Metal acceleration:

Install the Metal extra using the command for the relevant installation type under Installing optional accelerator bindings. This installs pyobjc-framework-Metal only on macOS.

Note

Metal is not installed by the core package or conda_env.yml. This keeps the same environment file portable across Linux, Windows, and macOS. It is available on compatible Macs after installing the metal extra.

Example

Open a terminal on macOS with the intended Python environment selected and work in the directory containing your copied examples or the matching checkout.

Run one of the test models with Metal acceleration:

$ python -m gprMax examples/gpr/basic/cylinder_Ascan_2D.in -metal

Note

The Metal backend automatically selects the best available GPU device on your Mac system
Metal is available only on macOS; select the CPU, CUDA, or OpenCL backend explicitly on other platforms
For debugging or development purposes, you can use the get_host_spec.py module (in gprMax/toolboxes/Utilities) to understand your hardware capabilities

CUDA/MPI

Message Passing Interface (MPI) has been utilised to implement a simple task farm that can be used to distribute a series of models as independent tasks. This is described in more detail in the HPC section. MPI can be combined with the GPU functionality to allow a series of models to be distributed to multiple GPUs on the same machine (node).

Example

For example, to run a B-scan that contains 60 A-scans (traces) on a system with 4 GPUs:

$ python -m gprMax examples/gpr/basic/cylinder_Bscan_2D.in -n 60 --taskfarm -gpu 0 1 2 3

Note

When running a task farm, one MPI rank runs on the CPU as a coordinator (master) while the remaining worker ranks each use their own GPU. Therefore the number of MPI ranks should equal the number of GPUs + 1. The integers given with the -gpu argument are the NVIDIA CUDA device IDs for the specific GPU cards to be used.