ReFrame Test Suite

gprMax includes a test suite built using ReFrame.

This is not a unit testing framework, instead it provides a mechanism to perform regression checks on whole model runs. Reference files for regression checks are automatically generated and stored in reframe_tests/regression_checks.

Attention

The regression checks are sensitive to floating point precision errors so are currently specific to the ARCHER2 system. Additional work is required to make them portable between systems.

Run the test suite

Running the test suite requires ReFrame to be installed:

$ pip install reframe-hpc

The full test suite can be run with:

$ cd reframe_tests
$ reframe -c tests/ -r

If you are running on a HPC system, you will need to be provide a configuration file:

$ reframe -C configuration/archer2_settings.py -c tests/ -r

A ReFrame configuration script for ARCHER2 is provided in the reframe_tests/configuration folder. Configurations for additional machines can be added here.

Tip

The full test suite is quite large. ReFrame provides a number of ways to filter the tests you want to run such as -n and -t (by name and tag respectively). There is much more information in the ReFrame documentation.

There is also an example job submission script for running the suite as a long running job on ARCHER2. Any additional arguments are passed forwarded to ReFrame. E.g.

$ sbatch job_scripts/archer2_tests.slurm -n Snapshot

would run all tests with “Snapshot” in the test name.

Developer guide

Tests are defined in the reframe_tests/tests folder with gprMax input files stored in reframe_tests/tests/src.

Base test classes

Every regression test inherits from the GprMaxBaseTest class. This class contains all the logic for launching a gprMax job, checking the simulation completed, and running any regression checks.

Additionally, every test depends on the CreatePyenvTest class. It creates a new Python environment that all other tests will use.

`CreatePyenvTest`	Create a fresh virtual environment for running the tests.
`GprMaxBaseTest`	Base class that all GprMax tests should inherit from.

Tip

Avoid rebuilding the Python environment every time you run the test suite by running ReFrame with the --restore-session flag.

Adding a new test

The easiest way to learn how to write a new test is by looking at the existing tests. The test below runs the B-scan model provided with gprMax:

import reframe as rfm
from reframe.core.builtins import parameter

from reframe_tests.tests.mixins import BScanMixin, ReceiverMixin

@rfm.simple_test
class TestBscan(BScanMixin, ReceiverMixin, GprMaxBaseTest):
    tags = {"test", "serial", "bscan"}
    sourcesdir = "src/bscan_tests"
    model = parameter(["cylinder_Bscan_2D"])
    num_models = parameter([64])

@rfm.simple_test - marks the class as a ReFrame test.
BScanMixin and ReceiverMixin - mixin classes alter the behaviour to test specific gprMax functionality.
tags - set tags that can be used to filter tests.
sourcesdir - path to test source directory.
model - gprMax input filename (without file extension). This is a ReFrame parameter so it can take multiple values to run the same test with multiple input files.
num_models - parameter specific to the BScanMixin.

Test dependencies

Tests can also define a test dependency. This uses the ReFrame test dependency mechanism to link tests. The dependent test can access the resources and outputs of the test it depends on. This means we can create a test that should produce an identical result to another test, but is configured differently. For example, to test the MPI domain decomposition functionality using the previous B-scan model, we can add:

from reframe_tests.tests.mixins import MpiMixin

@rfm.simple_test
class TestBscanMPI(MpiMixin, TestBscan):
    tags = {"test", "mpi", "bscan"}
    mpi_layout = parameter([[2, 2, 1]])
    test_dependency = TestBscan

Our new class inherits from the above TestBscan class.
Use the MpiMixin to run with the gprMax domain decomposition functionality.
Override tags.
mpi_layout - parameter specific to the MpiMixin.
test_dependency - Depend on the TestBscan class. It is not sufficient to just inherit from the class. The output from this test will compared with the output from the TestBscan test.

Note

Some parameters, such as model are unified between test dependencies. I.e. the dependent test and test dependency will have the same value for the parameter.

If a mixin class adds a new parameter, this may need to be unified as well. For an example of how to do this, see the BScanMixin class and the num_models parameter.

Mixin classes

The different mixin classes are used to alter the behaviour of a given test to support testing gprMax functionality - snapshots, geometry objects, geometry views - and different runtime configurations such as task farms, MPI, and the Python API.

Important

When creating a new test, the mixin class must be specified earlier in the inheritance list than the base ReFrame test class:

class TestAscan(ReceiverMixin, GprMaxBaseTest):
    pass

`BScanMixin`	Test a B-scan model - a model with a moving source and receiver.
`GeometryObjectsReadMixin`	Read geometry object(s) created by a test dependency.
`GeometryObjectsWriteMixin`	Add regression tests for geometry objects.
`GeometryOnlyMixin`	Run test with geometry only flag
`GeometryViewMixin`	Add regression tests for geometry views.
`MpiMixin`	Run test using GprMax MPI functionality.
`PythonApiMixin`	Use the GprMax Python API rather than a standard input file.
`ReceiverMixin`	Add regression tests for receivers.
`SnapshotMixin`	Add regression tests for snapshots.
`TaskfarmMixin`	Run test using GprMax taskfarm functionality.

Regression checks

Note

To make the test suite portable between systems, the main changes would be to these regression check classes. Specifically the way hdf5 files are compared.

There are a number of classes that perform regression checks.

`RegressionCheck`	Compare two files using diff
`H5RegressionCheck`	Compare two hdf5 files using h5diff
`ReceiverRegressionCheck`	Run regression check on individual reveivers in output files.
`GeometryObjectRegressionCheck`	Run regression check on a GprMax GeometryObject.
`GeometryObjectMaterialsRegressionCheck`	Run regression check on materials output by a GeometryObject.
`GeometryViewRegressionCheck`	Run regression check on a GprMax GeometryView.
`SnapshotRegressionCheck`	Run regression check on a gprMax Snapshot.