All notable changes to this project will be documented here.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

0.5.0 - 2020-06-05


  • The name of the runner shell script in the output directories is the one passed to the CLI instead of the hardcoded one.

  • All the names of the CLI options have been prefixed with --exe- to prevent name clashes with other plugins options.

  • It is easier to define the settings to execute the runner shell script for a test case thanks to a dedicated section in test-settings.yaml.

  • Rename test_case.yaml to test-settings.yaml.


  • Testing on MacOS.

  • --exe-test-module CLI option for setting the default test module

  • Add timeout setting for the runner execution.


  • The log files testing in the builtin test module.


  • Tests execution order when a test module is in sub-directory of the yaml settings.

  • Marks of a test case not propagated to all test modules.

0.4.0 - 2020-05-03


  • equal_nan option is too specific and can easily be added with a custom fixture.

0.3.1 - 2020-03-30


  • Report generation can handle a sphinx _static directory.

0.3.0 - 2020-03-19


  • How to use skip and xfail marks in the docs.

  • How to use a proxy with anaconda in the docs.

  • Better error message when --runner do not get a script.


  • Placeholder in the runner script are compliant with bash (use {{}} instead of {}).

  • Report generation is done for all the tests at once and only requires a report generator script.


  • #8393: check that --clean-output and --overwrite-output are not used both.

  • Output directory creation no longer fails when the input directory tree has one level.


  • Useless --nproc command line argument, because this can be done with a custom default test_case.yaml passed to the command line argument --default-settings.

0.2.1 - 2020-01-14


  • #7043: skip regression tests when reference files are missing, no longer raise error.